Software automated testing method and system based on visual attention mechanism

Abstract

The present invention provides a software automated testing method and system based on a visual attention mechanism, which relates to the field of automated testing technology, and includes obtaining an initial interface of a software to be tested and performing preprocessing, inputting an enhanced interface image into an interface detection model, and identifying control information by analyzing the enhanced interface image; inputting the control information into a pre-trained attention mechanism model, generating semantic associations and context information between controls based on the control information, determining the execution order and execution parameters of test actions, simulating manual test actions in turn, and performing test operations on the controls according to the execution parameters; obtaining interface feedback information of the software to be tested, extracting text content in the interface feedback information based on a text recognition algorithm, determining the semantics corresponding to the text content in combination with a pre-built semantic understanding model, comparing the text content with a preset expected result, and generating a test conclusion corresponding to the test operation.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a software automated testing method and system based on a visual attention mechanism. Background Art

[0002] Modern software development and testing face complex and ever-changing challenges. Traditional software testing methods often rely on manual design and execution of test cases, which is inefficient and difficult to cover all possible usage scenarios and exceptions when facing large-scale, complex, and multi-functional software systems.

[0003] With the development of artificial intelligence and machine learning technologies, visual attention mechanisms have been introduced into software automated testing to improve test efficiency and coverage. The visual attention mechanism imitates the working principle of the human visual system and can automatically identify and focus on important visual elements and controls in complex software interfaces. This technology allows automated testing to more accurately simulate user vision and operational behavior, thereby more comprehensively checking the functionality and performance of the software.

[0004] In summary, software automation testing methods not only improve the efficiency and coverage of testing, but also better adapt to the challenges of complexity and diversity of modern software systems and solve problems in existing technologies. Summary of the invention

[0005] The embodiments of the present invention provide a software automation testing method and system based on a visual attention mechanism, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A software automation testing method based on visual attention mechanism is provided, including:

[0008] Acquire the initial interface of the software to be tested and perform preprocessing to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify control information by parsing the enhanced interface image, wherein the control information includes control type, control attribute and control position;

[0009] Input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters;

[0010] After each of the test operations is completed, the interface feedback information of the software to be tested is obtained, and the text content in the interface feedback information is extracted based on the text recognition algorithm. The semantics corresponding to the text content is determined in combination with the pre-built semantic understanding model, and compared with the preset expected results to generate a test conclusion corresponding to the test operation.

[0011] In an optional embodiment,

[0012] Inputting the enhanced interface image into an interface detection model, and identifying the control information by parsing the enhanced interface image includes:

[0013] The interface detection model is built based on the semantic segmentation model. The receptive field of the enhanced interface image is expanded through the dilated convolution, the multi-scale context information of each pixel is captured, pixel-level classification is performed, a semantic label is assigned to each pixel, and boundary smoothing operation is performed through conditional random field post-processing to generate a semantic segmentation map.

[0014] Based on the semantic segmentation map, instance segmentation is performed using a regional watershed algorithm to combine pixels belonging to the same control into independent control instances, thereby generating a control instance segmentation map;

[0015] Based on the control instance segmentation graph, the control attributes corresponding to each control instance are extracted, the control nodes are constructed, the spatial position relationship between the control nodes is analyzed, the directed edges are established, and the control relationship graph is generated. The control relationship graph is input into a pre-trained graph neural network model, the control topology structure and spatial node features are extracted, and the control relationship graph attributes are generated;

[0016] Based on the semantic segmentation map, the control instance segmentation map and the control relationship map attributes, a structured control information representation is constructed to generate control information.

[0017] In an optional embodiment,

[0018] Based on the semantic segmentation map, instance segmentation is performed using a regional watershed algorithm to combine pixels belonging to the same control into independent control instances. Generating a control instance segmentation map includes:

[0019] Based on a multi-scale Sobel operator, a horizontal gradient and a vertical gradient are respectively calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are respectively squared and added to obtain square roots to obtain a gradient amplitude map;

[0020] Traverse each pixel point in the gradient amplitude map, and compare the gradient amplitude of the current pixel point with that of the surrounding 8 neighboring pixels based on the current pixel point. If the gradient amplitude of the current pixel point is the minimum value of the domain and is less than the preset gradient amplitude threshold, mark the current pixel point as a seed point; otherwise, skip the current pixel point; repeat the iterative comparison until all the pixels in the gradient amplitude map are traversed;

[0021] The gradient amplitudes corresponding to the seed points are sorted from small to large to form a priority queue, and the first seed point is selected from the priority queue as the current seed point to start region expansion:

[0022] The current seed point is used as the current processing area, and an empty expansion queue is created. Unmarked pixels are selected from the 8 neighboring pixels corresponding to the current seed point. If the gradient amplitude of the unmarked pixel is less than the preset gradient amplitude threshold, the unmarked pixel is used as an extended pixel and added to the expansion queue. At the same time, the current processing area is expanded and marked. The expansion is repeated for each extended pixel in the expansion queue, and the pixels are removed from the expansion queue until the expansion queue is empty. The expansion of the current processing area is completed, and the expansion completion area is generated;

[0023] Process each seed point in the priority queue in turn until all seed points are processed and a series of expansion completion areas are determined;

[0024] Calculate the geometric properties of each extended area, retain the valid area according to the preset area invalidation rules, generate a unique identification number for each valid area, extract the corresponding circumscribed rectangular bounding box, and output the control instance segmentation map.

[0025] In an optional embodiment,

[0026] Based on the multi-scale Sobel operator, the horizontal gradient and the vertical gradient are calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are squared and added to obtain the gradient amplitude map, including:

[0027] The multi-scale Sobel operator has the following formula:

[0028] ;

[0029] in, s Indicates scale, G x s Representation scale s The horizontal gradient is G y s Representation scale s The vertical gradient isk represents the operator radius, m Indicates the horizontal offset. n Indicates the vertical offset, x Represents the horizontal coordinate of the pixel. y Represents the vertical coordinate of the pixel. I (·) represents the pixel value, S x ( m,n,s ) indicates scale s Sobel horizontal operator template, S y ( m,n, s ) indicates scale s Sobel vertical operator template.

[0030] In an optional embodiment,

[0031] Inputting the control information into a pre-trained attention mechanism model, generating semantic associations and context information between controls based on the control information, and determining the execution order and execution parameters of the test actions include:

[0032] The attention mechanism model is built based on the Transformer architecture. The encoder in the attention mechanism model calculates the correlation and attention weight between controls through the self-attention mechanism, extracts high-level feature representations, performs nonlinear transformation on the high-level feature representations through a feedforward neural network, models the semantic association and context information between controls, and converts the control information into a low-dimensional dense vector representation.

[0033] At each time step, the decoder in the attention mechanism model receives the action embedding vector generated at the previous time step and the low-dimensional dense vector representation output by the encoder, calculates the correlation between the test action at the current time step and the historically generated action through the multi-head self-attention mechanism, performs nonlinear transformation on the features through the feedforward neural network, obtains the hidden state vector corresponding to the current time step, maps the hidden state vector to the test action space through linear transformation, obtains the probability distribution of the next test action based on the softmax function, selects the action corresponding to the maximum probability from the probability distribution, and adds it to the test action sequence as the output of the current time step, until the length of the test action sequence reaches a preset maximum length threshold;

[0034] Based on the test action sequence, determining the execution order of the test actions;

[0035] For each test action, the execution parameters of the test action are determined based on the semantic association and context information of the control.

[0036] In an optional embodiment,

[0037] After each of the test operations is completed, interface feedback information of the software to be tested is obtained, text content in the interface feedback information is extracted based on a text recognition algorithm, and the semantics corresponding to the text content is determined in combination with a pre-built semantic understanding model, and compared with the preset expected results to generate a test conclusion corresponding to the test operation, including:

[0038] After the test operation is completed, the interface feedback information of the software to be tested is obtained, and the interface feedback information includes interface screenshots, control properties, log information and network request response data;

[0039] Based on the control attribute, determining a control response representation corresponding to a control state;

[0040] Based on the log information and the network request response data, extract the corresponding response text representation;

[0041] Extracting text content from the interface screenshot based on a text recognition algorithm, wherein the text recognition algorithm is constructed based on a convolutional neural network and a recurrent neural network, extracting local features of the image through a convolutional layer, modeling feature sequence dependencies through a recurrent layer, and converting the feature sequence dependencies into text content through a transcription layer;

[0042] Inputting the text content into a pre-built semantic understanding model for semantic analysis, wherein the semantic understanding model converts the text content into a semantic structured representation based on pre-defined semantic parsing rules;

[0043] Compare any one or more of the control response representation, the response text representation and the semantic structured representation with the preset expected result, obtain the comparison result and confidence level, determine the execution result, and if the execution result belongs to the preset failure type, backtrace the entire path of the test operation, combine the execution result, and construct failure information; if the execution result belongs to the preset success type, continue to execute the next test operation until all test operations are completed and determine the success information; generate a test conclusion based on the failure information and the success information.

[0044] According to a second aspect of the embodiments of the present invention,

[0045] Provided is a software automated testing system based on visual attention mechanism, including:

[0046] The first unit is used to obtain the initial interface of the software to be tested and perform preprocessing to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify control information by parsing the enhanced interface image, wherein the control information includes control type, control attribute and control position;

[0047] The second unit is used to input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters;

[0048] The third unit is used to obtain interface feedback information of the software to be tested after each of the test operations is completed, extract the text content in the interface feedback information based on the text recognition algorithm, determine the semantics corresponding to the text content in combination with a pre-built semantic understanding model, compare it with the preset expected results, and generate a test conclusion corresponding to the test operation.

[0049] According to a third aspect of the embodiments of the present invention,

[0050] An electronic device is provided, comprising:

[0051] processor;

[0052] a memory for storing processor-executable instructions;

[0053] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0054] A fourth aspect of the embodiments of the present invention is:

[0055] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0056] In the embodiments of the present invention, the use of a pre-trained interface detection model can automatically identify the location and type of controls in an image; automatically identify the type and properties of controls, reduce the need for manual operation, and improve the speed and accuracy of recognition; quickly process a large amount of image data, quickly identify and locate various controls in the interface, and save time and labor costs in the testing and development process; capture image edge information at different scales, accurately detect the boundaries of controls, and combine the gradient amplitude calculation in the horizontal and vertical directions to more comprehensively extract boundary information and generate high-quality gradient amplitude maps; dynamically select seed points to avoid interference from noise and irrelevant points, and improve the accuracy and robustness of seed point selection; Automatically generate test action sequences to realize automated execution of test processes, reduce the need for manual intervention, and improve test efficiency and reliability; through intelligent control test operations, ensure the consistency and comprehensiveness of tests, while reducing omissions and errors that may occur in manual testing; after each test operation is completed, the system can obtain complete interface feedback information of the software under test, including interface screenshots, control properties, log information, network request response data and other data; if the test operation does not achieve the expected effect, trace back the execution path of the test operation, and record the input, output and status information of each step in detail. The failure information is used for problem location and repair, providing a key reference for software improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the process of software automation testing method based on visual attention mechanism according to an embodiment of the present invention;

[0058] Figure 2 Schematic diagram of the structure of a software automation testing system based on a visual attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0061] Figure 1 FIG. 1 is a flow chart of a software automation testing method based on a visual attention mechanism according to an embodiment of the present invention. Figure 1As shown, the method includes:

[0062] S101. Acquire the initial interface of the software to be tested and preprocess it to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify the control information by parsing the enhanced interface image, wherein the control information includes the control type, control properties and control location;

[0063] In this embodiment, the use of a pre-trained interface detection model can automatically identify the location and type of controls in an image; automatically identify control types and properties, reduce the need for manual operations, and improve recognition speed and accuracy; quickly process large amounts of image data, quickly identify and locate various controls in the interface, and save time and labor costs in the testing and development process; and is suitable for various types of software interfaces, including desktop applications, mobile applications, and web applications, with strong versatility and adaptability.

[0064] In an optional embodiment, inputting the enhanced interface image into an interface detection model, and identifying the control information by parsing the enhanced interface image includes:

[0065] The interface detection model is built based on the semantic segmentation model. The receptive field of the enhanced interface image is expanded through the dilated convolution, the multi-scale context information of each pixel is captured, pixel-level classification is performed, a semantic label is assigned to each pixel, and boundary smoothing operation is performed through conditional random field post-processing to generate a semantic segmentation map.

[0066] Based on the semantic segmentation map, instance segmentation is performed using a regional watershed algorithm to combine pixels belonging to the same control into independent control instances, thereby generating a control instance segmentation map;

[0067] Based on the control instance segmentation graph, the control attributes corresponding to each control instance are extracted, the control nodes are constructed, the spatial position relationship between the control nodes is analyzed, the directed edges are established, and the control relationship graph is generated. The control relationship graph is input into a pre-trained graph neural network model, the control topology structure and spatial node features are extracted, and the control relationship graph attributes are generated;

[0068] Based on the semantic segmentation map, the control instance segmentation map and the control relationship map attributes, a structured control information representation is constructed to generate control information.

[0069] For the input enhanced interface image, dilated convolution is used to extract image features. By inserting holes in the convolution kernel, dilated convolution can expand the receptive field and capture the multi-scale contextual information of pixels without increasing the number of parameters and computation. The extracted feature maps are classified at the pixel level, and corresponding semantic labels are assigned to each pixel, such as buttons, text boxes, pictures, etc. The semantic segmentation results are post-processed with conditional random fields to optimize the segmentation boundaries and improve segmentation accuracy. The conditional random field smoothes and optimizes the preliminary segmentation results by modeling the interactions and contextual relationships between pixels, making the segmentation boundaries more accurate and smooth.

[0070] Based on the semantic segmentation map, the regional watershed algorithm is used to perform instance segmentation on the image. The regional watershed algorithm divides the image into different regions by treating the image as a terrain surface and simulating the process of water flooding from the lowest point of the terrain. Each region corresponds to a control instance. The gradient amplitude of each pixel is calculated on the semantic segmentation map, and the gradient amplitude is used as the height of the terrain. Starting from the pixel with the smallest gradient amplitude, that is, the lowest point of the terrain, the flooding process of water is simulated; when two different water areas meet, a watershed is established between the water areas to separate the water areas; through the regional watershed algorithm, pixels belonging to the same control can be combined into an independent instance to generate a control instance segmentation map. Each control instance corresponds to a unique identifier.

[0071] According to the control instance segmentation map, the corresponding attribute information of each control instance is extracted, such as the control type, position, size, etc. The control attribute information can be obtained by analyzing the semantic labels and spatial distribution of the pixels contained in the control instance; each control instance is represented as a node, and the node attributes include the type, position, size, etc. of the control. The spatial position relationship between control nodes is analyzed, such as relative position, alignment, inclusion relationship, etc., and directed edges are established between the corresponding nodes to represent the relationship between them, and a control relationship graph is constructed, in which the nodes represent the control instances and the edges represent the spatial relationship between the controls. The control relationship graph carries the hierarchy and layout information of the interface.

[0072] The constructed control relationship graph is input into the pre-trained graph neural network model to learn the feature representation of control nodes. The graph neural network can effectively process graph structured data, learn the feature representation of nodes through message passing and aggregation operations, and capture the interactions and relationships between nodes.

[0073] Exemplarily, the graph neural network iteratively aggregates the neighbor information of each node, updates the feature representation of the node, uses different aggregation functions such as summation, averaging, attention mechanism, etc., and gradually captures the high-order interactions and relationships between control nodes through multiple rounds of iterations; after learning the graph neural network, the feature representation of each control node and the overall attribute representation of the control relationship graph are obtained. The control attributes include the topological structure information and spatial layout information of the control.

[0074] A structured control information representation is constructed by comprehensively utilizing the semantic segmentation map, the control instance segmentation map, and the control relationship map attributes. For each control instance, the corresponding semantic label is extracted from the semantic segmentation map, the instance identifier is obtained from the control instance segmentation map, and the attribute information and relationship information with other controls are obtained from the control relationship map attributes. This information is organized into a structured data format, such as JSON or XML, for subsequent processing and application. The structured control information representation should be able to fully describe the control composition, hierarchy, and layout relationship of the interface.

[0075] In this embodiment, a semantic label is assigned to each pixel to accurately identify different types of controls in the interface, making interface detection more detailed and capable of identifying complex interface structures; by viewing the image as a terrain surface, the watershed algorithm can be used to effectively combine pixels of the same control into an independent instance, and different control instances can be accurately segmented to generate a control instance segmentation map; the control relationship graph not only contains basic properties such as the type, position, and size of the control, but also reflects the spatial position relationship and layout information between the controls, which can better describe the hierarchy of the interface and the relationship between the controls, facilitating subsequent analysis and processing; the interface detection and control information extraction are automatically processed, reducing human intervention and manual operations, improving processing speed and efficiency, and being suitable for large-scale interface testing and analysis; the different controls in the interface can be identified in a fine-grained manner, rich attribute information can be extracted, and a relationship graph between the controls can be established, providing a more comprehensive and in-depth interface analysis capability.

[0076] In an optional embodiment, based on the semantic segmentation map, instance segmentation is performed by using a regional watershed algorithm, pixels belonging to the same control are combined into independent control instances, and generating a control instance segmentation map includes:

[0077] Based on a multi-scale Sobel operator, a horizontal gradient and a vertical gradient are respectively calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are respectively squared and added to obtain square roots to obtain a gradient amplitude map;

[0078] Traverse each pixel point in the gradient amplitude map, and compare the gradient amplitude of the current pixel point with that of the surrounding 8 neighboring pixels based on the current pixel point. If the gradient amplitude of the current pixel point is the minimum value of the domain and is less than the preset gradient amplitude threshold, mark the current pixel point as a seed point; otherwise, skip the current pixel point; repeat the iterative comparison until all the pixels in the gradient amplitude map are traversed;

[0079] The gradient amplitudes corresponding to the seed points are sorted from small to large to form a priority queue, and the first seed point is selected from the priority queue as the current seed point to start region expansion:

[0080] The current seed point is used as the current processing area, and an empty expansion queue is created. Unmarked pixels are selected from the 8 neighboring pixels corresponding to the current seed point. If the gradient amplitude of the unmarked pixel is less than the preset gradient amplitude threshold, the unmarked pixel is used as an extended pixel and added to the expansion queue. At the same time, the current processing area is expanded and marked. The expansion is repeated for each extended pixel in the expansion queue, and the pixels are removed from the expansion queue until the expansion queue is empty. The expansion of the current processing area is completed, and the expansion completion area is generated;

[0081] Process each seed point in the priority queue in turn until all seed points are processed and a series of expansion completion areas are determined;

[0082] Calculate the geometric properties of each extended area, retain the valid area according to the preset area invalidation rules, generate a unique identification number for each valid area, extract the corresponding circumscribed rectangular bounding box, and output the control instance segmentation map.

[0083] For each semantic category of the semantic segmentation map, the horizontal and vertical gradients are calculated respectively. The multi-scale Sobel operator is used to extract the gradient information at different scales through convolution kernels of different scales. For each pixel, the horizontal and vertical gradients of the pixel are squared respectively, then added and squared to obtain the gradient amplitude corresponding to the pixel. Repeat the above steps to calculate the gradient amplitude of each pixel in the semantic segmentation map and generate a gradient amplitude map. The gradient amplitude map reflects the edge and contour information of different regions in the semantic segmentation map.

[0084] Traverse each pixel in the gradient amplitude map and set the current pixel as the center pixel; for the current center pixel, compare the gradient amplitude of the center pixel with the surrounding 8 neighborhood pixels. If the gradient amplitude of the center pixel is the minimum value in the neighborhood and is less than the preset gradient amplitude threshold, mark the center pixel as a seed point; if the center pixel does not meet the above conditions, skip the pixel and do not use it as a seed point; repeat the above steps until all pixels in the gradient amplitude map are traversed and all seed points are found. The seed point is usually located inside the control instance, with a small gradient amplitude, and is the starting point for area expansion.

[0085] All the found seed points are sorted from small to large according to the corresponding gradient amplitude to form a priority queue; the seed point with the smallest gradient amplitude in the priority queue is at the head of the queue, and the seed point with the largest gradient amplitude is at the end of the queue; the priority queue ensures that when expanding the area, it starts from the seed point with the smallest gradient amplitude and gradually expands to the seed point with the larger gradient amplitude.

[0086] Take the seed point at the head of the priority queue as the current seed point and start the region expansion. Create an empty expansion queue with the current seed point as the center to store the newly added pixels during the expansion process; check the 8 neighboring pixels around the current seed point. For each unmarked neighboring pixel point, if the corresponding gradient amplitude is less than the preset gradient amplitude threshold, each unmarked neighboring pixel point is taken as an extended pixel point, added to the expansion queue, and marked as processed; update the current processing area, and include the extended pixel point in the current processing area; repeat the above steps, process each extended pixel point in the extension queue in turn, take the extended pixel point as the new center point, and continue to check and expand the neighboring pixel points until the extension queue is empty, indicating that the region corresponding to the current seed point has been expanded; mark the extended region as an independent control instance, and remove the current seed point from the priority queue.

[0087] The remaining seed points are taken out from the priority queue in turn, and the region expansion process is repeated for each seed point; the region expansion process of each seed point is independent of each other, and independent control instance regions are generated; the region expansion is repeated until the priority queue is empty, that is, all seed points have been processed.

[0088] For each expanded control instance area, calculate the geometric properties, such as area, aspect ratio, and circumscribed rectangular bounding box; filter out invalid areas such as those with too small an area or abnormal aspect ratio according to the preset invalid area rules, and retain valid control instance areas; generate a unique identification number for each valid control instance area to distinguish different control instances; extract the circumscribed rectangular bounding box of each control instance area, and record the corresponding position and size information.

[0089] Integrate the identification numbers and bounding rectangle box information of all valid control instance areas into a data structure to generate a control instance segmentation map; each control instance in the control instance segmentation map corresponds to a unique identification number and contains the position and size information of the control instance; the control instance segmentation map can be used for subsequent control attribute extraction, control relationship analysis and other tasks.

[0090] In this embodiment, by using a multi-scale Sobel operator to calculate the gradient, the edge information of the image at different scales is captured, the boundary of the control is accurately detected, and the gradient amplitude calculation in the horizontal and vertical directions is combined to more comprehensively extract the boundary information and generate a high-quality gradient amplitude map; by comparing the gradient amplitude of each pixel point with the surrounding 8 neighborhood pixels and dynamically selecting the seed point, the interference of noise and irrelevant points can be avoided, and the accuracy and robustness of the seed point selection can be improved; the priority queue driven region expansion method is adopted to ensure that the expansion process starts from the seed point with the lowest gradient amplitude and expands step by step, so as to better control the order and range of the region expansion and ensure the rationality and consistency of the segmentation results; in the process of region expansion, each expanded pixel point is dynamically processed, which can flexibly adapt to controls of different shapes and sizes and improve the accuracy and flexibility of segmentation.

[0091] In an optional embodiment, based on a multi-scale Sobel operator, a horizontal gradient and a vertical gradient are respectively calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are respectively squared and added to obtain a square root, so as to obtain a gradient amplitude map including:

[0092] The multi-scale Sobel operator has the following formula:

[0093] ;

[0094] in, s Indicates scale, G x s Representation scale s The horizontal gradient is G y s Representation scale s The vertical gradient is k represents the operator radius, m Indicates the horizontal offset. n Indicates the vertical offset, x Represents the horizontal coordinate of the pixel. y Represents the vertical coordinate of the pixel. I (·) represents the pixel value, S x ( m,n,s ) indicates scales Sobel horizontal operator template, S y ( m,n, s ) indicates scale s Sobel vertical operator template.

[0095] Through the image I ( x+m,y+n ) is weighted summed up in the horizontal direction of pixels, using a scale of s Sobel horizontal operator template S x ( m,n,s ) is weighted, the process is given by x,y ) k The vertical direction is also the same logic. s The value range is 3 or 5;

[0096] according to k The calculation formula is k The value of represents the radius of the Sobel operator template. k Determines the range of pixels considered when calculating the gradient. All considered pixels are located in ( x+m,y+n ) in the neighborhood of;

[0097] For example, when the scale s At 3 o'clock, S x ( m,n, 3) The values ​​are as follows:

[0098] ;

[0099] S y ( m,n, 3) The values ​​are as follows:

[0100] ;

[0101] According to the formula, the details and edge features of the image can be captured at different scales using a multi-scale Sobel operator template. Operator templates of different scales can adapt to target objects of different sizes and shapes, improving the adaptability and accuracy of image processing; by calculating the horizontal and vertical gradients of each pixel, and then performing square and square root operations on them, a gradient amplitude map is obtained, which effectively highlights the edge information and intensity changes of each area in the image, providing important auxiliary information for subsequent segmentation and recognition.

[0102] S102. Input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters;

[0103] In this embodiment, the attention mechanism model is used to effectively capture and model the complex relationships between controls, including semantic relevance and functional dependency, thereby improving the intelligence and automation level of test actions. By intelligently determining the order and parameters of test actions, the complexity and errors of manual testing are reduced, and test efficiency and accuracy are improved. By automatically generating test action sequences, the test process is automatically executed, the need for manual intervention is reduced, and the efficiency and reliability of testing are improved. Through intelligent control test operations, the consistency and comprehensiveness of the test are ensured, while reducing omissions and errors that may occur in manual testing.

[0104] In an optional embodiment, inputting the control information into a pre-trained attention mechanism model, generating semantic associations and context information between controls based on the control information, and determining the execution order and execution parameters of the test actions include:

[0105] The attention mechanism model is built based on the Transformer architecture. The encoder in the attention mechanism model calculates the correlation and attention weight between controls through the self-attention mechanism, extracts high-level feature representations, performs nonlinear transformation on the high-level feature representations through a feedforward neural network, models the semantic association and context information between controls, and converts the control information into a low-dimensional dense vector representation.

[0106] At each time step, the decoder in the attention mechanism model receives the action embedding vector generated at the previous time step and the low-dimensional dense vector representation output by the encoder, calculates the correlation between the test action at the current time step and the historically generated action through the multi-head self-attention mechanism, performs nonlinear transformation on the features through the feedforward neural network, obtains the hidden state vector corresponding to the current time step, maps the hidden state vector to the test action space through linear transformation, obtains the probability distribution of the next test action based on the softmax function, selects the action corresponding to the maximum probability from the probability distribution, and adds it to the test action sequence as the output of the current time step until the length of the test action sequence reaches a preset maximum length threshold;

[0107] Based on the test action sequence, determining the execution order of the test actions;

[0108] For each test action, the execution parameters of the test action are determined based on the semantic association and context information of the control.

[0109] The attention mechanism model adopts the Transformer architecture, which includes two main components: encoder and decoder. Both the encoder and decoder are stacked by multiple identical sub-layers. Each sub-layer includes a self-attention mechanism and a feedforward neural network. Information transmission and interaction are achieved between the encoder and decoder through the attention mechanism.

[0110] The encoder receives control information as input, encodes the control information and extracts features; through the self-attention mechanism, the encoder calculates the correlation and attention weights between controls; the self-attention mechanism allows controls to pay attention to each other and captures the dependencies and interaction information between controls; the encoder calculates the attention weights between controls from different subspaces and angles through a multi-head self-attention mechanism, and extracts diversified feature representations; the feature representation obtained by the self-attention mechanism is nonlinearly transformed through a feedforward neural network to enhance the expressive power of the features; by stacking multiple encoder sublayers, the semantic associations and contextual information between controls are extracted layer by layer; finally, the encoder converts the control information into a low-dimensional dense vector representation as the input of the decoder.

[0111] The decoder receives the low-dimensional dense vector representation output by the encoder and the action embedding vector generated in the previous time step; at each time step, the decoder calculates the correlation between the test action in the current time step and the historically generated action through a multi-head self-attention mechanism, combining the influence and constraints of the historically generated action on the current action; the feature representation obtained by the multi-head self-attention mechanism is nonlinearly transformed through a feedforward neural network to obtain the hidden state vector corresponding to the current time step; the hidden state vector is mapped to the test action space through a linear transformation to obtain the probability distribution of the next test action in the current time step; the probability distribution is normalized based on the softmax function, and the action corresponding to the maximum probability is selected as the output of the current time step; the action output in the current time step is added to the test action sequence and used as the input of the next time step; the above steps are repeated until the generated test action sequence reaches the preset maximum length threshold.

[0112] According to the generated test action sequence, the execution order of the test actions is determined; the actions in the test action sequence are arranged in the order of generation to form an orderly execution queue; the test actions in the execution queue will be executed in sequence to simulate the actual test process.

[0113] For each test action in the execution queue, its execution parameters need to be determined; the semantic associations and contextual information of controls extracted by the attention mechanism model are used to infer the relationship and interaction method between the test action and the control; based on the properties, status and interaction requirements of the control, the specific execution parameters of the test action are determined, such as click position, input content, waiting time, etc.; the execution parameters are associated with the test action to form a complete test action description.

[0114] In this embodiment, the encoder receives control information and performs feature extraction and encoding through a self-attention mechanism, so that the model can deeply understand the semantic associations and contextual information between controls, and improve the abstraction and expression capabilities of control features; the multi-head self-attention mechanism in the encoder and decoder allows the model to calculate the attention weights between controls in different subspaces and angles, capture the complex dependencies and interaction information between controls, and enhance the model's understanding and modeling capabilities of control semantic associations; the decoder determines the execution order of test actions based on the generated action sequence to ensure that the test actions are executed in a reasonable logical order, simulate the order of operations in the actual test process, and improve the fidelity and accuracy of the test; based on the control semantic associations and contextual information extracted by the attention mechanism model, the model infers the specific execution parameters of each test action, and performs intelligent inference based on the current properties, status, and interaction requirements of the controls, so that the execution of test actions is more in line with actual application scenarios and operational requirements.

[0115] S103. After each of the test operations is completed, the interface feedback information of the software to be tested is obtained, and the text content in the interface feedback information is extracted based on the text recognition algorithm. The semantics corresponding to the text content is determined in combination with the pre-built semantic understanding model, and compared with the preset expected results to generate a test conclusion corresponding to the test operation.

[0116] In this embodiment, after each test operation is completed, the system obtains the interface feedback information of the software to be tested and captures the text content displayed on the software interface, usually through screenshots or interface capture technology, to obtain complete visual information; uses a text recognition algorithm to extract the text content in the interface feedback information, identifies and converts the text information in the image into processable text data, and provides a data basis for subsequent semantic understanding and comparison; combined with a pre-built semantic understanding model, the system performs semantic understanding on the extracted text content and converts the text content into a form with semantic expression capabilities so that the system can understand the meaning and information contained in the text.

[0117] In an optional embodiment, after each of the test operations is executed, interface feedback information of the software to be tested is obtained, text content in the interface feedback information is extracted based on a text recognition algorithm, and the semantics corresponding to the text content is determined in combination with a pre-built semantic understanding model, and compared with a preset expected result, and a test conclusion corresponding to the test operation is generated, including:

[0118] After the test operation is completed, the interface feedback information of the software to be tested is obtained, and the interface feedback information includes interface screenshots, control properties, log information and network request response data;

[0119] Based on the control attribute, determining a control response representation corresponding to a control state;

[0120] Based on the log information and the network request response data, extract the corresponding response text representation;

[0121] Extracting text content from the interface screenshot based on a text recognition algorithm, wherein the text recognition algorithm is constructed based on a convolutional neural network and a recurrent neural network, extracting local features of the image through a convolutional layer, modeling feature sequence dependencies through a recurrent layer, and converting the feature sequence dependencies into text content through a transcription layer;

[0122] Inputting the text content into a pre-built semantic understanding model for semantic analysis, wherein the semantic understanding model converts the text content into a semantic structured representation based on pre-defined semantic parsing rules;

[0123] Compare any one or more of the control response representation, the response text representation and the semantic structured representation with the preset expected result, obtain the comparison result and confidence level, determine the execution result, and if the execution result belongs to the preset failure type, backtrace the entire path of the test operation, combine the execution result, and construct failure information; if the execution result belongs to the preset success type, continue to execute the next test operation until all test operations are completed and determine the success information; generate a test conclusion based on the failure information and the success information.

[0124] After each test operation is completed, the interface feedback information of the software under test is obtained; the interface feedback information includes interface screenshots, control properties, log information and network request response data; the interface screenshots capture the visual information of the current interface, the control properties record the status and property values ​​of the controls, and the log information and network request response data provide the internal operation information and data interaction information of the software.

[0125] Based on the acquired control property information, the control status and property value are analyzed; according to the predefined control status mapping rules, the control status and property value are converted into the corresponding control response representation; the control response representation describes the response and change of the control to the test operation in a structured form.

[0126] Analyze the acquired log information and network request response data, and extract the text information contained therein; the log information may contain output information, error information, etc. during the software operation process, and the network request response data may contain the data content returned by the server; organize the extracted text information into a response text representation for subsequent semantic analysis and comparison.

[0127] Text recognition is performed on the obtained interface screenshots to extract the text content contained therein; the text recognition algorithm is built based on convolutional neural networks and recurrent neural networks; the convolutional layer is used to extract local features of the image and capture the visual features of the text area; the recurrent layer is used to model the dependencies between the extracted feature sequences and capture the contextual information of the text; the transcription layer converts the feature sequence dependencies into the final text content output.

[0128] The text content obtained by identifying the interface screenshot is input into a pre-built semantic understanding model for semantic analysis; the semantic understanding model converts the text content into a semantic structured representation based on predefined semantic parsing rules; the semantic parsing rules define the mapping relationship between text content and semantic concepts, such as keywords, entities, relationships, etc.; the semantic structured representation describes the semantic information of the text content in a structured form to facilitate subsequent comparison and analysis.

[0129] Compare any one or more of the control response representation, answer text representation and semantic structured representation with the preset expected results; the expected results represent the interface state, output content and semantic information expected after the test operation is executed; through comparison and analysis, calculate the similarity and confidence between the actual results and the expected results; based on the similarity and confidence, determine the execution result of the test operation to judge whether it meets expectations.

[0130] If the execution result belongs to the preset failure type, it means that the test operation did not achieve the expected effect; trace back the execution path of the test operation, and record the input, output, and status information of each step; combine the execution result and backtracking information to construct detailed failure information, including the cause of failure, location, context, etc.; the failure information is used for subsequent problem location and repair.

[0131] If the execution result belongs to the preset success type, it means that the test operation has achieved the expected effect; record the success information, including the input, output and status information of the test operation; continue to execute the next test operation until all test operations are completed; the success information of each test operation is accumulated to form a complete test execution record.

[0132] Based on the failure information and success information collected during the test process, the final test conclusion is generated; the test conclusion comprehensively evaluates the execution status and results of the test operations, including the number of failed test operations, the number of successful test operations, the overall test coverage, etc.; the test conclusion can also include problem summary, optimization suggestions, etc., to provide a reference for subsequent software improvements.

[0133] In this embodiment, after each test operation is completed, the system can obtain complete interface feedback information of the software to be tested, including interface screenshots, control properties, log information, network request response data and other data; based on the obtained control property information, the system analyzes and converts the state and property value of the control into a structured control response representation, which is helpful for subsequent understanding and analysis of the control behavior; the system analyzes the log information and network request response data, extracts text information, and organizes it into a response text representation, reflecting the output information and interactive feedback during the software operation process; through semantic analysis of the text content obtained by interface screenshot recognition, the text content is converted into a structured semantic representation, which is helpful for subsequent semantic analysis and comparison operations; if the test operation does not achieve the expected effect, the execution path of the test operation is traced back, and the input, output and status information of each step are recorded in detail. The failure information is used for problem location and repair, providing a key reference for software improvement.

[0134] Figure 2 FIG. 1 is a schematic diagram of the structure of a software automation testing system based on a visual attention mechanism according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0135] The first unit is used to obtain the initial interface of the software to be tested and perform preprocessing to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify control information by parsing the enhanced interface image, wherein the control information includes control type, control attribute and control position;

[0136] The second unit is used to input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters;

[0137] The third unit is used to obtain interface feedback information of the software to be tested after each of the test operations is completed, extract the text content in the interface feedback information based on the text recognition algorithm, determine the semantics corresponding to the text content in combination with a pre-built semantic understanding model, compare it with the preset expected results, and generate a test conclusion corresponding to the test operation.

[0138] According to a third aspect of the embodiments of the present invention,

[0139] An electronic device is provided, comprising:

[0140] processor;

[0141] a memory for storing processor-executable instructions;

[0142] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0143] A fourth aspect of the embodiments of the present invention is:

[0144] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0145] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A software automated testing method based on visual attention mechanism, characterized in that: include: Acquire the initial interface of the software to be tested and perform preprocessing to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify control information by parsing the enhanced interface image, wherein the control information includes control type, control attribute and control position; Input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters; After each of the test operations is completed, interface feedback information of the software to be tested is obtained, text content in the interface feedback information is extracted based on a text recognition algorithm, and the semantics corresponding to the text content is determined in combination with a pre-built semantic understanding model, and compared with the preset expected results to generate a test conclusion corresponding to the test operation; Inputting the enhanced interface image into an interface detection model, and identifying the control information by parsing the enhanced interface image includes: The interface detection model is built based on the semantic segmentation model. The receptive field of the enhanced interface image is expanded through the dilated convolution, the multi-scale context information of each pixel is captured, pixel-level classification is performed, a semantic label is assigned to each pixel, and boundary smoothing operation is performed through conditional random field post-processing to generate a semantic segmentation map. Based on the semantic segmentation map, instance segmentation is performed using a regional watershed algorithm to combine pixels belonging to the same control into independent control instances, thereby generating a control instance segmentation map; Based on the control instance segmentation graph, the control attributes corresponding to each control instance are extracted, the control nodes are constructed, the spatial position relationship between the control nodes is analyzed, the directed edges are established, and the control relationship graph is generated. The control relationship graph is input into a pre-trained graph neural network model, the control topology structure and spatial node features are extracted, and the control relationship graph attributes are generated; Based on the semantic segmentation map, the control instance segmentation map and the control relationship map attributes, construct a structured control information representation to generate control information; Based on the semantic segmentation map, instance segmentation is performed using a regional watershed algorithm to combine pixels belonging to the same control into independent control instances. Generating a control instance segmentation map includes: Based on a multi-scale Sobel operator, a horizontal gradient and a vertical gradient are respectively calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are respectively squared and added to obtain square roots to obtain a gradient amplitude map; Traverse each pixel point in the gradient amplitude map, and compare the gradient amplitude of the current pixel point with that of the surrounding 8 neighboring pixels based on the current pixel point. If the gradient amplitude of the current pixel point is the minimum value of the domain and is less than the preset gradient amplitude threshold, mark the current pixel point as a seed point; otherwise, skip the current pixel point; repeat the iterative comparison until all the pixels in the gradient amplitude map are traversed; The gradient amplitudes corresponding to the seed points are sorted from small to large to form a priority queue, and the first seed point is selected from the priority queue as the current seed point to start region expansion: The current seed point is used as the current processing area, and an empty expansion queue is created. Unmarked pixels are selected from the 8 neighboring pixels corresponding to the current seed point. If the gradient amplitude of the unmarked pixel is less than the preset gradient amplitude threshold, the unmarked pixel is used as an extended pixel and added to the expansion queue. At the same time, the current processing area is expanded and marked. The expansion is repeated for each extended pixel in the expansion queue, and the pixels are removed from the expansion queue until the expansion queue is empty. The expansion of the current processing area is completed, and the expansion completion area is generated; Process each seed point in the priority queue in turn until all seed points are processed and a series of expansion completion areas are determined; Calculate the geometric properties of each extended area, retain the valid area according to the preset area invalidation rules, generate a unique identification number for each valid area, extract the corresponding circumscribed rectangular bounding box, and output the control instance segmentation map; Based on the multi-scale Sobel operator, the horizontal gradient and the vertical gradient are calculated for each semantic category of the semantic segmentation map, and the horizontal gradient and the vertical gradient are squared and added to obtain the gradient amplitude map, including: The multi-scale Sobel operator has the following formula: ； in, s Indicates scale, G x s Representation scale s The horizontal gradient is G y s Representation scale s The vertical gradient is k represents the operator radius, m Indicates the horizontal offset. n Indicates the vertical offset, x Represents the horizontal coordinate of the pixel. y Represents the vertical coordinate of the pixel. I (·) represents the pixel value, S x ( m,n,s ) indicates scale s Sobel horizontal operator template, S y ( m,n,s ) indicates scale s Sobel vertical operator template.

2. The method according to claim 1, characterized in that Inputting the control information into a pre-trained attention mechanism model, generating semantic associations and context information between controls based on the control information, and determining the execution order and execution parameters of the test actions include: The attention mechanism model is built based on the Transformer architecture. The encoder in the attention mechanism model calculates the correlation and attention weight between controls through the self-attention mechanism, extracts high-level feature representations, performs nonlinear transformation on the high-level feature representations through a feedforward neural network, models the semantic association and context information between controls, and converts the control information into a low-dimensional dense vector representation. At each time step, the decoder in the attention mechanism model receives the action embedding vector generated at the previous time step and the low-dimensional dense vector representation output by the encoder, calculates the correlation between the test action at the current time step and the historically generated action through the multi-head self-attention mechanism, performs nonlinear transformation on the features through the feedforward neural network, obtains the hidden state vector corresponding to the current time step, maps the hidden state vector to the test action space through linear transformation, obtains the probability distribution of the next test action based on the softmax function, selects the action corresponding to the maximum probability from the probability distribution, and adds it to the test action sequence as the output of the current time step, until the length of the test action sequence reaches a preset maximum length threshold; Based on the test action sequence, determining the execution order of the test actions; For each test action, the execution parameters of the test action are determined based on the semantic association and context information of the control.

3. The method according to claim 1, characterized in that After each of the test operations is completed, interface feedback information of the software to be tested is obtained, text content in the interface feedback information is extracted based on a text recognition algorithm, and the semantics corresponding to the text content is determined in combination with a pre-built semantic understanding model, and compared with the preset expected results to generate a test conclusion corresponding to the test operation, including: After the test operation is completed, the interface feedback information of the software to be tested is obtained, and the interface feedback information includes interface screenshots, control properties, log information and network request response data; Based on the control attribute, determining a control response representation corresponding to a control state; Based on the log information and the network request response data, extract the corresponding response text representation; Extracting text content from the interface screenshot based on a text recognition algorithm, wherein the text recognition algorithm is constructed based on a convolutional neural network and a recurrent neural network, extracting local features of the image through a convolutional layer, modeling feature sequence dependencies through a recurrent layer, and converting the feature sequence dependencies into text content through a transcription layer; Inputting the text content into a pre-built semantic understanding model for semantic analysis, wherein the semantic understanding model converts the text content into a semantic structured representation based on pre-defined semantic parsing rules; Compare any one or more of the control response representation, the response text representation and the semantic structured representation with the preset expected result, obtain the comparison result and confidence level, determine the execution result, and if the execution result belongs to the preset failure type, backtrace the entire path of the test operation, combine the execution result, and construct failure information; if the execution result belongs to the preset success type, continue to execute the next test operation until all test operations are completed and determine the success information; generate a test conclusion based on the failure information and the success information.

4. A software automated testing system based on visual attention mechanism, used to implement the method described in any one of claims 1 to 3, characterized in that: include: The first unit is used to obtain the initial interface of the software to be tested and perform preprocessing to obtain an enhanced interface image, input the enhanced interface image into the interface detection model, and identify control information by parsing the enhanced interface image, wherein the control information includes control type, control attribute and control position; The second unit is used to input the control information into the pre-trained attention mechanism model, generate semantic associations and context information between controls based on the control information, determine the execution order and execution parameters of the test actions, generate a test action sequence based on the execution order, simulate manual test actions in sequence, and perform test operations on the controls according to the execution parameters; The third unit is used to obtain interface feedback information of the software to be tested after each of the test operations is completed, extract the text content in the interface feedback information based on the text recognition algorithm, determine the semantics corresponding to the text content in combination with a pre-built semantic understanding model, compare it with the preset expected results, and generate a test conclusion corresponding to the test operation.

5. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images