Linux desktop system testing method and device and storage medium

Abstract

The embodiment of the invention discloses a Linux desktop system testing method and device and a storage medium, and the method comprises the steps: inputting a collected to-be-tested image into a convolutional neural network model, and obtaining image elements and corresponding positions outputted by the convolutional neural network model; classifying the image elements, and respectively calculating the probability of the image elements to which the key pixel points of the image elements belong according to the classification result; according to the probability of the image element to which the key pixel point belongs, recombining the image element to obtain a recombined image element; setting a test label of each graphic element based on the combination relation of the recombined image elements; and determining a corresponding test script according to the test label, and testing by using the test script. The technical problems that the definition of the collected image is not high, the test accuracy is low due to the fact that the category cannot be determined, and an accurate test result cannot be obtained are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to a Linux desktop system testing method, device and storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technology and the rise of information technology, domestic Linux operating systems have made great progress, and the number of various applications supporting domestic Linux operating systems has also developed widely. In domestic Linux operating systems, software testing has become a vital link. With the rapid development of technology, the emergence of various testing tools and methods has provided strong support for software quality assurance.

[0003] In the prior art, various scripts are usually used for automated testing when testing domestic Linux operating systems and supported applications. In combination with various neural network models, various interfaces are effectively identified and corresponding scripts are executed for testing. In the process of implementing the present invention, the inventors found the following technical problems: Although the neural network model can effectively identify various test elements in the desktop system, it has high requirements for images and is prone to produce erroneous recognition results, which in turn affects the selection of test cases and makes it impossible to obtain accurate test results. In addition, since some application icons are not only displayed on the desktop, but also on the application level interface, and in the above two interfaces, the test scripts corresponding to the icons are also different. The neural network model cannot effectively identify the above two situations, and thus cannot obtain accurate test results. Summary of the invention

[0004] The embodiment of the present invention provides a Linux desktop system testing method, device and storage medium to solve the technical problem of low accuracy of Linux desktop system test results in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a Linux desktop system testing method, comprising:

[0006] Input the collected image to be tested into the convolutional neural network model, and obtain the image elements and corresponding positions output by the convolutional neural network model;

[0007] Classifying the image elements, and calculating the probability of the key pixels of the image elements belonging to the image elements according to the classification results;

[0008] Reorganize the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements;

[0009] Setting a test label for each graphic element based on the combination relationship of the reorganized image elements;

[0010] A corresponding test script is determined according to the test tag, and the test is performed using the test script.

[0011] In a second aspect, an embodiment of the present invention further provides a Linux desktop system testing device, comprising:

[0012] An input module is used to input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model;

[0013] A calculation module, used to classify the image elements and calculate the probability of the key pixel points of the image elements belonging to the image elements according to the classification results;

[0014] A reorganization module, used for reorganizing the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements;

[0015] A setting module, used for setting a test label of each graphic element based on the combination relationship of the reorganized image elements;

[0016] The test module is used to determine the corresponding test script according to the test tag and perform the test using the test script.

[0017] In a third aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the Linux desktop system testing method provided in the above embodiment.

[0018] The Linux desktop system testing method, device and storage medium provided by the embodiment of the present invention input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model; classify the image elements, and calculate the probability of the key pixel points of the image elements belonging to the image elements according to the classification results; reorganize the image elements according to the probability of the key pixel points belonging to the image elements to obtain the reorganized image elements; set the test label of each graphic element based on the combination relationship of the reorganized image elements; determine the corresponding test script according to the test label, and use the test script to perform the test. Through the output result of the convolutional neural network, the corresponding probability is determined respectively by the classification to which the output image elements belong, and the interference of noise is eliminated. The image elements are reorganized by determining the key points to obtain clear and accurate image elements, and the desktop application icons or application interfaces are classified based on the combination relationship between the reorganized image elements, and the appropriate script is determined according to the classification results and the content of the corresponding image elements for testing. The technical problems of low clarity of the collected images, low test accuracy caused by the inability to determine the categories, and inability to obtain accurate test results are effectively overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0020] Figure 1 It is a flowchart of a Linux desktop system testing method provided by Embodiment 1 of the present invention;

[0021] Figure 2 It is a flowchart of a Linux desktop system testing method provided by Embodiment 2 of the present invention;

[0022] Figure 3 It is a structural diagram of a Linux desktop system testing device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] Embodiment 1

[0025] Figure 1: is a flowchart of a Linux desktop system testing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of accurately and effectively testing a Linux desktop system. The method can be executed by a Linux desktop system testing device, and specifically includes the following steps:

[0026] Step 110: input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model.

[0027] In this embodiment, the image to be tested can be an interface image related to the Linux desktop system that needs to be tested. The convolutional neural network model can be a VGG-19 model. The VGG model replaces a large convolution kernel by stacking multiple 3x3 convolution kernels, which can reduce the number of parameters required. By expanding the number of channels and reducing the height and width, the model is deeper and wider while the increase in the amount of calculation is slowed down. This design helps to improve the performance of the model while reducing the computational burden. It has the advantages of fast computing speed and strong generalization ability. The VGGNet convolutional network architecture, which is good at extracting high-level features of images, is used as the backbone. A 19-layer deep convolutional neural network was constructed, which not only ensures performance, but also improves the generalization ability of the model.

[0028] The trained VGG-19 convolutional neural network model can output image elements and the specific position coordinates of each image element in the original image to be tested. The image element can be an icon or a text module.

[0029] Exemplarily, before inputting the collected test image into the convolutional neural network model, the method may further include the following steps: scaling the image in the data set in equal proportion, and cropping or padding it into an image of a preset size; calculating the attribute p of the image of the preset size = [v 1 , v 2 , v 3 】

ε 1 β 1 , ε 2 β 2 , ε 3 β 3

[0030] Step 120: classify the image elements, and calculate the probability of the key pixels of the image elements belonging to the image elements according to the classification results.

[0031] Exemplarily, a rough classification result can be obtained using the VGG-19 convolutional neural network. The classification result can be an application icon class or an application interface class. Exemplarily, since there is no mature way to apply the existing model to desktop system visual recognition in the prior art, and there is no available data set for effective training. Therefore, in the embodiment of the present invention, image elements can be annotated by manual annotation during the training process, and the corresponding types can be marked. Therefore, in the embodiment of the present invention, image elements can be annotated by manual annotation during the training process, and the corresponding types can be marked. And the output result of the last convolution layer of the application icon class or the application interface class can be used to determine the probability of the image element to which the pixel point in each image element area belongs.

[0032] Furthermore, the probability of the key pixel points of the image element belonging to the image element is calculated according to the classification results, which may include: when it is an application icon class, the key pixel points of the key point area are obtained by using the rectangular border of the image element; when it is an application interface class, the character center probability is obtained by using the character-level bounding box, the probability of the pixel being the picture area is determined by using the image element border, and the key pixel points are determined according to the character center probability and the probability of the picture area. The above method can be used to determine the key pixel points of each image element for different categories.

[0033] In addition, in this embodiment, in order to further enhance the accuracy of image reconstruction, key pixels can be further optimized by other means. Exemplarily, it can include: obtaining the neighborhood pixels of the key pixel, calculating the color, brightness and texture features of the neighborhood pixels and the key pixel; judging whether the color, brightness and texture features of the neighborhood pixels are consistent with those of the key pixel, and deleting the key pixel when they are not consistent. For the processing of icon images, determining the valid pixels of the icon is a crucial step. In this process, valid pixels are clearly marked as positive, and these pixels constitute the main part of the icon, carrying key information for visual recognition and function guidance. The core principle of analyzing such images in this embodiment is to examine the continuity between pixels, which not only concerns the overall structure of the image, but also directly affects the accuracy and efficiency of icon recognition.

[0034] Specifically, if the edge pixels of the entire image present a continuous distribution state, that is, the edge and the pixels inside it logically form a complete, unbroken closed area, then these pixels are considered positive pixels. This continuity is an important basis for judging the integrity and effectiveness of the icon. On the contrary, if the edge pixels are discontinuous, with breaks or gaps, then these pixels and the pixels inside them will be considered negative, because they may represent noise, defects or non-icon parts in the image. Further, it is necessary to examine the continuity between a single pixel and its eight surrounding neighboring pixels. In image processing, these eight neighboring pixels are usually referred to as the "neighborhood" of the pixel. If a pixel is continuous with all the pixels in its neighborhood, that is, they show consistent or similar changes in features such as color, brightness or texture, then the connection between these pixels is considered positive. This positive connection is the basis for the internal consistency and integrity of the icon. On the contrary, if a pixel and the pixels in its neighborhood have significant differences or breaks in features, then the connection between them is considered negative, which may mean that there are features such as edges, noise or texture changes in the image.

[0035] In addition, non-key pixels can be further excluded in the following way: when there is consistency, the shape and size of the graphic composed of continuous key pixels are calculated; when the shape of the graphic is a square and the size is less than a preset size threshold, the continuous key pixels are deleted. Optionally, it is also necessary to further filter valid pixels in combination with specific judgment rules. For example, when a group of connected pixels is detected, if the shape they form is a square, and the side length of the square is less than 30 pixels, or its area is less than 900 pixels squared, then these pixels may be judged as interference points. This is because square areas that are too small often do not have the visual characteristics and functionality that an icon should have. They are more likely to be noise, defects or insignificant details in the image. By setting such judgment rules, interference can be removed more effectively and key information in the image can be retained, thereby improving the accuracy and efficiency of icon recognition.

[0036] Step 130 , reorganize the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements.

[0037] According to the probability of the key pixel points determined in the above steps, the key pixel points are determined by a probability threshold, and the image elements are reorganized according to the specific positions of the key pixel points to obtain the reorganized image elements.

[0038] Step 140: setting a test label for each graphic element based on the combination relationship of the reorganized image elements.

[0039] Exemplarily, it may include: determining the classification of the input image according to the combination relationship of the reorganized image elements; setting the test label corresponding to the test script according to the attributes of the reorganized image elements and the classification. If the image is mainly composed of one or more continuous and complete icons, and there are no or very few non-icon elements, the label is the application icon class. If the image contains multiple elements, including at least one icon and other identifiable objects, it can be classified as an application interface class. The image element attribute can be the function corresponding to the icon.

[0040] Step 150, determining a corresponding test script according to the test tag, and performing a test using the test script.

[0041] Exemplarily, it may include: when classified as an icon image, executing a full script corresponding to the image element attribute; when classified as an interface image, performing an OCR comparison method, respectively executing the test scripts corresponding to the image element attribute and the corresponding OCR text recognition, and comparing whether the test images are consistent, and outputting the test results. Using the above method, a suitable test script can be selected and a suitable test result can be given to achieve accurate and effective testing of the Linux desktop system.

[0042] This embodiment inputs the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model; classifies the image elements, and calculates the probability of the key pixel points of the image elements belonging to the image elements according to the classification results; reorganizes the image elements according to the probability of the key pixel points belonging to the image elements to obtain the reorganized image elements; sets the test label of each graphic element based on the combination relationship of the reorganized image elements; determines the corresponding test script according to the test label, and uses the test script for testing. Through the output results of the convolutional neural network, the corresponding probabilities are determined respectively by the classification to which the output image elements belong, and the interference of noise is eliminated. The image elements are reorganized by determining the key points to obtain clear and accurate image elements, and the desktop application icons or application interfaces are classified based on the combination relationship between the reorganized image elements, and the appropriate script is determined for testing according to the classification results and the content of the corresponding image elements. The technical problems of low clarity of the collected images, low test accuracy caused by the inability to determine the categories, and inability to obtain accurate test results are effectively overcome.

[0043] Embodiment 2

[0044] Figure 2It is a flow chart of the Linux desktop system testing method provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment to classify the image elements. The specific optimization is as follows: using the common attention module provided in the convolutional neural network model, by analyzing the visual features, position information and association with surrounding objects of the icon, accurately locate and pay attention to the icon information that may exist in the upper right corner area, so as to achieve common attention to the upper right corner area in the input image; using the contrastive attention module provided in the convolutional neural network model, by analyzing the unique features, position and relative relationship of each object with the icon, to distinguish different objects and assign them corresponding semantic labels, so as to focus on the remaining objects in the interface image and perform non-shared semantic classification on the remaining objects; according to the results output in the convolutional neural network model, it is determined whether the image is composed of one or more continuous and complete icon elements, and the number of non-icon elements is less than the preset element number threshold, and it is determined to be an application icon class; it is determined whether the image includes at least one icon element and other identifiable objects, and the elements are identified by the common attention and contrastive attention mechanisms respectively, and it is determined to be an application interface class.

[0045] See also Figure 2 , the Linux desktop system testing method comprises:

[0046] Step 210: input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model.

[0047] Step 220, using the co-attention module provided in the convolutional neural network model, by analyzing the visual features, position information and association with surrounding objects of the icon, to accurately locate and focus on the icon information that may exist in the upper right corner area, so as to achieve co-attention to the upper right corner area in the input image.

[0048] In this embodiment, the co-attention mechanism comes from the icon in the upper right corner of the object area of ​​common attention. The core of this mechanism is to jointly pay attention to the icon in the upper right corner of the object area in the interface image. Since the icon is usually located in the upper left corner of the interface, but in some cases it may shift to the upper right corner due to layout adjustment, the co-attention mechanism we designed can flexibly capture this feature. It accurately locates and focuses on these key icons by analyzing the visual features, location information, and association with surrounding objects of the icons. This mechanism not only improves the accuracy of icon recognition, but also helps to understand the layout and structure of the entire interface.

[0049] Step 230, utilizing the contrastive attention module provided in the convolutional neural network model, by analyzing the unique features, position, and relative relationship of each object with the icon, to distinguish different objects and assign them corresponding semantic labels, so as to focus on the remaining objects in the interface image and perform non-shared semantic classification on the remaining objects.

[0050] The contrastive attention mechanism focuses on non-shared semantic classification of the remaining objects. Unlike the shared attention mechanism, the contrastive attention mechanism focuses on the remaining objects in the interface image and aims to perform non-shared semantic classification on these objects. It distinguishes different objects and assigns them corresponding semantic labels by analyzing the unique features, position, and relative relationship of each object to the icon. This mechanism can effectively handle redundant information and noise in the interface and improve the accuracy and robustness of semantic classification.

[0051] By using the co-attention module and the contrastive attention module, we can more comprehensively understand and process the interface images. The co-attention mechanism ensures that key icons can be accurately captured and identified, while the contrastive attention mechanism enables the effective classification and processing of other objects in the interface.

[0052] Step 240, based on the output results of the convolutional neural network model, determine whether the image is composed of one or more continuous and complete icon elements, and the number of non-icon elements is less than a preset element quantity threshold, and then determine it as an application icon class.

[0053] According to the image characteristics of the application icon, that is, the image is composed of one or more continuous and complete icon elements, including a small amount of text, or corresponding wallpaper graphics on the desktop, etc., it is determined to be an application icon class.

[0054] Step 250, determine whether the image includes at least one icon element and other identifiable objects, and the elements are identified through the common attention and contrast attention mechanisms respectively, and then determine it as an application interface class.

[0055] The application interface class has not only icon factors but also various operation interfaces. Therefore, the application interface can be identified according to its characteristics and whether it is an application interface class can be determined based on the identified path.

[0056] Step 260 , calculating the probability of the key pixel points of the image elements belonging to the image elements according to the classification results.

[0057] Step 270: reorganize the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements.

[0058] Step 280: setting a test label for each graphic element based on the combination relationship of the reorganized image elements.

[0059] Step 290, determining a corresponding test script according to the test tag, and performing testing using the test script.

[0060] This embodiment classifies the image elements, and is specifically optimized as follows: utilizing the common attention module provided in the convolutional neural network model to accurately locate and focus on the icon information that may exist in the upper right corner area by analyzing the visual features, position information and association with surrounding objects of the icon, so as to achieve common attention to the upper right corner area in the input image; utilizing the contrastive attention module provided in the convolutional neural network model to distinguish different objects and assign them corresponding semantic labels by analyzing the unique features, position and relative relationship of each object with the icon, so as to focus on the remaining objects in the interface image and perform non-shared semantic classification on the remaining objects; judging whether the image is composed of one or more continuous and complete icon elements according to the results output in the convolutional neural network model, and the number of non-icon elements is less than a preset element quantity threshold, and then determining it as an application icon class; judging whether the image includes at least one icon element and other identifiable objects, and the elements are identified by the common attention and contrastive attention mechanisms respectively, then determining it as an application interface class. By optimizing the VGG model, the contrast attention module and the co-attention module are set according to the characteristics of the application interface and application icon images respectively. This can effectively extract the focused features and accurately classify the extracted features and the obtained results, which can effectively avoid the problem of low classification prediction accuracy of the original VGG19 model.

[0061] In a preferred implementation of this embodiment, the image elements can also be classified and further optimized as follows: the number of edges, edge density and / or edge direction of each image element are calculated respectively; and the image element is determined to belong to the application icon class or the application interface class according to the number of edges, edge density and / or edge direction of each image element. Since the application icon image usually contains concise graphics and bright colors, while the application interface image may contain more text, buttons and complex layouts. Therefore, the edge detection operator can be applied to process the grayscale image to obtain the edge image. Then the edge image is analyzed to extract the edge features of the image. Then, a judgment is made based on the edge features: in terms of the number of edge pixels, the application icon image usually has a relatively concise edge and a relatively small number of edges; the application interface image: the interface image may contain multiple elements and components, so the number of edges is relatively large; in terms of edge density, the edge density of the application icon image is usually low, because the icon design focuses on the overall sense and simplicity. The edge density in the application interface image is usually high, because the interface needs to contain multiple elements and details. In terms of edge direction, the application icon image: the edge direction often has a certain regularity and consistency. The edge direction in the application interface image may be more diverse and complex. Optionally, the gradient magnitude and direction can be calculated as follows:

[0062]

[0063] , where N(x, y) represents the gradient strength of the pixel, G represents the Sobel operator, and G y and G x Represents the gradient values ​​in the horizontal and vertical directions respectively.

[0064] θ(x, y) represents the gradient direction of the pixel.

[0065] The non-maximum suppression of the gradient is calculated as:

[0066]

[0067] Among them, N L is the low threshold of gradient strength, φ x and φ y is the offset value in the gradient direction.

[0068] The edge region score is calculated as:

[0069]

[0070] Among them, N H is the high threshold of gradient strength, where strong represents strong edge, weak represents weak edge, and non-region represents non-edge.

[0071] Through the above-mentioned method, the amount of calculation for classifying the image elements can be effectively simplified, which is convenient for use in occasions with strict requirements on the operation time.

[0072] Embodiment 3

[0073] Figure 3 is a schematic diagram of the structure of the Linux desktop system testing device provided by the third embodiment of the present invention, see Figure 3 , the Linux desktop system testing device comprises:

[0074] An input module 310 is used to input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model;

[0075] A calculation module 320, used to classify the image elements and calculate the probability of the key pixels of the image elements belonging to the image elements according to the classification results;

[0076] A reorganization module 330, configured to reorganize the image elements according to the probability of the key pixel points belonging to the image elements, to obtain reorganized image elements;

[0077] A setting module 340, configured to set a test label of each graphic element based on the combination relationship of the reorganized image elements;

[0078] The testing module 350 is used to determine the corresponding test script according to the test tag and perform testing using the test script.

[0079] The Linux desktop system testing device provided in this embodiment inputs the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model; classifies the image elements, and calculates the probability of the key pixel points of the image elements belonging to the image elements according to the classification results; reorganizes the image elements according to the probability of the key pixel points belonging to the image elements to obtain the reorganized image elements; sets the test label of each graphic element based on the combination relationship of the reorganized image elements; determines the corresponding test script according to the test label, and uses the test script for testing. Through the output results of the convolutional neural network, the corresponding probabilities are determined respectively by the classification to which the output image elements belong, and the interference of noise is eliminated. The image elements are reorganized by determining the key points to obtain clear and accurate image elements, and the desktop application icons or application interfaces are classified based on the combination relationship between the reorganized image elements, and the appropriate script is determined for testing according to the classification results and the content of the corresponding image elements. It effectively overcomes the technical problems that the clarity of the collected images is not high, the test accuracy rate generated by the inability to determine the categories is low, and accurate test results cannot be obtained.

[0080] Based on the above embodiments, the calculation module includes:

[0081] A separate calculation unit, for separately calculating the edge quantity, edge density and / or edge direction of each image element;

[0082] The determination unit is used to determine whether the image element belongs to the application icon class or the application interface class according to the edge quantity, edge density and / or edge direction of each image element.

[0083] Based on the above embodiments, the calculation module includes:

[0084] The path search module is used to find the path where the container stores files on the host machine according to the container configuration file;

[0085] The control module is used to control the directory corresponding to the path using the security mark.

[0086] Based on the above embodiments, the path finding module includes:

[0087] A first key pixel point determination unit, used for obtaining key pixel points of a key point area by using a rectangular border of the image element when the image element is an application icon;

[0088] The second key pixel point determination unit is used to obtain the character center probability using the character-level bounding box when it is an application interface type, and to determine the probability that the pixel is a picture area using the image element border, and to determine the key pixel point according to the character center probability and the picture area probability respectively.

[0089] Based on the above embodiments, the device further includes:

[0090] An acquisition module is used to acquire neighborhood pixels of the key pixel and calculate the color, brightness and texture features of the neighborhood pixels and the key pixel;

[0091] The first deletion module is used to determine whether the color, brightness and texture features of the neighborhood pixel points are consistent with those of the key pixel points, and if they are not consistent, delete the key pixel points.

[0092] Based on the above embodiments, the device further includes:

[0093] A shape and size calculation module, used to calculate the shape and size of a graphic composed of continuous key pixels when there is consistency;

[0094] The second deleting module is used to delete the continuous key pixel points when the shape of the graphic is a square and the size is smaller than a preset size threshold.

[0095] Based on the above embodiments, the calculation module includes:

[0096] A first analysis unit is used to use a common attention module provided in the convolutional neural network model to accurately locate and focus on the icon information that may exist in the upper right corner area by analyzing the visual features, position information and association with surrounding objects of the icon, so as to realize common attention to the upper right corner area in the input image;

[0097] A second analysis unit is used to distinguish different objects and assign corresponding semantic labels to them by using a contrastive attention module provided in the convolutional neural network model by analyzing the unique features, position and relative relationship of each object with the icon, so as to focus on the remaining objects in the interface image and perform non-shared semantic classification on the remaining objects;

[0098] An application icon class determination unit, used to determine whether the image is composed of one or more continuous and complete icon elements and has less non-icon elements than a preset element quantity threshold according to the output result of the convolutional neural network model, and then determine it as an application icon class;

[0099] The application interface class determination unit is used to determine whether the image includes at least one icon element and other identifiable objects, and if the elements are identified through the co-attention and contrast attention mechanisms respectively, then it is determined to be an application interface class.

[0100] Based on the above embodiments, the device further includes:

[0101] The scaling module is used to scale the images in the dataset in equal proportions and crop or fill them into images of preset sizes;

[0102] Attribute calculation module, used to calculate the attributes of the image of preset size

[0103] p=

v 1 , v 2 , v 3

ε 1 β 1 , ε 2 β 2 , ε 3 β 3

[0104] Based on the above embodiments, the setting module includes:

[0105] A classification determination unit, used to determine the classification of the input image according to the combination relationship of the reorganized image elements;

[0106] a label determination unit, configured to set a test label corresponding to a test script according to the attributes of the reorganized image elements and the classification;

[0107] Accordingly, the test module includes:

[0108] A first execution unit, configured to execute a full script corresponding to the image element attribute when the image element is classified as an icon image type;

[0109] The second execution unit is used to perform an OCR comparison method on the images classified as interface images, respectively execute the test scripts corresponding to the image element attributes and the corresponding OCR text recognition, compare whether the test images are consistent, and output the test results.

[0110] The Linux desktop system testing device provided in the embodiment of the present invention can execute the Linux desktop system testing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment 4

[0112] Embodiment 4 of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute any of the Linux desktop system testing methods provided in the above embodiments.

[0113] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0115] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0116] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0117] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A Linux desktop system testing method, characterized in that: include: Input the collected image to be tested into the convolutional neural network model, and obtain the image elements and corresponding positions output by the convolutional neural network model; Classifying the image elements, and calculating the probability of the key pixels of the image elements belonging to the image elements according to the classification results; Reorganize the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements; Setting a test label for each graphic element based on the combination relationship of the reorganized image elements; A corresponding test script is determined according to the test tag, and the test is performed using the test script.

2. The method according to claim 1, characterized in that The classifying the image elements comprises: Calculate the number of edges, edge density and / or edge direction of each image element respectively; It is determined whether the image element belongs to the application icon class or the application interface class according to the edge quantity, edge density and / or edge direction of each image element.

3. The method according to claim 1, characterized in that The step of respectively calculating the probability of the key pixels of the image elements belonging to the image elements according to the classification results includes: When the icon class is applied, the key pixel points of the key point area are obtained by using the rectangular border of the image element; In the case of the application interface class, the character center probability is obtained using the character-level bounding box, the probability of the pixel being a picture area is determined using the image element border, and the key pixel points are determined based on the character center probability and the picture area probability.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining neighboring pixels of the key pixel, and calculating color, brightness and texture features of the neighboring pixels and the key pixel; It is determined whether the color, brightness and texture features of the neighborhood pixel points are consistent with those of the key pixel points. If they are not consistent, the key pixel points are deleted.

5. The method according to claim 4, characterized in that The method further comprises: When there is consistency, calculate the shape and size of the graphics composed of continuous key pixels; When the shape of the graphic is a square and the size is smaller than a preset size threshold, the continuous key pixel points are deleted.

6. The method according to claim 1, characterized in that The classifying the image elements comprises: The co-attention module in the convolutional neural network model is used to accurately locate and focus on the icon information that may exist in the upper right corner by analyzing the icon's visual features, location information, and association with surrounding objects, so as to achieve co-attention to the upper right corner of the input image. The contrastive attention module in the convolutional neural network model is used to distinguish different objects and assign them corresponding semantic labels by analyzing the unique features, position and relative relationship of each object to the icon, so as to focus on the remaining objects in the interface image and perform non-shared semantic classification on the remaining objects. According to the output results of the convolutional neural network model, it is determined whether the image is composed of one or more continuous and complete icon elements, and the number of non-icon elements is less than a preset element number threshold, and then it is determined to be an application icon class; It is determined whether the image includes at least one icon element and other identifiable objects, and the elements are identified through the co-attention and contrastive attention mechanisms respectively, and then it is determined to be an application interface class.

7. The method according to claim 1, characterized in that Before inputting the collected image to be tested into the convolutional neural network model, the method further includes: Scale the images in the dataset proportionally and crop or fill them to preset size images; Calculate the properties of the image of the preset size p = [v1, v2, v3] [ε1β1, ε2β2, ε3β3], where v represents the feature vector, ε is a random vector, β is the image channel pixel value, and the attributes of all images of preset sizes are taken as the images to be tested.

8. The method according to claim 1, characterized in that The step of setting a test label for each graphic element based on the combination relationship of the reorganized image elements includes: Determine the classification of the input image according to the combination relationship of the reorganized image elements; Setting a test tag corresponding to the test script according to the attributes of the reorganized image elements and the classification; Accordingly, determining a corresponding test script according to the test tag and performing a test using the test script includes: When it is classified as an icon image, a full script corresponding to the image element attributes is executed; In the method of performing OCR comparison on the image classified as the interface image class, the test scripts corresponding to the image element attributes and the corresponding OCR text recognition are executed respectively, and the test images are compared for consistency, and the test results are output.

9. A Linux desktop system testing device, characterized in that: include: An input module is used to input the collected image to be tested into the convolutional neural network model to obtain the image elements and corresponding positions output by the convolutional neural network model; A calculation module, used to classify the image elements and calculate the probability of the key pixel points of the image elements belonging to the image elements according to the classification results; A reorganization module, used for reorganizing the image elements according to the probability of the key pixel points belonging to the image elements to obtain reorganized image elements; A setting module, used for setting a test label of each graphic element based on the combination relationship of the reorganized image elements; The test module is used to determine the corresponding test script according to the test tag and perform the test using the test script.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the Linux desktop system testing method as described in any one of claims 1-8 when executed by a computer processor.

Images