Automatic testing method and device, computer equipment and readable storage medium

Abstract

The invention relates to an automatic testing method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring an image sample, and storing a target area in the image sample; extracting image features and text features in the target area; fusing the image features and the text features through feature splicing and attention mechanism weighting to obtain fused features, and generating a data set including the image features, the text features and the fused features; training a classification model through the data set, wherein the classification model is used for classifying input data; taking the image features and the text features in the data set as the input data, classifying the input data through the trained classification model, and outputting a classification result; and outputting the classification result as structured data, wherein the structured data comprises a classification label, a confidence score and associated information of the image and the text. The method can automatically manage the test cases, reduce manual intervention and improve software test efficiency.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to an automated testing method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] As software development becomes increasingly complex, automated testing becomes increasingly important. Currently, most mainstream automated testing frameworks rely on simple rules or manual execution, limiting their applicable scenarios. Furthermore, they lack intelligent and automated support for test case classification, hindering timely and comprehensive analysis of test results and generation of detailed reports.

[0003] Therefore, how to achieve efficient and comprehensive automated testing in multiple scenarios has become an urgent problem that needs to be solved. Summary of the Invention

[0004] Based on this, it is necessary to provide an automated testing method, device, computer equipment and readable storage medium to address the above technical problems, which can automatically manage the execution status and results of test cases, reduce manual intervention, and significantly improve software quality and testing efficiency.

[0005] In a first aspect, the present application provides an automated testing method for jointly analyzing images and texts of the operating status of automated testing equipment in industrial automated testing, comprising:

[0006] Obtain an image sample, save a target area in the image sample; extract image features and text features in the target area;

[0007] The image feature and the text feature are fused by feature concatenation and weighted attention mechanism to obtain fused features, and a dataset including the image feature, the text feature and the fused features is generated; a classification model is trained using the dataset, and the classification model is used to classify the input data;

[0008] The image features and the text features in the data set are used as the input data, the input data is classified by the trained classification model, and a classification result is output;

[0009] The classification result is output as structured data, which includes classification labels, confidence scores, and association information between images and text.

[0010] In combination with the first aspect, the automated testing equipment includes a visual sensor, which obtains image samples and saves the target area in the image samples, including: collecting the image samples based on the visual sensor, obtaining the target area of the image sample through a preprocessing operation, and saving the target area, and the preprocessing operation includes image segmentation based on an edge detection algorithm and image rotation based on a Hough transform.

[0011] In combination with the first aspect, the extraction of image features and text features in the target area includes: identifying text data in the target area, marking the remaining area in the target area as image data, extracting the image features from the image data, and extracting the text features from the text data.

[0012] In combination with the first aspect, in some embodiments of the first aspect, the image features are extracted from the image data, including using a convolutional neural network CNN or a pre-trained model to extract the image features from the image data, and the pre-trained model includes a visual geometry group network VGG and a residual network ResNet; the text features are extracted from the text data, including using word segmentation, removing stop words, and vectorization, and combining a domain dictionary to enhance the semantic relevance of text vectorization, to extract the text features from the text data.

[0013] In combination with the first aspect, in some embodiments of the first aspect, the image feature and the text feature are fused by feature concatenation and weighted attention mechanism to obtain a fused feature, including:

[0014] Based on the splicing and sharing of the image features and the text features, dynamic weight distribution of the image and text features is achieved through the cross-modal attention mechanism in the deep learning framework to generate fusion features. The deep learning framework includes TensorFlow and PyTorch.

[0015] In combination with the first aspect, the classification model includes a convolutional neural network (CNN) based on the image feature extraction and a support vector machine (SVM) based on the text feature classification. The classification result includes the test case type, test case execution status and test case priority.

[0016] In combination with the first aspect, after taking the image features and the text features in the data set as the input data, classifying the input data through the trained classification model, and outputting the classification results, the method also includes: generating enhanced training data based on the classification results, optimizing the classification performance and generalization ability of the classification model, including the robustness of the classification model.

[0017] It should be noted that, in the absence of conflict, the features of the various embodiments of the first aspect can be combined with each other, and any combination of features in different embodiments is also within the scope of protection of this application. That is to say, the multiple embodiments described above can also be arbitrarily combined according to actual needs.

[0018] In a second aspect, an automated testing device is provided, comprising: a data acquisition module for acquiring an image sample and saving a target area in the image sample;

[0019] A feature extraction and fusion module is used to extract image features and text features in the target area, and is also used to fuse the image features and the text features through feature splicing and attention mechanism weighting to obtain fused features, and generate a data set including the image features, the text features and the fused features;

[0020] A model training module is used to train a classification model using the data set, and the classification model is used to classify the input data;

[0021] An intelligent classification module is used to use the image features and the text features in the data set as the input data, classify the input data using the trained classification model, and output a classification result;

[0022] The result display module is used to output the classification result as structured data, which includes a classification label, a confidence score, and association information between the image and the text.

[0023] In a third aspect, the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0024] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0025] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the above method.

[0026] The automated testing method, apparatus, computer device, and readable storage medium described above obtain an automated test image as a sample, extract image and text features from the sample, and fuse the image and text features to obtain a fused feature. A classification model is trained based on the fused feature, and the image and text features are input into the trained classification model to output a classification result. Based on the classification result, structured data is output, including a classification label, a confidence score, and information related to the image and text. This method can classify test cases and automatically manage their execution status and results, reducing manual intervention and significantly improving software quality and testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An application environment diagram of an automated testing method provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of an automated testing method provided in an embodiment of the present application;

[0029] Figure 3 A flow chart of a feature extraction and fusion method provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a data structure for feature extraction and fusion provided in an embodiment of the present application;

[0031] Figure 5 A flowchart of a data classification method provided in an embodiment of the present application;

[0032] Figure 6 A structural block diagram of an automated testing device provided in an embodiment of the present application;

[0033] Figure 7 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0034] Figure 8 This is a diagram of the internal structure of a readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0036] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The technical solutions in the embodiments of the present application will be clearly and comprehensively described below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0038] Currently, mainstream automated testing frameworks are executed based on simple rules or manual specifications. They lack corresponding automated testing support for the classification operations of test cases, and are unable to timely and comprehensively analyze test results and generate detailed reports.

[0039] In response to the above technical problems, the present application provides an automated testing method, apparatus, computer equipment, and readable storage medium. The method includes obtaining an image for automated testing as a sample, extracting image features and text features from the sample, and fusing the image features and text features to obtain fused features; generating a data set using the image features, text features, and fused features; training a classification model based on the data set so that the classification model can classify the input data; then using the image features and text features in the data set as input data, the classification model outputs a classification result; and outputting structured data including a classification label, a confidence score, and information related to the image and text based on the classification result. This method can automatically manage the execution status and results of test cases, reduce manual intervention, and significantly improve software quality and testing efficiency.

[0040] The following three embodiments describe the automated testing method, apparatus, computer equipment and storage medium provided in this application. Example 1 describes the application environment of the automated testing method, Example 2 describes the process of the automated testing method, and Example 3 describes the apparatus, computer equipment and readable storage medium of the automated testing method.

[0041] Example 1

[0042] The automated testing method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Figure 1 : The figure shows an automated testing device 100, an image sample 200, a text sample 201, and an image-text mixed sample 202 acquired by the automated testing device 100. The image sample 200, the text sample 201, and the image-text mixed sample 202 can all be stored in the automated testing device 100 in the form of pictures. The automated testing device 100 can pre-store the image sample 200, the text sample 201, and the image-text mixed sample 202 in a storage unit, or acquire them from the environment through a sensor. For example, the automated testing device 100 captures a picture through a visual sensor. An image without text is an image sample 200, a picture containing only text is a text sample 201, and a picture containing both an image and text is an image-text mixed sample 202.

[0043] In the embodiment of the present application, the automated testing device 100 can perform classification tests on acquired image samples 200, text samples 201, and mixed image and text samples 202, and output classification test results via structured data, including but not limited to classification labels, confidence scores, and image-text association information. Furthermore, the automated testing device 100 supports operations including real-time classification, data management, and analysis of acquired image samples 200, text samples 201, and mixed image and text samples 202.

[0044] In the embodiment of the present application, the automated test device 100 may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0045] Example 2

[0046] like Figure 2 As shown, the embodiment of the present application provides an automated testing method, which is applied to Figure 1 The automated test device 100 in FIG. 1 is used as an example for explanation. It is understandable that the computer device may include the automated test device 100. The method includes the following steps:

[0047] S101. Collect image samples.

[0048] In the embodiment of the present application, the automated testing device 100 obtains an image sample and stores the image sample in the automated testing device 100 .

[0049] In some embodiments, the automated testing equipment 100 may have image samples pre-stored in a storage unit, or may capture the original image of the test object through a sensor (such as a visual sensor, an image sensor, etc.) and store the original image as an image sample in the automated testing equipment 100. This embodiment of the present application does not limit this.

[0050] S102. Crop the image sample.

[0051] In some embodiments, the automated testing device 100 may obtain the target region of the image sample through a preprocessing operation and save the target region. The preprocessing operation includes image segmentation based on an edge detection algorithm and image rotation based on a Hough transform.

[0052] S103. Extract image features.

[0053] In the embodiment of the present application, the automated testing device 100 recognizes text data in a target area using optical character recognition (OCR) technology, marks the remaining area in the target area as image data, and extracts image features from the image data.

[0054] In some embodiments, the automated testing equipment 100 includes extracting image features from the image data using convolutional neural networks (CNN) or pre-trained models, wherein the pre-trained models include visual geometry group networks (VGG) and residual networks (ResNet).

[0055] S104. Label the text sample.

[0056] In some embodiments, the automated testing device 100 uses OCR technology to identify text data in the target area, mark the target area corresponding to the text data, and associate the marked target area with the text data. The automated testing device 100 can automatically set and store the marked target area and text data.

[0057] S105. Extract text features.

[0058] In the embodiment of the present application, the automated testing device 100 extracts the text feature from the marked text data.

[0059] In some embodiments, the automated testing device 100 includes using word segmentation, stop word removal, vectorization, and combining a domain dictionary in the automated testing device to enhance the semantic relevance of text vectorization, and extracting the text features from the text data.

[0060] In the embodiment of the present application, step S103 is used to extract image features in the target area, and steps S104 to S105 are used to extract text features in the target area. Step S103 can be executed in parallel with steps S104 to S105, that is, the automated testing device 100 can simultaneously obtain image features and text features in the target area. The embodiment of the present application does not limit the specific execution order of steps S103 and steps S104 to S105.

[0061] S106. Fusion of image features and text features.

[0062] In the embodiment of the present application, the automated testing device 100 fuses the image feature and the text feature using a fusion algorithm to obtain a fused feature. The fusion algorithm includes a feature concatenation algorithm and an attention mechanism weighted algorithm.

[0063] In some embodiments, the fusion algorithm used by the automated testing equipment 100 includes a deep learning framework, such as a deep learning framework based on TensorFlow and PyTorch, which implements dynamic weight distribution of image and text features through the cross-modal attention mechanism therein, and generates fusion features for the image features and the text features using methods including splicing and sharing.

[0064] S107. Generate a data set.

[0065] In the embodiment of the present application, the automated testing device 100 generates a data set including the image feature, the text feature, and the fusion feature, wherein the fusion feature may indicate a corresponding relationship between the image feature and the text feature.

[0066] S108. Perform learning and training on the classification model.

[0067] In an embodiment of the present application, the automated testing device 100 trains a classification model using the data set, and the classification model is used to classify the input data. The classification model training includes classification based on the test case type, test case execution status, and test case priority in the input data. Test case types include, but are not limited to, smoke test types, unit test types, compatibility test types, and performance test types; test case execution status includes, but is not limited to, pass status, fail status, retest status, and blocked status; and test case priority includes, but is not limited to, high priority, medium priority, and low priority.

[0068] In some embodiments, the classification model can screen and classify image samples by sample similarity or keywords, and use a dataset for training and verification.

[0069] S109. Implement classification based on the classification model.

[0070] In an embodiment of the present application, the automated testing device 100 uses the image features and text features in the dataset as input data, classifies the input data using the trained classification model, and outputs classification results. The classification results include classification results based on test case type, classification results based on test case execution status, and classification results based on test case priority sorting from high to low and / or low to high.

[0071] In some embodiments, the classification model can use deep learning and natural language processing technology to perform correlation analysis on image features in the data set and keywords in text features to achieve accurate classification of image samples and filter non-target data in image samples.

[0072] In some embodiments, the classification model of the automated testing equipment 100 may be optimized based on the results of deep learning, for example, by improving the classification performance and generalization ability of the classification model through iterative calculations.

[0073] S110. Output classification results.

[0074] In the embodiment of the present application, the automated testing device 100 outputs the classification result as structured data, where the structured data includes a classification label, a confidence score, and association information between the image and the text.

[0075] In some embodiments, the automated testing device 100 may also support functions such as real-time classification, data management and analysis.

[0076] It can be seen that in the embodiment of the present application, the automated testing method supports using text and images as samples to build an automated testing framework. Figure 1 The various application scenarios shown include image scenarios, text scenarios, and image and text mixed scenarios.

[0077] In the embodiments of this application, Figure 2 The method shown can realize the classification operation of test cases and automatically manage the execution status and results of test cases, reduce manual intervention, and significantly improve software quality and testing efficiency.

[0078] like Figure 3 As shown, for Figure 2 From step S101 to step S106, the present invention provides a feature extraction and fusion method, which is applied to Figure 1 The automated test device 100 in FIG. 1 is used as an example for explanation. The method comprises the following steps:

[0079] S201. Data input.

[0080] In some embodiments, the automated test equipment 100 uses the acquired image samples as input data, and supports batch processing and real-time input of the input data. The data structure of the input data 20 is as follows: Figure 4 As shown, Figure 4 A data structure for feature extraction and fusion is shown. Input data 20 includes text data 21 and image data 22. The text data 21 includes text features, and the image data 22 includes image features.

[0081] S202. Preprocess the input data.

[0082] In some embodiments, the automated test device 100 generates a dataset using natural language processing (NLP) technology (e.g., a bidirectional transformer-based encoder model, BERT) and image processing technology. Specifically, the automated test device 100 can use an image processing library (e.g., an open-source computer vision library, OpenCV) to perform operations such as cropping, rotating, and normalizing the image data 22 to ensure that the image data 22 is in a uniform format.

[0083] S203. Extract text features and image features.

[0084] In some embodiments, the automated test device 100 can use a CNN or pre-trained model to extract image features. The automated test device 100 can also perform word segmentation, stop word removal, vectorization, and other processing on the text data 21 to generate standardized text features. The automated test device 100 can use multithreading or a graphics processing unit (GPU) to accelerate the feature extraction process.

[0085] S204. Fusion of image features and text features.

[0086] In some embodiments, the automated testing device 100 can generate fused features based on the splicing and sharing of the image features and the text features through a deep learning framework, such as TensorFlow and PyTorch. The automated testing device 100 can also use multithreading or a GPU to accelerate the feature fusion process.

[0087] It can be seen that in this embodiment, the feature extraction and fusion method supports the feature extraction of text samples and image samples, and obtains the fusion features after the fusion of text features and image features. This method can be used in Figure 1 The various application scenarios shown are suitable for various application scenarios. At the same time, the corresponding feature extraction and fusion can be dynamically adjusted based on the model of feature extraction and fusion to improve the performance of the execution solution of the automated testing equipment 100.

[0088] In the embodiments of this application, Figure 3 The method shown can adjust the feature extraction strategy and improve the performance of feature extraction.

[0089] like Figure 5 As shown, for Figure 2 In step S107 to step S109, the present application embodiment provides a data classification method, which is applied to Figure 1 The automated test device 100 in FIG. 1 is used as an example for explanation. The method comprises the following steps:

[0090] S301. Fusion feature storage.

[0091] In some embodiments, the automated testing device 100 may store the extracted image features, text features, and fusion features in a database or file, and use the database or file as a data set for subsequent calls.

[0092] S302. Model training and classification.

[0093] In some embodiments, the automated testing equipment 100 may input the fused features into a classification model (such as a CNN model, a support vector machine (SVM) model, etc.) for training.

[0094] In some embodiments, the automated test equipment 100 uses the trained classification model to Figure 4 The input data 20 is classified and a category label and confidence score are output. The classification model includes a CNN based on image feature extraction and an SVM based on text feature classification. The classification results and input data include test case type, test case execution status and test case priority.

[0095] In some embodiments, the automated testing device 100 classifies the input data into smoke test type, unit test type, compatibility test type, and performance test type based on the classification model of the input data, and outputs the test case type in the classification result.

[0096] In some embodiments, the automated testing device 100 classifies the input data into a pass state, a fail state, a retest state, and a blocked state based on the state of the input data, and outputs a test case execution state in the classification result.

[0097] In some embodiments, the automated testing device 100 classifies the input data into high priority, medium priority, and low priority based on the priority of the input data, and outputs a test case priority ranking from high to low and / or from low to high in the classification results.

[0098] S303. Output the results.

[0099] In some embodiments, the automated testing device 100 supports visual display of classification results, and can display the classification results by generating a detailed test report, wherein the test report can classify or uniformly output the test result type, failure cause, risk level, etc.

[0100] S304. Model optimization.

[0101] In some embodiments, the automated testing device 100 continuously optimizes the classification algorithm of the classification model based on the accuracy of the classification results and user feedback in step S303. For example, enhanced training data is generated based on the classification results to optimize the classification performance and generalization ability of the classification model, including the robustness of the classification model.

[0102] In the embodiments of this application, Figure 5 The method shown can timely and comprehensively analyze the test results and generate detailed reports, thereby improving the classification accuracy and efficiency of the test results.

[0103] It should be understood that although Figures 3 to 5 The steps in the flowcharts are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, as mentioned above, Figures 3 to 5 At least part of the steps in the flowchart involved may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0104] Example 3

[0105] Based on Figures 2 to 5 In accordance with the inventive concept shown, embodiments of the present application further provide an automated testing device. The implementation solution provided by the device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more automated testing device embodiments provided below can be found in the aforementioned limitations of the automated testing method and will not be further elaborated here.

[0106] like Figure 6 As shown, the embodiment of the present application provides an automated testing device 60, comprising:

[0107] The data acquisition module 61 is used to obtain an image sample and save the target area in the image sample;

[0108] A feature extraction and fusion module 62 is used to extract image features and text features in the target area, and further used to fuse the image features and the text features through feature splicing and weighted attention mechanism to obtain fused features, and generate a data set including the image features, the text features and the fused features;

[0109] A model training module 63 is used to train a classification model using the data set, and the classification model is used to classify the input data;

[0110] An intelligent classification module 64 is configured to use the image features and the text features in the data set as the input data, classify the input data using the trained classification model, and output a classification result;

[0111] The result display module 65 is used to output the classification result as structured data, where the structured data includes a classification label, a confidence score, and association information between the image and the text.

[0112] In some embodiments, in terms of automated testing, the data acquisition module 61 is specifically used to: pre-store image samples in the storage unit, and can also be used to collect the original image of the test object through a sensor (such as a visual sensor, an image sensor, etc.), and store the original image as an image sample in the automated testing equipment 100.

[0113] In some embodiments, in terms of feature extraction and fusion, the data acquisition module 61 is further configured to obtain and save a target region of an image sample through a preprocessing operation.

[0114] In some embodiments, in terms of automated testing, the feature extraction and fusion module 62 is specifically used to: use CNN or a pre-trained model to extract image features from the image data, mark a part of the target area as text data, obtain the marked image samples and text data, and associate the marked image samples and text data, and extract the text features from the image marked as text data.

[0115] In some embodiments, in terms of automated testing, the feature extraction and fusion module 62 is specifically used to: obtain fusion features using a fusion model based on a deep learning framework (such as TensorFlow and PyTorch), and generate fusion features for the image features and the text features using methods including splicing and sharing.

[0116] In some embodiments, in terms of automated testing, the model training module 63 is specifically used to: screen and classify image samples by sample similarity or keywords, and use data sets for training and verification.

[0117] In some embodiments, in automated testing, the intelligent classification module 64 is specifically configured to: utilize deep learning and natural language processing techniques to perform correlation analysis between keywords within image features and text features within a dataset, thereby accurately classifying image samples and filtering out non-target data within the image samples. Furthermore, the intelligent classification module 64 optimizes the classification model based on the deep learning results, for example, by generating enhanced training data based on the classification results to optimize the classification performance and generalization capabilities of the classification model, including its robustness.

[0118] In some embodiments, in terms of automated testing, the result display module 65 is specifically used to support functions such as real-time classification, data management and analysis.

[0119] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional modules to complete all or part of the functions described above.

[0120] The functional modules in the embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0121] In addition, the specific names of the functional modules are only for the purpose of distinguishing each other and are not intended to limit the scope of protection of this application.

[0122] Each module in the automated testing apparatus 60 may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the form of software in the computer device, so that the processor can call and execute the corresponding operations of each module.

[0123] In some embodiments, a computer device is provided. The computer device may be an automated testing device, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned automated testing method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen may be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc. The display unit of the computer device may be a touch screen covering the display screen, or a trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc. Figure 2 In step S110 , the output classification results of the structured data may be displayed, including displaying the classification label, confidence score, and association information between the image and the text.

[0124] Those skilled in the art will appreciate that each step in the above method embodiments provided herein can be accomplished by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0125] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0126] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0127] In some embodiments, as Figure 8The figure shows an internal structure diagram of a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0130] In some embodiments, the present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the automated testing equipment in any of the above embodiments.

[0131] In one possible design, the chip system also includes a memory for storing program instructions and data, and the memory is located inside or outside the processor.

[0132] The chip system can be composed of chips, or can include chips and other discrete devices.

[0133] In some embodiments, the chip system may include one or more processors. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0134] In some embodiments, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately provided with the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or may be provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the arrangement of the memory and the processor.

[0135] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0136] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0137] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] Those skilled in the art will appreciate that all or part of the processes in the aforementioned method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a readable storage medium, and when executed, the program can include the processes in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An automated testing method, characterized in that: Applied to industrial automation testing for joint analysis of images and texts of the operating status of automated test equipment, including: Acquire an image sample, and save a target area in the image sample; Extracting image features and text features in the target area; The image features and the text features are fused by feature concatenation and weighted attention mechanism to obtain fused features, and a data set including the image features, the text features and the fused features is generated; Training a classification model using the data set, wherein the classification model is used to classify input data; Using the image features and the text features in the data set as the input data, classifying the input data using the trained classification model, and outputting a classification result; The classification result is output as structured data, where the structured data includes a classification label, a confidence score, and association information between the image and the text.

2. The method according to claim 1, characterized in that The automated testing equipment includes a visual sensor, and the acquiring of image samples and the storing of target areas in the image samples include: The image samples are collected based on the visual sensor, the target area of the image samples is obtained through a preprocessing operation, and the target area is saved. The preprocessing operation includes image segmentation based on an edge detection algorithm and image rotation based on a Hough transform.

3. The method according to claim 1, characterized in that The extracting of image features and text features from the target area includes: The text data in the target area is identified, the remaining area in the target area is marked as image data, the image features are extracted from the image data, and the text features are extracted from the text data.

4. The method according to claim 3, characterized in that Extracting the image features from the image data includes extracting the image features from the image data using a convolutional neural network (CNN) or a pre-trained model, wherein the pre-trained model includes a visual geometry group network (VGG) and a residual network (ResNet); The extracting of the text features from the text data includes using word segmentation, removing stop words, vectorization, and combining a domain dictionary to enhance the semantic relevance of text vectorization to extract the text features from the text data.

5. The method according to claim 1, wherein The image features and the text features are fused by feature concatenation and weighted attention mechanism to obtain fused features, including: Based on the splicing and sharing of the image features and the text features, dynamic weight distribution of image and text features is achieved through a cross-modal attention mechanism in a deep learning framework to generate fusion features. The deep learning framework includes TensorFlow and PyTorch.

6. The method according to claim 1, characterized in that The classification model includes a convolutional neural network (CNN) based on the image feature extraction and a support vector machine (SVM) based on the text feature classification. The classification result includes the test case type, the test case execution status and the test case priority.

7. The method according to claim 1, characterized in that After taking the image features and the text features in the data set as the input data, classifying the input data using the trained classification model, and outputting the classification results, the method further includes: Enhanced training data is generated based on the classification results to optimize the classification performance and generalization ability of the classification model, including the robustness of the classification model.

8. An automated testing device, characterized in that: include: A data acquisition module, configured to acquire image samples and save target areas in the image samples; a feature extraction and fusion module, configured to extract image features and text features in the target area, and further configured to fuse the image features and text features through feature concatenation and weighted attention mechanism to obtain fused features, thereby generating a data set including the image features, the text features, and the fused features; A model training module, configured to train a classification model using the data set, wherein the classification model is configured to classify input data; An intelligent classification module, configured to use the image features and the text features in the data set as the input data, classify the input data using the trained classification model, and output a classification result; The result display module is used to output the classification result as structured data, where the structured data includes a classification label, a confidence score, and association information between the image and the text.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images