A universal image recognition method for airborne display systems based on test cases

Abstract

The present invention discloses a universal image recognition method for an airborne display system based on a test case, comprising: receiving a test case; the test case comprising test operation steps, a recognition area, and an expected result, specifying an image recognition model for each information in the recognition area, and key information related to the deployment of the image recognition model; controlling the airborne display system to be tested to perform corresponding operations according to the execution steps in the test case; intercepting the airborne display system graphical interface according to the recognition area; calling the image recognition model to recognize the intercepted real-time image information according to the specified image recognition model and the deployment information of the image recognition model; and comparing the output result of the image recognition model with the expected result to determine whether the test has passed. Based on user-edited test cases, the present invention realizes automatic adaptation of different airborne display system interface information, image recognition, and automatic training of target detection models, greatly improving test efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of avionics and relates to a universal image recognition method for an airborne display system driven by test cases, which realizes automated testing of the airborne display system. Background Art

[0002] Existing general image-based testing technologies include:

[0003] a). Machine Vision

[0004] Vision is the most powerful form of perception for humans. By giving machines visual perception, machines can have scene perception capabilities similar to biological vision systems, allowing machines to complete the following main tasks: by analyzing images, generating a set of descriptive information for the scenes or objects involved in the images. Generally speaking, the input of a machine vision system is an image (or image sequence), and the output is a perceptual description of these images. This set of descriptions is closely related to the objects or scenes in these images, and these descriptions can help the machine complete specific subsequent tasks. Image recognition technology based on machine vision usually requires hard-coding the adaptation of different graphical interfaces and icon meanings to different airborne display systems, which increases the development workload of the automated test system. As the airborne display system iterates, the test system also requires the continuous participation, modification, and adaptation of system developers, which increases time and labor costs.

[0005] b). Deep Learning

[0006] Deep learning, a branch of machine learning, is an algorithm that uses artificial neural networks (ANNs) as its architecture to learn features from data. Compared to traditional machine learning, deep learning eliminates the complex manual feature extraction process; instead, deep neural networks can automatically perform this process. However, deep learning also requires large training datasets, using backpropagation and gradient descent algorithms to adjust the weights and bias parameters in the ANN. Deep learning technology has been widely applied in image recognition, achieving performance that surpasses traditional image recognition methods in various subfields, such as image classification and text recognition, and on publicly available datasets. While its widespread adoption in image recognition has circumvented the hard-coding required by traditional methods, its application to airborne display system testing still requires manual labeling and adaptation (i.e., creating datasets specific to each airborne display system). This requires the deep learning model developer's involvement in dataset construction, model selection, offline training, and deployment in real-world testing environments.

[0007] c). Test case driven automated testing

[0008] In the conventional automated testing process, writing test cases is a necessary preparation before testing. Conventional airborne display system test cases often only include the test operating steps, the image coordinates that need to be identified in the test, and the expected results of the recognition content. Generally speaking, these contents are independent of the specific image recognition process and serve as references for the image recognition results only when verifying the test results. The present invention combines the test case with the image recognition algorithm itself, in particular, adds the picture slices (for graphic recognition), recognition box coordinates and content (for graphic target detection) required by the deep learning model during the test case writing process, drives the automatic training of the model under the supervised learning paradigm, and automatically integrates the model into the automated testing process after the training is completed. Summary of the Invention

[0009] In response to the problems existing in previous airborne display systems, the purpose of the present invention is to provide a universal image recognition method for airborne display systems based on test cases, so that the test system can adapt to different airborne display systems and does not require the continuous intervention of software developers. Testers only need to write test cases according to conventional procedures and follow the software prompts to realize the content recognition of the airborne display system based on the test cases.

[0010] The object of the present invention is achieved through the following technical solutions:

[0011] A universal image recognition method for an airborne display system based on test cases, including preparation work before automated testing and automated testing stages;

[0012] The preparation work before automated testing includes:

[0013] 101: Manually write test cases based on the test requirements and expected results of the airborne display system. The test cases include test steps, recognition areas, expected results, image recognition models for each graphic, color, and text information in the recognition area, data preprocessing required for images, colors, and text before input into the image recognition model, and key information related to image recognition model training and deployment.

[0014] In the automated testing phase, it includes:

[0015] 301: Receive test case;

[0016] 302: Control the onboard display system under test to perform corresponding operations according to the execution steps in the test case;

[0017] 303: capturing the graphical interface of the onboard display system after the test operation is executed by means of a video capture card according to the identification area in the test case;

[0018] 304: Based on the image recognition model specified in the test case and the image recognition model deployment information, the locally stored trained image recognition model is called and combined with the collected real-time image information to recognize the information to be detected;

[0019] 305: Compare the output of the image recognition model with the expected result in the test case to determine whether the test passes.

[0020] 306: Output test results.

[0021] Furthermore, the preparation work before automated testing also includes:

[0022] 102: Manually label the graphic information and color information that need to be recognized, and then call the data enhancement algorithm to automatically generate the data set;

[0023] 103: Create a deep learning model for machine vision as an image recognition model, learn the generated data set, and obtain the weights and bias parameters of each layer of the neural network;

[0024] 104: For the same type of object to be identified, create image recognition models with different architectures by automatically adjusting the values ​​of hyperparameters. After training with the generated dataset, cross-validate the effects of image recognition models with different architectures to obtain the optimal image recognition model.

[0025] Furthermore, the specific operation process of the data enhancement algorithm is as follows:

[0026] 201: The user manually labels the image information to be recognized by the airborne display system;

[0027] 202: For the manually annotated image information, randomly select the image orientation that needs to be changed and randomly select the ratio that needs to be changed to determine the image range for data augmentation;

[0028] 203: For the image region selected in step 202, randomly determine to use a copy or a predefined perturbation method to transform the tensor data in the region and generate a new image;

[0029] 204: Randomly sample from the user's manually labeled data and the generated data, and repeat steps 202 and 203 for the sampled images until a predefined size image set is generated.

[0030] Furthermore, in step 103, corresponding image recognition models are created for the graphic information of different control categories of the airborne display system. An end-to-end deep learning paradigm is used, and a backbone network based on a convolutional neural network architecture is used in the image feature extraction part. Different backbone networks are replaced for the graphic information of different control categories.

[0031] Furthermore, the preparation work before the automated test also includes, for text information, adding expected numerical values ​​or text information to the general optical character recognition model to generate an image recognition model.

[0032] The beneficial effects of the present invention are:

[0033] By using the method of the present invention, the test system can adapt to different airborne display systems without the need for continuous intervention by software developers. Testers only need to write test cases according to conventional procedures and follow the software prompts to achieve airborne display system content recognition based on the test cases. Specifically:

[0034] a) Ability to process the image information contained in the input onboard display system interface and extract key information of user interest, such as page name, control name, and control coordinates;

[0035] b) Based on user-edited test cases, this system automatically adapts to different onboard display system interface information, and automatically trains image recognition and target detection models, significantly improving testing efficiency. In particular, the universal design of the image recognition and detection component allows the deep learning models used in this component to be replaced, updated, and iterated at any time.

[0036] c) Provides the creation, editing, and management of deep learning models, automatic generation of training datasets, and automatic training and screening of optimal models, which are ultimately deployed in actual test systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the preparatory steps before testing in the generalized image recognition method for airborne display systems based on test cases.

[0038] Figure 2 Interface sample of the onboard display system.

[0039] Figure 3 Schematic diagram of data augmentation methods.

[0040] Figure 4 Schematic diagram of the end-to-end deep neural network model for graphic information recognition.

[0041] Figure 5 UML design diagram of the image recognition model for the airborne display system.

[0042] Figure 6 Schematic diagram of the automated image recognition test process for airborne display systems.

[0043] Figure 7 Schematic diagram of the implementation plan of the image recognition method for airborne display systems driven by test cases.

[0044] Figure 8 Text detection and recognition results of the image recognition model for airborne display system images.

[0045] Figure 9 Example of user-annotated graphic information.

[0046] Figure 10 Dataset after data augmentation.

[0047] Figure 11 Example of model configuration information after training is completed.

[0048] Figure 12 Example of graphic information recognition results. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0050] The generalized image recognition method for an airborne display system based on a test case shown in this embodiment consists of two parts. The first part is the preparation work before the automatic test of the airborne display system. Figure 1As shown in the figure, the process includes writing test cases, creating a dataset, training and cross-validating the image recognition model, and deploying the final image recognition model on a computer that can access the onboard display system graphical interface. The specific process is as follows.

[0051] 101: Constructing test cases: Manually write test cases based on the test requirements and expected results of the airborne display system. In addition to traditional test operation steps, recognition areas, expected results, and other information, the test cases also include image recognition models for each graphic information, color information, and text information in the recognition area based on the characteristics of machine vision in recognizing graphics, colors, and text. The types of data preprocessing operations that need to be performed on images, colors, and text before inputting them into the image recognition model, as well as key information related to image recognition model training and deployment, are also included in the test cases.

[0052] The deployment information of the image recognition model is used to call the matching image recognition model during testing.

[0053] Data preprocessing is to transform and crop the original image input to the image recognition model to meet the requirements of the image recognition model for the input image specifications.

[0054] See also Figure 2 As shown, the interface of the onboard display system generally includes graphic information, color information and text information.

[0055] For graphic information (i.e., the state represented by the graphic control), additional training of the image recognition model is required.

[0056] For color information (i.e., graphics remain unchanged, and color is used to represent a certain state), traditional RGB thresholds can be used in image recognition models for discrimination. However, this requires that the color information is relatively accurate and single, and that the color does not vary much between regions. If traditional methods do not provide good discrimination results, additional image recognition model training can be used to achieve relatively accurate color recognition based on feature extraction.

[0057] For text information (mainly various sensor values ​​and alarm information), since the image recognition model of text information is relatively fixed, it is usually only necessary to add the expected values ​​or text information to the original general optical character recognition model OCR, and there is no need to additionally train the image recognition model.

[0058] 102: Create a training set: Manually label the graphic information and color information that need to be recognized, and then call the data enhancement algorithm to automatically generate the data set.

[0059] Since the graphic information and color information of the onboard display system may contain specific meanings, traditional data enhancement methods, such as rotating, cropping, and grayscale adjustment of the entire graphic, may not be applicable. At the same time, since the labor and time costs required to label large amounts of data are very high, this embodiment designs a set of self-developed data augmentation algorithms to generate a data set that can adapt to the scale required by deep learning. The image to be recognized is randomly resized (Resizing), and the resized expanded part is subjected to data augmentation operations. Without affecting the image information in the core area, the data set size is expanded to meet the needs of deep learning technology. This can not only significantly reduce the above costs, but also realize the automation of the deep learning process. See Figure 3 As shown in Figure 2, the specific operation process of the data enhancement part is as follows:

[0060] 201: User annotation icon: The user manually annotates the image information to be identified by the airborne display system through a graphical interface.

[0061] 202: Randomly select image enhancement area and ratio: For manually annotated image information, randomly select the image orientation that needs to be changed (such as the left side of the image) and randomly select the ratio that needs to be changed (for example, sampling using a uniform distribution within the range of 5% to 30%) to determine the image range for data enhancement.

[0062] 203: Randomly generate data enhancement content: For the image area selected in step 202, randomly determine to use copying or a predefined perturbation method (such as superimposing Gaussian noise), transform the tensor data in the area (corresponding to the three RGB channels), and generate a new image.

[0063] 204: Generate a dataset of sufficient size: Randomly sample from the user's manually labeled data and the generated data, and repeat steps 202 and 203 for the sampled images until a predefined size image set is generated.

[0064] 103: Model training: Create a deep learning model for machine vision, learn from the generated data set, and obtain the weights and bias parameters of each layer of the neural network.

[0065] In this embodiment, a deep learning model is mainly used as an image recognition model to realize the recognition of text information, graphic information and color information. Taking the recognition of graphic information as an example, this embodiment uses an end-to-end deep learning paradigm and uses a backbone network based on the Convolutional Neural Network architecture in the image feature extraction part to realize the recognition of the state information represented by a specific category of graphics. The end-to-end recognition process is as follows: Figure 4 shown.

[0066] In particular, in this embodiment, corresponding image recognition models are created for the graphic information of different control categories of the airborne display system (such as doors, valves, power supplies, etc., each type of control has several recognizable states), rather than using a single image recognition model to recognize all control categories and states. This can greatly reduce the complexity of the model and thus improve the recognition rate; at the same time, the backbone network used for image recognition also supports free replacement.

[0067] Specific UML modeling design such as Figure 5 shown.

[0068] 104: Cross-validation: For the same type of object to be identified, image recognition models with different architectures are created by automatically adjusting the values ​​of hyperparameters. After training using the generated dataset, the effects of image recognition models with different architectures are cross-validated to obtain the optimal image recognition model.

[0069] After building test cases based on the method proposed in this implementation, unlike previous automated test systems based on machine vision and deep learning technologies, which required developers to implement fixed coding or provide pre-trained models, this approach automatically learns and deploys model parameters based on the expected results in the test cases and the associated image recognition model type. This means that when adapting to different airborne display systems, additional developers with machine vision and deep learning expertise are no longer required. Testers only need to write test cases according to the test case construction method and the required test content, and the system can automatically adapt to the graphical information contained in the current airborne display system.

[0070] The second part is to perform automated testing on the airborne display system. The test system receives the test steps (test cases) to be tested through network communication, and remotely controls the computer where the airborne display system is located to perform the corresponding operations. Finally, the test system captures the image of the airborne display system and calls the image recognition model to identify the object under test, completing information comparison and detection. Figure 6The specific process is as follows:

[0071] 301: Receiving test cases: During automated testing, data transmission is achieved in the form of a microservice architecture, and the pre-compiled test cases are the starting point of the entire automated testing process.

[0072] 302: Execute test operations: According to the execution steps in the test case, the automated test system remotely controls the onboard display system to be tested and executes corresponding operations (eg, clicking a mouse to view onboard sensor information).

[0073] 303: Obtaining interface content: According to the identification area in the test case, the graphic interface of the onboard display system after the test operation is captured by a video capture card, and transmitted back to the automated test system via a network.

[0074] 304: Calling the specified model: Based on the specified image recognition model and the deployment information of the image recognition model in the test case, the automated test system calls the locally stored trained image recognition model and recognizes the information to be detected in combination with the collected real-time image information.

[0075] 305: Comparison of the content to be tested: Combine the output results of the model (graphics, color information, and text information) and compare them with the expected results in the test case to determine whether the test has passed.

[0076] 306: Output the test results. Through comprehensive testing of the automated test system, various test cases are structured and stored, and a user-customized test report is generated to complete the test process of the airborne display system.

[0077] According to the above-mentioned generalized image recognition method for airborne display system based on test cases, in terms of system hardware, such as Figure 7 As shown in the figure, it can be divided into the software system under test (i.e., the onboard display system) and the test system. The test system includes two main services: machine vision and image recognition services and test case management and automated testing services. The machine vision and image recognition services implement the first part, while the test case management and automated testing services implement the second part.

[0078] Although the software system under test and the test system can be deployed on the same computer, in order to take advantage of the microservice architecture designed in this embodiment, the general optimal implementation method should use multiple computers for parallel testing. The specific implementation method is as follows:

[0079] (1) Divide the items and steps to be tested into N parts according to the scale of the test case;

[0080] (2) Deploy the onboard display system to be tested on N computers and run it in N processes (or use n computers but ensure that N display devices are connected, where n ≤ N);

[0081] (3) Use m computers to deploy the machine vision and image recognition services of the test system and run them in N processes, connecting them to the system under test through video capture cards and other means, so as to capture the image information of the N onboard display systems to be tested (where m ≤ N);

[0082] (4) Deploy the test case management and automated testing services of the test system using p computers and run them in N processes. After loading the N parts of the test case and establishing connections with the N image recognition services in step (3) and N (or n) computers running the onboard display system to be tested, execute all test operations contained in the test case (where p ≤ N).

[0083] Figure 8 The figure shows the text detection and recognition results of the image recognition model on the airborne display system image, which mainly includes the recognized text content and the image area where the corresponding text is detected (parameters of the four coordinate points).

[0084] For the recognition of graphic information, Figure 9 An example of user-annotated graphic information is shown, including three graphic states corresponding to the cabin door: invalid state (invalid), unsafe state (nsafe), and safe state (safe).

[0085] The data enhancement method provided in this embodiment is used to Figure 9 Data enhancement is performed on the pictures, and a large number of training data sets are automatically generated (such as Figure 10 ) is used for training and cross-validation of image recognition models.

[0086] The machine vision and image recognition service can provide image recognition model training and inference functions independent of the testing function (implemented based on the open source deep learning framework PaddlePaddle in this implementation example), and automatically generates the corresponding configuration file after completing the model training to facilitate subsequent search, management, and call. Figure 11 The main contents of the configuration information are shown in the figure, including the model name (named after the content or control to be identified), model ID number, input image size (48x48 pixels in the example), preprocessing method, training environment version, and label information to be identified (inval-id, nsafe, and safe in the example).

[0087] During the test, when the machine vision and image recognition service receives the image to be recognized, it will call the correct model to recognize the input image and return the recognition result according to the model information specified in the test case and the locally stored model configuration information. Figure 12 This is the result of cabin door status recognition in this implementation example.

[0088] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A generalized image recognition method for an airborne display system based on a test case, comprising preparatory work before automated testing and an automated testing phase, characterized in that: The preparation work before automated testing includes: 101: Manually write test cases based on the test requirements and expected results of the airborne display system. The test cases include test steps, recognition areas, expected results, image recognition models for each graphic, color, and text information in the recognition area, data preprocessing required for images, colors, and text before input into the image recognition model, and key information related to image recognition model training and deployment. 102: Manually label the graphic information and color information that need to be recognized, and then call the data enhancement algorithm to automatically generate the data set; 103: Create a deep learning model for machine vision as an image recognition model, learn from the generated dataset, and obtain the weights and bias parameters of each neural network layer. Specifically, create corresponding image recognition models for the graphic information of different control categories of the airborne display system. Using an end-to-end deep learning paradigm, use a backbone network based on a convolutional neural network architecture for image feature extraction, and replace different backbone networks for different control categories of graphic information. 104: For the same type of object to be identified, create image recognition models with different architectures by automatically adjusting the values ​​of hyperparameters. After training with the generated dataset, cross-validate the effects of the image recognition models with different architectures to obtain the optimal image recognition model. In the automated testing phase, it includes: 301: Receive test case; 302: Control the onboard display system under test to perform corresponding operations according to the execution steps in the test case; 303: capturing the graphical interface of the onboard display system after the test operation is executed by means of a video capture card according to the identification area in the test case; 304: Based on the image recognition model specified in the test case and the image recognition model deployment information, the locally stored trained image recognition model is called and combined with the collected real-time image information to recognize the information to be detected; 305: Compare the output of the image recognition model with the expected result in the test case to determine whether the test passes; 306: Output test results.

2. The universal image recognition method for an onboard display system based on a test case according to claim 1, characterized in that The specific operation process of the data enhancement algorithm is as follows: 201: The user manually labels the image information to be recognized by the airborne display system; 202: For the manually annotated image information, randomly select the image orientation that needs to be changed and randomly select the ratio that needs to be changed to determine the image range for data augmentation; 203: For the image region selected in step 202, randomly determine to use a copy or a predefined perturbation method to transform the tensor data in the region and generate a new image; 204: Randomly sample from the user's manually labeled data and the generated data, and repeat steps 202 and 203 for the sampled images until a predefined size image set is generated.

3. The universal image recognition method for an onboard display system based on a test case according to claim 1, characterized in that The preparation work before automated testing also includes adding expected numerical or text information to the general optical character recognition model to generate an image recognition model for text information.

Images