A multi-language testing method and system for vehicle cockpit software

Abstract

The invention belongs to the technical field of vehicle software testing, and specifically relates to a vehicle cockpit software multi-language testing method and system; each APP in a central control screen system of a car is opened respectively, and a screenshot of the APP homepage is taken; and a screenshot of the APP subpage is taken; text detection is performed on the screenshot, and a picture of the text information is captured according to the initial position information of the text, and a text image data set is established; the text in the text image data set is traversed, and the text in each text image is extracted respectively; the text content is matched with the content in a translation table in turn, and the sheet page with the most matches is taken as the final matching result; it is judged whether all the text in the pre-processed text content corresponds to the matching result, if so, the test passes, otherwise the test fails; each interface in the software in each language is recorded, and then the text of the corresponding interface is obtained through OCR recognition, and finally the classical Chinese is compared through an algorithm to see whether it is consistent with the expected file input by the product, and then whether each classical Chinese meets the design requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle computer software testing, and in particular relates to a vehicle computer cockpit software multi-language testing method and system. Background Art

[0002] With the continuous development of the globalization of domestic automobiles, more and more cars are no longer limited to domestic sales, but are often sold all over the world. Therefore, cockpit software needs to be adapted to multiple languages ​​to meet the needs of users in different regions. However, how to ensure that the cockpit software correctly handles multiple languages ​​and displays texts in different languages ​​has become an important issue.

[0003] At present, there are mainly two ways to conduct multi-language testing on cockpit software:

[0004] 1. Use a purely manual mode: Based on the corresponding table file of the product's input Chinese and translation, the tester manually verifies whether the translation in each interface is correct.

[0005] The defects of this method are: first, the person conducting the test needs to be familiar with the business logic of the car cockpit and be able to click all the functions of the software.

[0006] Secondly, testers are required to be familiar with other languages. For example, when testing cars sold to Russia, they need to be familiar with the Russian language. Otherwise, a lot of time will be spent on translation during testing, and testing errors are prone to occur.

[0007] Finally, a car model will be sold to multiple regions. If all functions in the language of each region need to be manually clicked and verified, it will increase the workload greatly and there will be a lot of duplication of work in the testing.

[0008] 2. Use translation software mode: send the content of other language interfaces to the translation software, which performs OCR recognition and translates it into Chinese. The tester verifies the correctness of the Chinese.

[0009] The defects of this method are: the solution is heavily dependent on the accuracy of the translation software. Since the software in the vehicle cockpit needs to provide very accurate information to the user, this solution can only be used as an auxiliary means and cannot completely replace manual labor.

[0010] In addition, the current shortened automobile R&D cycle has also shortened the entire testing cycle. With limited manpower, testers will invest more in testing business functions and actual vehicle road tests. In the end, there is very little time left for testers to use for multilingual testing, and there is no guarantee that there will be no omissions in the test. Summary of the invention

[0011] In order to overcome the above problems, the present invention provides a multi-language testing method and system for vehicle cockpit software. By means of operation playback, each interface in the software in each language is recorded, and then the text of the corresponding interface is obtained through OCR recognition. Finally, an algorithm is used to compare whether the classical Chinese is consistent with the expected file input by the product, and then whether each classical Chinese meets the design requirements.

[0012] A multi-language testing method for vehicle cockpit software includes the following contents:

[0013] Step 1: Open each APP in the car's central control screen system and take a screenshot of the homepage of each APP; then click each clickable element in each APP and take a screenshot of the subpage that appears after clicking each element;

[0014] Step 2: Perform text detection on the screenshots obtained in step 1, determine the initial position information of the text in each area of ​​the screenshot containing text, capture the text information according to the initial position information of the text, obtain the text image, and establish a text image dataset;

[0015] Step 3, traverse the text in each text image in the text image data set, and extract the text in each text image respectively;

[0016] Step 4: pre-process the extracted text content, including unifying full-width and half-width characters, unifying uppercase and lowercase characters, and removing symbols and spaces;

[0017] Step 5: Match the preprocessed text content with the content of each sheet in the translation table in turn, and take the sheet page that matches the preprocessed text content the most as the final matching result;

[0018] Step six, determine whether all the text in the preprocessed text content corresponds to the content in the sheet page where the matching result is located. If so, the test passes, otherwise the test fails.

[0019] In the step 5, the matching starts from the first sheet page in the translation table and continues to the last sheet page.

[0020] In step six, if there is content corresponding to a certain text content in the sheet page where the matching result is located, it means that the text content is matched successfully. If all the text contents are matched successfully, it means that the test is passed.

[0021] If there is text content that does not successfully match the content in the sheet page where the matching results are located, it is considered that the translation of the APP is incorrect, and manual intervention is required to analyze whether it is a translation error.

[0022] A vehicle cockpit software multi-language testing method is implemented by a vehicle cockpit software multi-language testing system, the testing system comprising an image acquisition module, a text area detection module, a text recognition module, a text matching module and a test result judgment module;

[0023] The image acquisition module calls the car central control screen system, opens each APP in the car central control screen system respectively, and takes a screenshot of the homepage of each APP; then clicks each clickable element in each APP, and takes a screenshot of the subpage that appears after clicking each element;

[0024] The text area detection module performs text detection on the screenshots obtained by the image acquisition module, determines the initial position information of the text in each area containing text in the screenshot, captures the text information according to the initial position information of the text, obtains the text image, and establishes a text image data set;

[0025] The text recognition module traverses the text in each text image in the text image data set and extracts the text in each text image respectively;

[0026] The text matching module pre-processes the text content extracted by the text recognition module, including unifying the full-width and half-width, unifying the uppercase and lowercase, and removing symbols and spaces; then matches the pre-processed text content with the content in each sheet in the translation table in turn, and takes the sheet page that matches the pre-processed text content the most as the final matching result;

[0027] The test result determination module determines whether all text contents in the preprocessed text contents correspond to the contents in the sheet page where the matching results are located. If so, the test passes, otherwise the test fails.

[0028] The translation table is an Excel file, in which each sheet page corresponds to the translation content in all required languages ​​in a cockpit software APP.

[0029] The matching process of the text matching module is as follows:

[0030] Read the content of each sheet page in the translation table Excel, put the translation content in each sheet into a hash table, where the key line is the translation content, and the value is the text content extracted by the text recognition module. Then put the hash table data of the entire sheet page into a linked list data structure, so that when matching, the Hash Key can be used to quickly match whether the text content output by the text recognition module exists in the corresponding translation table.

[0031] In the picture acquisition module, clickable elements on all pages of the car central control screen system are traversed. The specific process is as follows:

[0032] Step 1.1, loop through every APP in the car's central control screen system;

[0033] Step 1.2, open each APP separately, obtain all clickable elements and element positions on the APP homepage, and take a screenshot of the APP homepage;

[0034] Step 1.3, traverse all elements and take a screenshot of the sub-page that appears after clicking each element.

[0035] The text region detection module adopts the DBNet network segmentation model, and its network structure consists of five groups of convolutional layers, among which:

[0036] The first convolutional layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 7×7. The pooling layer adopts maximum pooling with a step size of 2.

[0037] The second convolution layer group includes the second and third convolution layers and a pooling layer. The number of convolution kernels in each layer is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3.

[0038] The third convolutional layer group includes the 4th and 5th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 4.

[0039] The fourth convolutional layer group includes the 6th and 7th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6.

[0040] The fifth convolutional layer group includes the 8th and 9th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3.

[0041] The training process of the DBNet network segmentation model in the text region detection module is as follows:

[0042] Step 1: Take the open source paddlepaddle OCR images, screenshots of the APP homepage in the old car models, and screenshots of each APP sub-page as the training set, and manually mark the frames of all the areas containing text in each image in the training set and the content of the text in the corresponding frames to obtain the labeled training set;

[0043] Step 2: Input the labeled training set obtained in the first step into the DBNet network segmentation model for training, learn the contour features of each labeled text area frame, segment and output all the areas containing text content in each image in the labeled training set according to their complete contours, and stop model training when the accuracy of the segmented area output by the DBNet network segmentation model reaches 90%. The accuracy of the segmented area output by the DBNet network segmentation model is: the number of intersections between the segmented areas predicted by the DBNet network segmentation model and the actual segmented areas / the number of unions between the segmented areas predicted by the DBNet network segmentation model and the actual segmented areas*100.

[0044] The text recognition module uses a CRNN network model, whose network structure consists of five groups of convolutional layers and one group of bidirectional long short-term memory network layers. Its structure is:

[0045] The first convolutional layer group includes a convolutional layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2.

[0046] The second convolution layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2.

[0047] The third convolutional layer group includes the third and fourth convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 1.

[0048] The fourth convolutional layer group includes the 5th and 6th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6.

[0049] The fifth convolutional layer group includes one convolutional layer and one pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 2×2. The pooling layer adopts maximum pooling with a step size of 2.

[0050] A set of bidirectional LSTM layers includes the 1st and 2nd bidirectional LSTM networks, each using 256 neurons.

[0051] The training process of the CRNN network model in the text recognition module is as follows:

[0052] Step 1: Use the images segmented by regions output by the text region detection module as the training set, and manually annotate the text content in each image in the training set, where the actual text content is annotated with a string label to obtain an annotated training set;

[0053] The second step is to preprocess the images in the labeled training set, including random rotation, grayscale, changing to a uniform size, and converting to tensor format;

[0054] Step 3: Input the converted samples obtained in the second step into the CRNN network model for training. When the CRNN network model has an extraction accuracy of 95% for the text annotated in the annotated training set, a trained CRNN text recognition network model is obtained. The extraction accuracy of the CRNN network model for the text annotated in the annotated training set is: the number of characters correctly recognized by the CRNN network model / the total number of characters*100.

[0055] Beneficial effects of the present invention:

[0056] The present invention introduces OCR technology into the multi-language verification of the vehicle system, can quickly complete the multi-language test verification, and ensure the correctness and efficiency of the multi-language test.

[0057] An OCR model in the present invention integrates the recognition of multiple languages ​​and adds automobile-related sample annotation data. The accuracy of this model in the automotive field is improved compared with the open source model.

[0058] The present invention uses the self-developed monkey to deeply traverse the pages, trigger the business and take screenshots, and complete the acquisition of all pages without the intervention of testers.

[0059] The present invention adopts the highest matching principle to automatically determine the correspondence between the screenshots during playback and the translation of the product definition, thereby reducing the degree of manual intervention.

[0060] In summary, the present invention greatly improves work efficiency, reduces the workload of personnel, and can compare the difference between the classical Chinese and the expected results, and is more accurate than the pure manual method. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings used in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without creative work.

[0062] Figure 1 It is a flow chart of the present invention.

[0063] Figure 2 A flow chart for in-depth traversal of the present invention.

[0064] Figure 3 This is the content of the translation table of the embodiment of the present invention.

[0065] Figure 4 The figure is a flow chart of comparing OCR text with classical Chinese in the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0067] In the description of the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0069] In the description of this embodiment, the terms "upper", "lower", "left", "right" and other directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of description and simplification of operation, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0070] Example 1

[0071] A multi-language testing method for vehicle cockpit software, such as Figure 1 As shown, including the following:

[0072] Step 1: Open each APP in the car's central control screen system and take a screenshot of the homepage of each APP; then click each clickable element in each APP and take a screenshot of the subpage that appears after clicking each element;

[0073] Step 2: Perform text detection on the screenshots obtained in step 1, determine the initial position information of the text in each area of ​​the screenshot containing text, capture the text information according to the initial position information of the text, obtain the text image, and establish a text image dataset;

[0074] Step 3, traverse the text in each text image in the text image data set, and extract the text in each text image respectively;

[0075] Step 4: pre-process the extracted text content, including unifying full-width and half-width characters, unifying uppercase and lowercase characters, and removing symbols and spaces;

[0076] Step 5: Match the preprocessed text content with the content of each sheet in the translation table in turn, and take the sheet page that matches the preprocessed text content the most as the final matching result;

[0077] Step six, determine whether all the text in the preprocessed text content corresponds to the content in the sheet page where the matching result is located. If so, the test passes, otherwise the test fails.

[0078] In the step 5, the matching starts from the first sheet page in the translation table and continues to the last sheet page.

[0079] In step six, if there is content corresponding to a certain text content in the sheet page where the matching result is located, it means that the text content is matched successfully. If all the text contents are matched successfully, it means that the test is passed.

[0080] If there is text content that does not successfully match the content in the sheet page where the matching results are located, it is considered that the translation of the APP is incorrect, and manual intervention is required to analyze whether it is a translation error.

[0081] A vehicle cockpit software multi-language testing method is implemented by a vehicle cockpit software multi-language testing system, the testing system comprising an image acquisition module, a text area detection module, a text recognition module, a text matching module and a test result judgment module;

[0082] The computer-readable storage medium calls the self-developed monkey of the car's central control screen system (the central control screen system supports multiple languages) through the adb data cable to automatically traverse the application interface, and save the interface as a picture. Then, through the multi-language OCR recognition function deployed on the computer, text is extracted from the picture, and the extracted text is matched with the content in the classical Chinese table, finally realizing the multi-language testing and verification of the cockpit software.

[0083] The image acquisition module calls the car central control screen system, opens each APP in the car central control screen system respectively, and takes a screenshot of the homepage of each APP; then clicks each clickable element in each APP, and takes a screenshot of the subpage that appears after clicking each element;

[0084] The text area detection module performs text detection on the screenshots obtained by the image acquisition module, determines the initial position information of the text in each area containing text in the screenshot, captures the text information according to the initial position information of the text, obtains the text image, and establishes a text image data set;

[0085] The text recognition module traverses the text in each text image in the text image data set and extracts the text in each text image respectively;

[0086] The output of the text recognition module is the text content in a text image. There can only be one language on the car computer at the same time. For example, if the car computer is in Russian, the text extracted by the text recognition module will be Russian, and then it will be matched with the Russian content in the translation table Excel; if the car computer is in English, the text extracted by the text recognition module will be English, and it will be matched with the English content in the translation table Excel.

[0087] The text matching module pre-processes the text content extracted by the text recognition module, including unifying the full-width and half-width, unifying the uppercase and lowercase, and removing symbols and spaces; then matches the pre-processed text content with the content in each sheet in the translation table in turn, and takes the sheet page that matches the pre-processed text content the most as the final matching result;

[0088] The test result determination module determines whether all text contents in the preprocessed text contents correspond to the contents in the sheet page where the matching results are located. If so, the test passes, otherwise the test fails.

[0089] The translation table is an Excel file, in which each sheet page corresponds to the translation content of all required languages ​​(Chinese, English, Arabic and Russian) in a cockpit software APP, that is, each sheet page corresponds to an app, and each sheet page contains all the translations.

[0090] Since we don't know which sheet page the cockpit software APP corresponds to, we need to match it with the translated contents in all sheet pages and assume that the sheet page with the most matches is the corresponding translation. That is, the text output by the text recognition module and the corresponding translated text on which sheet page have the most matching segments. The specific type of match is related to the language set on the car computer. If it matches, it means that the translation of the language is correct and consistent with the expectation. Otherwise, if it does not match, it is considered that the translation of the cockpit software app is wrong and needs to be modified.

[0091] The matching process of the text matching module is as follows:

[0092] like Figure 3 As shown, the content of each sheet page in the translation table Excel is read, and the translation content in each sheet is put into a hash table, where the key line is the translation content, and the value is the text content extracted by the text recognition module. Then the hash table data of the entire sheet page is put into a linked list data structure, so that when matching, the Hash Key can be used to quickly match whether the text content output by the text recognition module exists in the corresponding translation table.

[0093] In the image acquisition module, the traversal of clickable elements on all pages of the car central control screen system is based on depth-first traversal, such as Figure 2 As shown, the specific process is as follows:

[0094] Step 1.1, loop through every APP in the car's central control screen system;

[0095] Step 1.2, open each APP separately, obtain all clickable elements and element positions on the APP homepage, and take a screenshot of the APP homepage;

[0096] Step 1.3, use the depth-first traversal algorithm to traverse all elements (ie, click each element separately), and take a screenshot of the sub-page that appears after clicking each element.

[0097] The text region detection module adopts the DBNet network segmentation model, and its network structure consists of five groups of convolutional layers, among which:

[0098] The first convolutional layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 7×7. The pooling layer adopts maximum pooling with a step size of 2.

[0099] The second convolution layer group includes the second and third convolution layers and a pooling layer. The number of convolution kernels in each layer is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3.

[0100] The third convolutional layer group includes the 4th and 5th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 4.

[0101] The fourth convolutional layer group includes the 6th and 7th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6.

[0102] The fifth convolutional layer group includes the 8th and 9th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3.

[0103] The training process of the DBNet network segmentation model in the text region detection module is as follows:

[0104] Step 1: Use the open source paddlepaddle OCR images, screenshots of the APP homepage in old models, and screenshots of each APP sub-page as the training set, manually annotate the frames of all the areas containing text in each image in the training set and the content of the text in the corresponding frames to obtain the annotated training set. The number of samples prepared is 30,000 samples for each language;

[0105] Step 2: Input the annotated training set obtained in the first step into the DBNet network segmentation model for training, learn the contour features of the annotated text area frames, and segment and output all the areas containing text content in each image in the annotated training set according to their complete contours (one image will be segmented into multiple small images for output according to the different text areas). The ratio of training samples to verification samples is 7:3. When the accuracy of the segmented area output by the DBNet network segmentation model reaches 90%, stop model training and use this model. The accuracy of the segmented area output by the DBNet network segmentation model is: the number of intersections between the segmented areas predicted by the DBNet network segmentation model and the true segmented areas / the number of unions between the segmented areas predicted by the DBNet network segmentation model and the true segmented areas*100.

[0106] The text recognition module uses a CRNN network model, whose network structure consists of five groups of convolutional layers and one group of bidirectional long short-term memory network layers. Its structure is:

[0107] The first convolutional layer group includes a convolutional layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2.

[0108] The second convolution layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2.

[0109] The third convolutional layer group includes the third and fourth convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 1.

[0110] The fourth convolutional layer group includes the 5th and 6th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6.

[0111] The fifth convolutional layer group includes one convolutional layer and one pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 2×2. The pooling layer adopts maximum pooling with a step size of 2.

[0112] A set of bidirectional LSTM layers includes the 1st and 2nd bidirectional LSTM networks, each using 256 neurons.

[0113] The training process of the CRNN network model in the text recognition module is as follows:

[0114] Step 1: Use the images segmented by regions output by the text region detection module as the training set, and manually annotate the text content in each image in the training set, where the actual text content is annotated with a string label to obtain an annotated training set;

[0115] The second step is to preprocess the images in the labeled training set, including random rotation, grayscale, changing to a uniform size (i.e., unifying the size of the images after the regional segmentation in the labeled training set), and converting them into tensor format; the segmented image samples output by the regional detection network segmentation model are 50,000 samples for each language;

[0116] Step 3: Input the converted samples obtained in the second step into the CRNN network model for training. The ratio of training samples to verification samples is 8:2. When the CRNN network model has an extraction accuracy of 95% for the text annotated in the annotated training set, the trained CRNN text recognition network model is obtained. The extraction accuracy of the CRNN network model for the text annotated in the annotated training set is: the number of characters correctly recognized by the CRNN network model / the total number of characters*100. The final output of the model is the content of the text in the picture. Only the text content can be compared with the translation in Excel. The final output is all the text content included in the entire screenshot.

[0117] Example 2

[0118] The core content of a multi-language testing method for vehicle cockpit software includes three parts: self-developed monkey deep traversal, multi-language OCR recognition and result matching.

[0119] First, the tester connects the computer-readable storage medium to the central control screen system via USB and turns on the adb debugging function on the central control screen system.

[0120] Secondly, the tester selects the type of classical language that needs to be verified, which currently includes: English, Russian and Arabic.

[0121] Again, use the monkey tool to perform a depth-first traversal to traverse all clickable elements on the page and click them. The monkey tool obtains all elements of the interface by calling the interface.

[0122] The monkey process based on depth traversal is as follows:

[0123] Get all the apps installed in the central control screen system and loop through every app in the car computer;

[0124] Open each APP separately, obtain all clickable elements and element positions on the APP homepage, and take a screenshot of the APP homepage;

[0125] Use the depth-first traversal algorithm to traverse all elements, capture the images of each page, and store them in the directory of the APP.

[0126] like Figure 2 As shown, the depth-first algorithm is as follows:

[0127] The core idea is to start from the starting node, visit deeply along a path until you reach the deepest node and then return to explore other unvisited nodes.

[0128] First, create a Stack data structure to store the clickable elements of each page.

[0129] Starting from the home page node, mark it as visited and push it into the stack.

[0130] Repeat the following steps until the Stack is empty: Pop a node from the stack and mark it as visited. Click the node and push all its subnodes into the stack.

[0131] Cannot continue, that is, there are no unvisited neighbor nodes, return to the previous node and continue.

[0132] Repeat steps bd.

[0133] Again, the screenshot of the interface taken in the previous step is synchronized to the local server.

[0134] Next, the tester uploads the translation table of the product definition to the test server as the expected result of the comparison. The translation table of the product definition is managed in Excel file format. Each sheet page contains the corresponding classical Chinese of all elements of an app application, including English, Russian, and Arabic. The format is as follows: Figure 3 ;

[0135] Reconstruct the data structure of the entire translation and store it in memory to speed up the matching process in the following process.

[0136] Read the content of each sheet in the translation table excel, put the translation content in each sheet into the hash table, where the key (row) is the translation content and the value is Y. Put the hash table data of the entire sheet page into the linked list data structure, so that when matching, it can quickly match whether there is a corresponding translation content through the Hash Key. The memory data structure is as follows:

[0137] Again, use the OCR model to obtain all the text in the screenshot (due to the car cockpit software, all are normal fonts, without artistic words, etc., so the OCR recognition accuracy is 99.99%) and store it in memory.

[0138] The OCR recognition model in the device of the present invention uses the DBNet algorithm for text detection and the CRNN algorithm for recognition. Due to the characteristics of the classical Chinese format of the cockpit software: fixed, standardized, and with few words, the model uses an open source data set plus annotated data for the vehicle cockpit software during training. The present invention integrates multi-language OCR recognition into one model. In order to improve the operating efficiency, when calling the model for recognition, the language type parameter needs to be passed in. The currently supported language types are:

[0139] Chinese, English, Russian, Arabic, Portuguese

[0140] The pattern training process is as follows Figure 4 ;

[0141] OCR model training process:

[0142] The DBNet algorithm is used to detect text in classical Chinese screenshots, determine the initial position information of the text in the screenshot, capture the text information based on the initial position information of the text, obtain the text image, and establish a text image dataset; then the text image dataset is input into the CRNN model for text recognition to obtain the text content in the screenshot.

[0143] DBNet uses ResNet as the backbone network to extract image features and output four graphic features {F1, F2, F3, F4}.

[0144] DBFPN is used to process the four graphic features {F1, F2, F3, F4}. After the processed features, the features processed by DBFPN are discharged using 2*2 convolution. After 5 layers of convolution in the backbone network, the five layers of convolution obtain feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original size, and the number of channels is 32, 64, 128, 256, and 512. The five layers of convolution obtain feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original size, and the number of channels is 6128, 256, 512, and 1024. The 3×3 convolutions of conv3 and conv4 in the ResNet-18 backbone network are replaced by deformable convolutions, and the UP values ​​are 1, 2, 4, and 8 respectively.

[0145] The fused features are decoded through the convolution operation HEAD to predict the probability map and threshold map. Finally, the approximate binary map for text range box annotation is obtained by measuring the probability map and threshold map F.

[0146] Use CNN to extract local features from the image output in the previous step. After 5 convolutional layers, feature extraction is performed without changing the size of the image. The convolution kernel size is 3X3, the step size s and padding are both 1, the number of channels is 64, 128, 256, 512, and there are four pooling layers with sizes of 2x2, 2x2, 1x2, and 1x2, respectively. The height of the image is halved by 16 times, the width is halved by 4 times, and the feature map scale of the final CNN output is 512X1X40.

[0147] For the feature map, 40 is used as the sequence number of the feature vector. Each column contains 512-dimensional features, forming a 40x512-dimensional matrix, which is input into the RNN network. The RNN has 256 hidden nodes. After passing through the RNN, it becomes a vector of length T×nclass. After softmax processing, each element of the column vector represents the corresponding character prediction probability.

[0148] Next, compare the OCR-recognized content with the product definition translation. The process is as follows: Figure 4 ;

[0149] Traverse the text in the interface screenshot recognized by OCR.

[0150] The text recognized by OCR is obtained and preprocessed, including unification of full-width and half-width, conversion of uppercase and lowercase letters, removal of symbols and other operations.

[0151] Match the preprocessed text with the content in the translation table defined by the product, starting from the first sheet page and continuing to the last sheet page, and take the one with the most matches as the result.

[0152] Output the texts that match successfully and fail to match in the screenshot. The test passes only when all the texts match successfully.

[0153] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the protection scope of the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention. These simple variations all belong to the protection scope of the present invention.

[0154] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0155] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A multi-language testing method for vehicle cockpit software, characterized in that: For each language to be tested, the following is included: Step 1: Open each APP in the car's central control screen system and take a screenshot of the homepage of each APP; then click each clickable element in each APP and take a screenshot of the subpage that appears after clicking each element; Step 2: Use the DBNet network segmentation model to perform text detection on the screenshots obtained in step 1, determine the initial position information of the text in each area containing text in the screenshot, capture the text information according to the initial position information of the text, obtain the text image, and establish a text image dataset; Step 3, traverse the text in each text image in the text image data set, and extract the text in each text image respectively; Step 4: pre-process the extracted text content, including unifying full-width and half-width characters, unifying uppercase and lowercase characters, and removing symbols and spaces; Step 5: Match the preprocessed text content with the content of each sheet in the translation table in turn, and take the sheet page that matches the preprocessed text content the most as the final matching result; The matching starts from the first sheet page in the translation table and continues to the last sheet page; The translation table is an Excel file, in which each sheet page corresponds to the translation content of all required languages ​​in a cockpit software APP, that is, each sheet page corresponds to an app, and each sheet page contains all the translations; Step 6: Determine whether all the characters in the preprocessed text content correspond to the content in the sheet page where the matching result is located. If there is content matching a certain text content in the sheet page where the matching result is located, it means that the text content is matched successfully. If all the text contents are matched successfully, it means that the test is passed. If there is text content that does not successfully match the content in the sheet page where the matching results are located, it is considered that the translation of the APP is incorrect, and manual intervention is required to analyze whether it is a translation error, and the test fails; The training process of the DBNet network segmentation model is as follows: Step 1: Take the open source paddlepaddle OCR images, screenshots of the APP homepage in the old car models, and screenshots of each APP sub-page as the training set, and manually mark the frames of all the areas containing text in each image in the training set and the content of the text in the corresponding frames to obtain the labeled training set; Step 2: Input the labeled training set obtained in the first step into the DBNet network segmentation model for training, learn the contour features of each labeled text area frame, segment and output all the areas containing text content in each image in the labeled training set according to their complete contours, and stop model training when the accuracy of the segmented area output by the DBNet network segmentation model reaches 90%. The accuracy of the segmented area output by the DBNet network segmentation model is: the number of intersections between the segmented areas predicted by the DBNet network segmentation model and the actual segmented areas / the number of unions between the segmented areas predicted by the DBNet network segmentation model and the actual segmented areas*100.

2. A vehicle cockpit software multi-language testing system, used to implement the vehicle cockpit software multi-language testing method of claim 1, characterized in that: The test system includes an image acquisition module, a text area detection module, a text recognition module, a text matching module and a test result judgment module; The image acquisition module calls the car central control screen system, opens each APP in the car central control screen system respectively, and takes a screenshot of the homepage of each APP; Then click on each clickable element in each APP and take a screenshot of the subpage that appears after clicking each element; The text area detection module performs text detection on the screenshots obtained by the image acquisition module, determines the initial position information of the text in each area containing text in the screenshots, captures the text information according to the initial position information of the text, obtains the text image, and establishes a text image data set; The text recognition module traverses the text in each text image in the text image data set and extracts the text in each text image respectively; The text matching module pre-processes the text content extracted by the text recognition module, including unifying the full-width and half-width, unifying the uppercase and lowercase, and removing symbols and spaces; Then, the preprocessed text content is matched with the content of each sheet in the translation table in turn, and the sheet pages that match the preprocessed text content the most are taken as the final matching results; The test result determination module determines whether all text contents in the preprocessed text contents correspond to the contents in the sheet page where the matching results are located. If so, the test passes, otherwise the test fails.

3. The multi-language testing system for vehicle cockpit software according to claim 2, characterized in that: The matching process of the text matching module is as follows: Read the content of each sheet page in the translation table Excel, put the translation content in each sheet into a hash table, where the key line is the translation content, and the value is the text content extracted by the text recognition module. Then put the hash table data of the entire sheet page into a linked list data structure, so that when matching, the Hash Key can be used to quickly match whether the text content output by the text recognition module exists in the corresponding translation table.

4. The multi-language testing system for vehicle cockpit software according to claim 3 is characterized in that: In the picture acquisition module, clickable elements on all pages of the car central control screen system are traversed. The specific process is as follows: Step 1.1, loop through every APP in the car's central control screen system; Step 1.2, open each APP separately, obtain all clickable elements and element positions on the APP homepage, and take a screenshot of the APP homepage; Step 1.3, traverse all elements and take a screenshot of the sub-page that appears after clicking each element.

5. The multi-language testing system for vehicle cockpit software according to claim 4, characterized in that: The text region detection module adopts the DBNet network segmentation model, and its network structure consists of five groups of convolutional layers, among which: The first convolutional layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 7×7. The pooling layer adopts maximum pooling with a step size of 2. The second convolution layer group includes the second and third convolution layers and a pooling layer. The number of convolution kernels in each layer is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3. The third convolutional layer group includes the 4th and 5th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 4. The fourth convolutional layer group includes the 6th and 7th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6. The fifth convolutional layer group includes the 8th and 9th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 3.

6. The multi-language testing system for vehicle cockpit software according to claim 2, characterized in that: The text recognition module uses a CRNN network model, whose network structure consists of five groups of convolutional layers and one group of bidirectional long short-term memory network layers. Its structure is: The first convolutional layer group includes a convolutional layer and a pooling layer. The number of convolution kernels is 64, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2. The second convolution layer group includes a convolution layer and a pooling layer. The number of convolution kernels is 128, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 2. The third convolutional layer group includes the third and fourth convolutional layers and a pooling layer. The number of convolution kernels in each layer is 256, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 1. The fourth convolutional layer group includes the 5th and 6th convolutional layers and a pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 3×3. The pooling layer adopts maximum pooling with a step size of 6. The fifth convolutional layer group includes one convolutional layer and one pooling layer. The number of convolution kernels in each layer is 512, and the size of each convolution kernel is 2×2. The pooling layer adopts maximum pooling with a step size of 2. A set of bidirectional LSTM layers includes the 1st and 2nd bidirectional LSTM networks, each using 256 neurons.

7. The multi-language testing system for vehicle cockpit software according to claim 6, characterized in that: The training process of the CRNN network model in the text recognition module is as follows: Step 1: Use the images segmented by regions output by the text region detection module as the training set, and manually annotate the text content in each image in the training set, where the actual text content is annotated with a string label to obtain an annotated training set; The second step is to preprocess the images in the labeled training set, including random rotation, grayscale, changing to a uniform size, and converting to tensor format; Step 3: Input the converted samples obtained in the second step into the CRNN network model for training. When the CRNN network model has an extraction accuracy of 95% for the text annotated in the annotated training set, a trained CRNN text recognition network model is obtained. The extraction accuracy of the CRNN network model for the text annotated in the annotated training set is: the number of characters correctly recognized by the CRNN network model / the total number of characters*100.

Images