Automobile instrument display function test system based on multi-modal model

Abstract

The invention discloses an automobile instrument display function test system based on a multi-modal model, and the system comprises a test document which is used for transmitting a test instruction, a test object and a test requirement to an instrument image collection module, the multi-modal model and a classification model; the instrument image acquisition module is used for issuing the received test instruction to the automobile instrument, shooting a display picture of the automobile instrument from multiple angles, carrying out light spot filtering on the shot display picture image, and then obtaining an image which is sent to the multi-modal model for analysis and processing through image splicing; the multi-modal model is used for analyzing the received display picture image content according to the received test object and outputting an analysis result to the classification model in a text form; and the classification model is used for analyzing the current test object based on the multi-modal model and judging whether the current display picture of the automobile instrument meets the test requirement in combination with the test requirement.

Description

Technical Field

[0001] The invention relates to the field of function testing, and in particular to an automobile instrument display function testing system based on a multi-modal model. Background Art

[0002] The vehicle dashboard is an important tool for the driver to understand the vehicle status and performance. They display information such as vehicle status and performance in the form of icons, allowing the driver to easily understand the vehicle's operating status. The dashboard uses sensors to detect functional components such as wheels to display information such as vehicle speed, power, and mileage. It controls the on and off of the lights by sending control signals to the vehicle's indicator lights. At the same time, the dashboard can also display various warning messages based on this information.

[0003] At present, when testing the display function of the instrument panel, the tester sends test instructions to the instrument panel one by one, and observes the display content on the instrument panel screen to determine whether the display function meets expectations. This testing method is time-consuming, labor-intensive, and requires high costs.

[0004] In order to solve the above problems, the patent document with patent number 202311765385.6 discloses a method, system and related equipment for functional testing of an electric vehicle instrument panel, which receives an initial image and determines the target type of the instrument panel based on the initial image; determines the target instruction set based on the target type; parses the target instruction set to obtain several test instructions; performs a functional test on the instrument panel according to the test instructions, and obtains the actual parameter group used for the test during the functional test; obtains various parameters updated in real time by the instrument panel during the functional test, and uses these parameters as comparison parameter groups corresponding to each test period; obtains a real-time updated image to be compared of the instrument panel collected by the identification device during the functional test; performs data analysis and processing on the comparison parameter group and the image to be compared according to the corresponding actual parameter group to obtain the data analysis and processing results.

[0005] However, when the above scheme is used for automated testing, the display screen of the vehicle's full LCD instrument panel is captured in real time by a camera, and the captured screen is then transmitted to a host computer for image processing and recognition. In this method, since the ambient light changes frequently, it is difficult to ensure the clarity of the captured screen, resulting in the inability to ensure the accuracy of image recognition. In addition, in order to obtain better clarity of the captured screen, a high-definition camera is often used, which places higher demands on the image processing capabilities of the host computer and makes the algorithm difficult to implement. At the same time, in this method, the cost of acquiring the images to be compared is high, and the system algorithm performs analysis by comparison, lacks scene understanding, and its robustness needs to be improved, and the cost of the entire test system is high.

[0006] Therefore, it is necessary to improve the above-mentioned prior art to overcome the above-mentioned defects. Summary of the invention

[0007] The purpose of the present invention is to provide an automobile instrument display function test system based on a multimodal model. According to the test requirements, test instructions are sent to automobile instruments one by one, and the automobile instrument panel screen display content under the instruction is acquired, detected and understood by a camera. According to the prompt word corresponding to the test instruction, it is judged whether the current display content meets the expectations, thereby realizing the automatic test of the automobile instrument display function.

[0008] The above technical objectives of the present invention are achieved through the following technical solutions:

[0009] The automobile instrument display function test system based on multimodal model includes:

[0010] A test document, which is used to send test instructions, test objects and test requirements to the instrument image acquisition module, the multimodal model and the classification model;

[0011] The instrument image acquisition module is used to send the received test instructions to the automobile instrument, and to shoot the display screen of the automobile instrument from multiple angles. After the captured display screen image is subjected to spot filtering, the image is obtained by image stitching for sending to the multimodal model for analysis and processing;

[0012] A multimodal model for analyzing the received display screen image content according to the received test object and outputting the analysis result in the form of text to the classification model;

[0013] The classification model analyzes the current test object based on the multimodal model and determines whether the current display screen of the automobile instrument meets the test requirements in combination with the test requirements.

[0014] Further, the test document includes a test object, test requirements and test instructions;

[0015] The test document is generated by inputting the multimodal model with the function definition text and the test matrix;

[0016] The content of the function definition text includes signal description and system requirements, and the content of the test matrix includes display requirements, requirement description, functional requirements and variable description;

[0017] The function definition text is processed by Tokenizer and Embedding network successively and then input into the LLM model; the test matrix is ​​processed by Tokenizer and Embedding network successively and then input into the LLM model through the cross attention mechanism;

[0018] The LLM model generates a draft test document based on the above input, in which the generated test instruction items are instruction variable names, and the variable names are manually replaced with test instructions.

[0019] Furthermore, the image acquisition process of the instrument image acquisition module is as follows:

[0020] 1) Before collecting images, set several identification points on the car instrument panel to align the images taken by different cameras;

[0021] 2) When collecting images, the display images taken by different cameras from multiple angles are projected based on the pixel coordinates of the identification points in the display images, the projection matrix is ​​calculated and the projection transformation is performed, and the display images taken by all cameras are aligned;

[0022] 3) The aligned display images are stitched together. During the stitching, a picture with fewer spots is selected as a reference picture through a spot detection algorithm, and the spot area in the reference picture is replaced with the corresponding spot-free area in other pictures to obtain the final picture for analysis and processing.

[0023] Furthermore, the light spot detection algorithm is trained by simultaneously inputting a picture without light spots and a picture with light spots, and the loss function loss of the light spot detection algorithm is set to:

[0024]

[0025] Among them, i represents no spots, i- represents samples with spots, h and w represent the image height and width respectively. is the result after the classifier processes the pixel at the (h, w) position; β is the threshold, which can be selected according to the underfitting or overfitting situation during the training process.

[0026] Furthermore, the multimodal model and the classification model are trained by a training data set;

[0027] The training data set includes data set A and data set B;

[0028] Among them, based on the content, special symbols, and test documents that need to be displayed on the car dashboard, symbol images and corresponding descriptive texts are constructed, and multiple different symbols and corresponding descriptive texts are randomly combined to construct dataset A; the car dashboard display interface and symbol images are crawled from the Internet, and the COGVLM2 and LLAMA3.2 multimodal models are used to understand the above dashboard display interface and symbols based on specific instructions and generate corresponding texts, and dataset B is constructed based on the above dashboard display interface, symbols and corresponding texts.

[0029] Furthermore, the multimodal model includes Tokenizer, Embedding, ImageEncoder, Adapter and LLM;

[0030] Among them, Tokenizer and Embedding are used to process text information. The Tokenizer and Embedding modules directly use the corresponding modules of LLAMA3.2; the ImageEncoder is initialized with some weights of the EVA2-CLIP-E model; the Adapter network is used to align the image text information with the text information; after splicing the text features and image features and mapping them with a layer of fully connected network, the mapping results are input into the LLM model, and the understanding results are output in the form of text.

[0031] Further, the test system includes a training phase and an inference phase;

[0032] In the training phase, the multimodal model is trained first, and then the classification model;

[0033] When training a multimodal model, first use dataset B for supervised learning. When the loss value is stable, use dataset A for supervised learning and reinforcement learning. After the reinforcement learning training, use the multimodal model to understand the images in dataset B and regenerate the image content. Then use dataset B and dataset A for supervised learning and reinforcement learning, and iterate until the accuracy reaches the expected level.

[0034] After the multimodal model training is completed, the classification model is trained with dataset B and dataset A respectively. During the training of the classification model, the multimodal model parameters are frozen. Finally, dataset A is used to fine-tune the entire model to improve the accuracy of the algorithm.

[0035] Furthermore, in the reasoning stage, the user needs to provide a detailed test document, which needs to clearly define the test process, and the test process content should include: test object, test requirements and test instructions;

[0036] According to the test process, the system sends test instructions to the vehicle instrument. After the sending is completed, the vehicle instrument is photographed through the camera to obtain the image displayed on the vehicle instrument screen; the system obtains the object to be tested on the current screen from the test document, and inputs the test object and the image displayed on the vehicle instrument screen into the multimodal model; the system obtains the test requirements from the test document, inputs the test requirements and the output results of the multimodal model into the classification model, obtains the output results of the classification model, and determines whether the test results are qualified.

[0037] In summary, the present invention has the following beneficial effects:

[0038] 1) The multimodal model used has a small number of parameters. The weights of the model's text understanding module are initialized based on part of the weights of the LLAMA3.2 model, and the weights of the image understanding module are initialized based on part of the weights of the EVA2-CLIP-E model. The prior knowledge of multiple models is used to build a dedicated and relatively small multimodal model to improve the training speed.

[0039] 2) Based on the content, special symbols, and test documents that need to be displayed on the car dashboard, construct symbol (including text, special symbols, special logo) pictures and corresponding description text (including instructions and description) pictures, randomly combine multiple different symbols and corresponding description texts to construct data set A; crawl the car dashboard display interface and symbol (including special symbols, characters, icons) pictures on the Internet, use COGVLM2, LLAMA3.2 (90B) and other multimodal models based on specific instructions (prompt words) to understand the above dashboard display interface and symbols and generate corresponding texts, and construct data set B based on the above dashboard display interface, symbols and corresponding texts to reduce data acquisition costs;

[0040] 3) Based on data sets A and B, supervised learning, reinforcement learning and other methods are used to train the multimodal model; based on the test requirements (test requirements are described in text, i.e. prompt words) and the output results of the multimodal model, a binary classification model is designed and trained (model parameters are initialized based on LLAMA3.2 weights) to improve the training speed and classification accuracy.

[0041] 4) Based on the pixel coordinates of the identification points in the display screen, the display screens taken by all cameras are aligned; when stitching, the picture with fewer spots is selected as the reference picture through the spot detection algorithm, and the spot area in the reference picture is replaced with the corresponding spot-free area in other pictures to obtain high-quality pictures for analysis and processing.

[0042] 5) Easy to deploy, high accuracy, and strong versatility. The automobile instrument display function can be tested based on the test document. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural block diagram of the automobile instrument display function testing system based on the multimodal model described in the present invention.

[0044] Figure 2 It is a schematic diagram of the processing flow of the test document described in the present invention.

[0045] Figure 3 The present invention discloses an image acquisition and splicing flow chart of the instrument image acquisition module.

[0046] Figure 4 It is a schematic diagram of the training process of the multimodal model and classification model described in the present invention.

[0047] Figure 5 It is a schematic diagram of the derivation process of the multimodal model and classification model described in the present invention. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with illustrations and specific embodiments.

[0049] like Figure 1 As shown, the present invention proposes test documents, instrument image acquisition, training data set, multimodal model, classification model, classification results and other parts.

[0050] The test document part includes the test object (prompt words), test requirements (prompt words), and test instructions. The test object informs the multimodal model of the symbols that need to be detected and understood (including specific text, indicator symbols, special symbols, etc.); the test requirements inform the classification model of the test requirements. The classification model determines whether the current instrument display content meets the test requirements based on the multimodal model's understanding of the current test object and the test requirements.

[0051] The instrument image acquisition part, after receiving the test instructions corresponding to the test requirements of the test document, sends the instructions to the vehicle instrument through CAN and other communications. After the vehicle instrument displays the content corresponding to the test instructions, the camera captures the instrument display screen.

[0052] As for the training data set, the training data set consists of a set A consisting of pictures of symbols (including text, special symbols, special logos) required to be displayed on the car dashboard and the corresponding descriptive texts (including instructions and descriptions), and a set B consisting of pictures of car dashboard display interfaces and symbols (including special symbols, characters, icons) crawled from the Internet and the corresponding texts generated by large models such as COGVLM and LLAMA3.2 (90B) to understand the above pictures.

[0053] In the multimodal model part, the multimodal model understands the content of the dashboard image according to the current test object (prompt word) and outputs the understanding result in the form of text.

[0054] In the classification model part, the classification model determines whether the content displayed by the car instrument meets expectations based on the test requirements and the multimodal model understanding results.

[0055] In the classification result part, the classification result is used for the development and tuning of the subsequent display interface program.

[0056] like Figure 2As shown in the figure, the test document includes the test object, test requirements, and test instructions. The text model analyzes the test matrix and function definition document to generate a draft of the test document, which is manually revised. The processing flow is shown in the figure below (the Tokenizer and Embedding network are shared with the multimodal model). The LLM used in this stage is obtained by fine-tuning the LLM in the multimodal model using previous documents such as function definition, test matrix, and test requirements.

[0057] The function definition in the figure is written by the user according to the test matrix and customer requirements; the test matrix is ​​provided by the customer. The content of each function in the function definition document includes signal description, system requirements, etc.; the test matrix format is an Excel table, etc., and the content includes display requirements, requirement description, functional requirements, variable description, etc. The function definition text is processed by Tokenizer and Embedding network (shared with multimodal model) and then input into the LLM model; the test matrix is ​​processed by Tokenizer and Embedding network (shared with multimodal model) and then input into the LLM model through the cross attention mechanism. The LLM model generates a first draft of the test document based on the above input (the generated test instruction item is the instruction variable name, and the variable name is manually replaced with the test instruction).

[0058] like Figure 3 As shown, the image acquisition process of the instrument image acquisition module is as follows:

[0059] 1) Before collecting images, set several identification points on the car instrument panel to align the images taken by different cameras;

[0060] 2) When collecting images, the display images taken by different cameras from multiple angles are projected based on the pixel coordinates of the identification points in the display images, the projection matrix is ​​calculated and the projection transformation is performed, and the display images taken by all cameras are aligned;

[0061] 3) The aligned display images are stitched together. During the stitching, a picture with fewer spots is selected as a reference picture through a spot detection algorithm, and the spot area in the reference picture is replaced with the corresponding spot-free area in other pictures to obtain the final picture for analysis and processing.

[0062] The light spot detection algorithm is trained by simultaneously inputting images without light spots and images with light spots. The loss function of the light spot detection algorithm is set to:

[0063]

[0064] Among them, i represents no spots, i- represents samples with spots, h and w represent the image height and width respectively. is the result after the classifier processes the pixel at the (h, w) position; β is the threshold, which can be selected according to the underfitting or overfitting situation during the training process.

[0065] like Figure 4 As shown, the training data set consists of data set A and data set B.

[0066] Based on the content, special symbols and test documents that need to be displayed on the car dashboard, symbol (including text, special symbols, special logo) pictures and corresponding description text (including instructions and description) are constructed, and multiple different symbols and corresponding description texts are randomly combined to construct data set A; car dashboard display interface and symbol (including special symbols, characters, icons) pictures are crawled from the Internet, and multimodal models such as COGVLM2 and LLAMA3.2 (90B) are used to understand the above dashboard display interface and symbols based on specific instructions (prompt words) and generate corresponding text, and dataset B is constructed based on the above dashboard display interface, symbols and corresponding text.

[0067] The data set content includes picture category, picture content description, test requirements, test object, picture, etc. The content of data set A is all constructed manually, with high data quality, but the data quantity is relatively small. In data set B, the pictures come from the Internet, and the picture content is understood and generated by models such as COGVLM2 and LLAMA3.2 (90B); the picture category is generated by models such as COGVLM2 and LLAMA3.2 (90B) based on the prompt words "Is there any unreasonable place in the instrument picture? If yes, please answer: yes, if not, please answer: no"; the test object is unified as "Please carefully analyze the {C} object on the display screen", and C is dynamically generated according to the picture content; the test requirements are unified as "{C} object is fully displayed and cannot overlap with other symbols; if {C} object is an indicator symbol, it should not appear in the middle area of ​​the display screen; if {C} object is a car model, it should be displayed in the middle area of ​​the display screen; if {C} object is text, the picture content must include {C} object content, text starting position coordinates, and line display situation".

[0068] The multimodal model includes Tokenizer, Embedding, ImageEncoder, Adapter, LLM and other parts. Tokenizer and Embedding are used to process text information. Tokenizer and Embedding modules directly use the corresponding modules of LLAMA3.2; ImageEncoder is initialized with some weights of the EVA2-CLIP-E model (the number of parameters of the ImageEncoder module is less than that of the EVA2-CLIP-E network. Therefore, when initializing the ImageEncoder module, the corresponding EVA2-CLIP-E model network layer weights with larger values ​​are used); the Adapter network is used to align the image text information with the text information; after splicing the text features and image features and mapping them with a layer of fully connected network, the mapping results are input into the LLM model (the image features are input through cross attention, and the number of parameters of the LLM model is less than that of LLAMA3.2 11B. Therefore, when initializing the LLM model, the corresponding LLAMA3.2 11B model network layer weights with larger values ​​are used), and the understanding results are output in text form.

[0069] Compared with the LLM model, the classification model has fewer parameters. The classification model is initialized with the partial weights of LLAMA3.2 1B. The classification model shares weights with the Tokenizer and Embedding modules of the multimodal model.

[0070] In the training phase, the multimodal model is trained first, and then the classification model is trained. When training the multimodal model, supervised learning is first performed using dataset B. When the loss value (using the cross entropy loss function) is stable, supervised learning and reinforcement learning are used to train dataset A in sequence. After the reinforcement learning training is completed, the multimodal model is used to understand the images in dataset B and regenerate the image content. Then, supervised learning and reinforcement learning are performed using dataset B and dataset A in sequence, and the accuracy is iterated until the expected accuracy is achieved. After the multimodal model training is completed, the classification model is trained using dataset B and dataset A in sequence. During the training of the classification model, the parameters of the multimodal model are frozen. Finally, the entire model is fine-tuned using dataset A (using the cross entropy loss function and superimposing the loss value) to further improve the accuracy of the algorithm.

[0071] See also Figure 5In the reasoning stage, users need to provide detailed test documents. The test documents need to clearly define the test process, which should include: test object, test requirements, and test instructions. According to the test process, the system sends test instructions to the car instrument. After sending, the car instrument is photographed by the camera to obtain the screen display image of the car instrument; the system obtains the object to be tested on the current screen from the test document, and inputs the test object (prompt word) and the screen display image of the car instrument into the multimodal model; the system obtains the test requirements (prompt word) from the test document, and inputs the test requirements and the output results of the multimodal model into the classification model, obtains the output results of the classification model, and determines whether the test results are qualified.

[0072] In this document, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "vertical", and "horizontal" are based on the directions or positional relationships shown in the accompanying drawings and are only for the clarity of expressing the technical solutions and the convenience of description, and therefore should not be understood as limitations of the present invention.

[0073] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0074] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The automobile instrument display function test system based on multimodal model is characterized by: Including a test document, which is used to send test instructions, test objects and test requirements to the instrument image acquisition module, the multimodal model and the classification model; The instrument image acquisition module is used to send the received test instructions to the automobile instrument, and to shoot the display screen of the automobile instrument from multiple angles. After the captured display screen image is subjected to spot filtering, the image is obtained by image stitching for sending to the multimodal model for analysis and processing; A multimodal model for analyzing the received display screen image content according to the received test object and outputting the analysis result in the form of text to the classification model; The classification model analyzes the current test object based on the multimodal model and determines whether the current display screen of the automobile instrument meets the test requirements in combination with the test requirements.

2. The automobile instrument display function test system based on multimodal model according to claim 1 is characterized in that: The test document includes a test object, test requirements and test instructions; The test document is generated by inputting the multimodal model with the function definition text and the test matrix; The content of the function definition text includes signal description and system requirements, and the content of the test matrix includes display requirements, requirement description, functional requirements and variable description; The function definition text is processed by Tokenizer and Embedding network successively and then input into the LLM model; the test matrix is ​​processed by Tokenizer and Embedding network successively and then input into the LLM model through the cross attention mechanism; The LLM model generates a draft test document based on the above input, in which the generated test instruction items are instruction variable names, and the variable names are manually replaced with test instructions.

3. The automobile instrument display function test system based on multimodal model according to claim 1, characterized in that: The image acquisition process of the instrument image acquisition module is as follows: 1) Before collecting images, set several identification points on the car instrument panel to align the images taken by different cameras; 2) When collecting images, the display images taken by different cameras from multiple angles are projected based on the pixel coordinates of the identification points in the display images, the projection matrix is ​​calculated and the projection transformation is performed, and the display images taken by all cameras are aligned; 3) The aligned display images are stitched together. During the stitching, a picture with fewer spots is selected as a reference picture through a spot detection algorithm, and the spot area in the reference picture is replaced with the corresponding spot-free area in other pictures to obtain the final picture for analysis and processing.

4. The automobile instrument display function test system based on multimodal model according to claim 1, characterized in that: The light spot detection algorithm is trained by simultaneously inputting images without light spots and images with light spots. The loss function of the light spot detection algorithm is set to: Among them, i represents no spots, i- represents samples with spots, h and w represent the image height and width respectively. is the result after the classifier processes the pixel at the (h, w) position; β is the threshold, which can be selected according to the underfitting or overfitting situation during the training process.

5. The automobile instrument display function test system based on multimodal model according to claim 1 is characterized in that: The multimodal model and the classification model are trained using a training data set; The training data set includes data set A and data set B; Among them, based on the content, special symbols, and test documents that need to be displayed on the car dashboard, symbol images and corresponding descriptive texts are constructed, and multiple different symbols and corresponding descriptive texts are randomly combined to construct dataset A; the car dashboard display interface and symbol images are crawled from the Internet, and the COGVLM2 and LLAMA3.2 multimodal models are used to understand the above dashboard display interface and symbols based on specific instructions and generate corresponding texts, and dataset B is constructed based on the above dashboard display interface, symbols and corresponding texts.

6. The automobile instrument display function test system based on multimodal model according to claim 5 is characterized in that: The multimodal model includes Tokenizer, Embedding, ImageEncoder, Adapter and LLM. Among them, Tokenizer and Embedding are used to process text information. The Tokenizer and Embedding modules directly use the corresponding modules of LLAMA3.2; the ImageEncoder is initialized with some weights of the EVA2-CLIP-E model; the Adapter network is used to align the image text information with the text information; after splicing the text features and image features and mapping them with a layer of fully connected network, the mapping results are input into the LLM model, and the understanding results are output in the form of text.

7. The automobile instrument display function test system based on multimodal model according to claim 1 is characterized in that: The test system includes a training phase and an inference phase; In the training phase, the multimodal model is trained first, and then the classification model; When training a multimodal model, first use dataset B for supervised learning. When the loss value is stable, use dataset A for supervised learning and reinforcement learning. After the reinforcement learning training is completed, the multimodal model is used to understand the images in dataset B, and the image content is regenerated. Then, dataset B and dataset A are used for supervised learning and reinforcement learning, and the training is iterated until the expected accuracy is achieved. After the multimodal model training is completed, the classification model is trained with dataset B and dataset A respectively; During the training of the classification model, the multimodal model parameters are frozen; Finally, dataset A is used to fine-tune the entire model to improve the accuracy of the algorithm.

8. The automobile instrument display function test system based on multimodal model according to claim 7 is characterized in that: In the reasoning stage, the user needs to provide a detailed test document. The test document needs to clearly define the test process. The test process should include: test object, test requirements and test instructions; According to the test process, the system sends test instructions to the vehicle instrument. After the sending is completed, the vehicle instrument is photographed through the camera to obtain the image displayed on the vehicle instrument screen; the system obtains the object to be tested on the current screen from the test document, and inputs the test object and the image displayed on the vehicle instrument screen into the multimodal model; the system obtains the test requirements from the test document, inputs the test requirements and the output results of the multimodal model into the classification model, obtains the output results of the classification model, and determines whether the test results are qualified.

Images