Method and system for detecting in-vehicle infotainment system by using image recognition technology
By applying image recognition technology to automate testing and abnormal monitoring in vehicle-machine systems, the problem of difficulty in monitoring and positioning abnormalities in vehicle-machine systems in the prior art is solved, and efficient testing and low-cost abnormal detection are achieved.
Patent Information
- Application Number
- CN202411464071.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art is difficult to effectively monitor and locate occasional abnormal problems such as floral screens and black screens in vehicle systems, resulting in low testing efficiency and high cost.
Image recognition technology is used to conduct automated testing and abnormality monitoring on the vehicle system screen. By importing whitelists, connecting the vehicle system, recording and analyzing screen images, we can determine whether abnormalities occur, and record the abnormal time period.
It realizes automated testing of vehicle and machine systems and rapid detection of abnormal conditions, saves testing costs, and provides R&D personnel with convenient problem positioning and solutions.
Smart Images
Figure CN119441032A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle computer systems, and in particular to a method and system for testing a vehicle computer system using image recognition technology. Background Art
[0002] With the development of intelligent and connected cars, the functions of car computer systems are becoming more and more abundant, and more and more application software is installed. Therefore, how to ensure the stable performance of car computer systems has become a problem that testers and developers need to solve. In the process of automobile development, car computer systems often have problems such as screen distortion and black screen. When these problems occur, testers or non-professionals often find it difficult to protect the site and provide accurate log information, which makes it impossible for engineers to directly locate and identify the problem. Therefore, how to simulate the abnormal scene of the car computer system has become a top priority for testers and developers.
[0003] At present, in the process of vehicle computer development, the monitoring of vehicle computer abnormal conditions, especially the monitoring of occasional difficult-to-locate problems such as screen distortion and black screen, is often carried out through two methods: manual simulation of scenarios or simulation testing through professional automated test cabinets. The manual testing method not only requires a large amount of manpower and equipment resources, but also testers are often unable to record and discover problems in a timely and accurate manner when problems occur. Automated test cabinets are often general-purpose equipment developed by equipment suppliers themselves, which need to be specially customized according to the application scenario before they can be used. They are very expensive and have high testing costs.
[0004] Based on the defects of the existing technology, there is an urgent need for a method and system for testing the vehicle system using image recognition technology to improve the efficiency of R&D personnel and testers in discovering and solving problems. Summary of the invention
[0005] The main purpose of the present invention is to provide a method and system for testing a vehicle system using image recognition technology, to complete automated testing of the vehicle system and record abnormal situations, providing convenience for R&D personnel to discover and solve problems.
[0006] The technical solution adopted by the present invention is: a method for testing a vehicle system using image recognition technology, comprising:
[0007] Import a whitelist containing the names of the application software packages to be tested; connect the vehicle system to be tested, and test each application software to be tested in the whitelist in the vehicle system to be tested; perform image recognition and analysis on the vehicle system screen during the test to determine whether any abnormal situation occurs, and record the time period when the abnormal situation occurs.
[0008] Furthermore, the testing of each application software to be tested in the white list running in the vehicle system to be tested specifically includes: using an open command to open each application software to be tested in the vehicle system to be tested, and simulating interaction through debugging instructions.
[0009] Furthermore, the format of the start command includes adding am start before the name of the software package to be tested, and the debugging instruction includes a click instruction adb shell input and a sliding instruction adb shell swip.
[0010] Furthermore, the method for performing image recognition on the vehicle system screen during the test specifically includes aiming a photographic device at the vehicle system screen during the test, and calling the photographic device based on a python program to complete recording and reading of the screen image.
[0011] Furthermore, the python program calls the CV2 library in opencv, wherein the method for calling the photographic device to record the screen image includes the CV2.read method, and the method for reading the screen image includes the imread method.
[0012] Furthermore, the method for reading the screen image of the vehicle system during the recorded test process specifically includes: for each frame in the recorded image, reading the test icon corresponding to the reference icon position based on the imread method, and the reference icon is an unchanged icon in the vehicle system, including a home icon and a wifi icon.
[0013] Furthermore, the method of determining whether an abnormal situation occurs by analyzing the identified image specifically includes: comparing the test icon and the reference icon, determining whether there is a reference icon in each frame of the recorded image, and if there is a reference icon in each frame, determining that no abnormal situation occurs; if the number of consecutive frames in which there is no reference icon exceeds a preset value, determining that an abnormal situation occurs.
[0014] Furthermore, the method for comparing the test icon and the reference icon includes: converting the images of the test icon and the reference icon in a certain frame read into grayscale images through the calcHist method; converting the grayscale images of the test icon and the reference icon into a three-dimensional matrix using normalize; obtaining the feature similarity value of the test icon and the reference icon through the compareHist method, and comparing them with a preset feature similarity limit value; if the feature similarity value is not lower than the preset feature similarity limit value, it is judged that a reference icon exists in the corresponding frame, otherwise it is judged that no reference icon exists.
[0015] Furthermore, the feature similarity value and the preset feature similarity limit value are between 0 and 1. The closer the feature similarity value is to 1, the higher the similarity between the test icon and the reference icon; the closer the feature similarity value is to 0, the lower the similarity between the test icon and the reference icon.
[0016] Furthermore, the recording of the time period when the abnormal situation occurs specifically includes: recording the start and end time when the abnormal situation occurs in the vehicle system to be tested during the test process, and storing the screen images within the start and end time.
[0017] Another aspect of the present invention provides a vehicle system test system, comprising a test tool, an image recognition module, an image analysis module and a storage module, wherein:
[0018] The test tool imports a white list containing the names of the application software packages to be tested, and is used to connect to the vehicle system to be tested, and test each application software to be tested in the white list running in the vehicle system to be tested;
[0019] The image recognition module is used to record and read screen images;
[0020] The image analysis module is used to analyze the screen image after recognition to determine whether the vehicle system to be tested has any abnormality;
[0021] The storage module is used to store video recordings and vehicle computer logs when abnormal situations occur during the test of the vehicle computer system to be tested.
[0022] The beneficial effects of the present invention are as follows: the present invention proposes a method and system for testing a vehicle system using image recognition technology, completes the automated testing of various application software by the vehicle system, identifies and analyzes the screen image of the vehicle system during the test, thereby detecting and recording abnormal situations. The present invention replaces manual testing with automated testing, saving testing costs, and the complete record of the abnormal situation time period provides convenience for R&D personnel to locate and solve problems.
[0023] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1is a flow chart of a method for testing a vehicle system using image recognition technology according to an embodiment of the present invention;
[0026] Figure 2 It is a schematic diagram of the structure of a vehicle system test system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0028] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.
[0029] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.
[0030] Example
[0031] This embodiment provides a method for testing a vehicle system using image recognition technology. The process is as follows: Figure 1 As shown, the steps include:
[0032] S1. Import a whitelist containing the names of the application packages to be tested.
[0033] Specifically, the whitelist is imported into the testing tool.
[0034] S2. Connect the vehicle system to be tested, and test each application software to be tested in the whitelist in the vehicle system to be tested.
[0035] Specifically, the test tool is connected to the vehicle system to be tested, and each application software to be tested is run in the vehicle system to be tested.
[0036] The test tool needs to simulate the interaction between the user and the vehicle system, including clicking to start the application software and sliding the page, so as to replace the manual test behavior to realize the automated test. In this embodiment, the command containing the name of the application software package to be tested is used to start the application software. The command format is am start+the name of the software package to be tested. At the same time, the debugging command is used to simulate the interactive behaviors such as clicking and sliding. In this embodiment, the format of the debugging command is adb shell+interactive behavior.
[0037] S3. Perform image recognition and analysis on the vehicle system screen during the test to determine whether any abnormality occurs and record the time period when the abnormality occurs.
[0038] S3.1. Call the photographic device based on the Python program to recognize the screen image.
[0039] Specifically, the photographic device is aimed at the screen image during the test. In this embodiment, the photographic device selects a portable camera, and the photographic device is called based on the python program to record and read the image frame by frame.
[0040] Furthermore, the python program calls the CV2 library in opencv, and the method for recording images includes the CV2.read method, which can be used to record images frame by frame. For each frame of the recorded image, the imread method is used to read the test icon corresponding to the reference icon. The reference icon is an icon in the vehicle system that does not change with various scenes, including the home icon and wifi icon that come with the vehicle system.
[0041] S3.2. Analyze the recorded images to determine whether any abnormality occurs.
[0042] Specifically, common abnormal situations include a distorted screen and a black screen. Therefore, whether the current screen image is normal can be judged by whether there is a reference icon on the screen. That is, if the reference icon cannot be read normally, a distorted screen or a black screen failure may occur that blocks or does not display the reference icon. The method for judging the occurrence of an abnormal situation includes reading the icon in the area corresponding to the position of the reference icon in each frame of the recorded image as a test icon, comparing the test icon and the reference icon. If the two are highly similar, it is judged that there is a test icon corresponding to the reference icon.
[0043] If a test icon exists in every frame of the recorded image, it is judged that no abnormality has occurred; if frames without a test icon appear continuously, and the number of such consecutive frames exceeds a preset value, it is judged that an abnormality has occurred. From the above, it can be seen that the basis for judging whether a reference icon exists in a certain frame is to compare the similarity between the test icon and the reference icon in the frame. This embodiment then compares the similarity between the two.
[0044] Furthermore, a method for comparing the test icon and the reference icon is specifically described. First, the images of the test icon and the reference icon are converted into grayscale images through the calcHist method. Then, the grayscale images of the test icon and the reference icon are converted into three-dimensional matrices using normalize for comparison. Finally, the feature similarity value of the test icon and the reference icon is obtained through the compareHist method. The value is compared with the preset feature similarity limit value. If it is not lower than the preset feature similarity limit value, that is, the similarity between the two is high, then it is judged that there is a test icon that matches the reference icon. If it is lower than the preset feature similarity limit value, that is, the similarity between the two is low, then it is judged that there is no test icon that matches the reference icon.
[0045] The feature similarity value is between 0 and 1. The closer the feature similarity value is to 1, the higher the match between the test icon and the reference icon, the higher the possibility that the test icon is consistent with the reference icon, and the lower the possibility that the vehicle system under test has an abnormality at the time corresponding to the frame; the closer the feature similarity value is to 0, the lower the match between the test icon and the reference icon, the lower the possibility that the test icon is consistent with the reference icon, and the higher the possibility that the vehicle system under test has an abnormality at the time corresponding to the frame. Since the feature similarity can only indicate the possibility that the test icon and the reference icon are consistent, an accurate value is needed to define when the test icon is judged to be consistent with the reference icon and when the test icon is judged to be inconsistent with the reference icon. On this basis, a preset feature similarity boundary value is proposed, and the feature similarity boundary value is also between 0 and 1.
[0046] Preferably, the feature similarity boundary value preset here is reasonably determined through actual debugging, so that whether the read test icon is consistent with the reference icon can be accurately determined based on the boundary value.
[0047] S3.3. If an abnormal situation occurs, record the time period when the abnormal situation occurs.
[0048] Specifically, if an abnormal situation occurs in the analysis results, the start and end time of the abnormal situation is recorded, and the images recorded within the start and end time are stored. The vehicle computer log is exported for verification by the developer. If there is no abnormal situation, the application software to be tested is replaced and the test is continued.
[0049] Through this step, based on the saved video and vehicle log, the abnormal situation in the vehicle system test can be completely saved and reproduced in front of the R&D personnel. By manually verifying the video and vehicle log, the R&D personnel can not only accurately know the time when the abnormal situation of the vehicle system occurred, but also intuitively understand the specific situation of the abnormality, so as to find and solve the problem and promote the progress of R&D.
[0050] Another aspect of this embodiment provides a vehicle system test system, the structure of which is as follows Figure 2 As shown, it includes a testing tool, an image recognition module, an image analysis module and a storage module.
[0051] Specifically, a whitelist containing the names of the application software packages to be tested is imported into the test tool, and the vehicle system to be tested is connected to run the application software to be tested in the whitelist in the vehicle system to be tested. The test tool is used to simulate the interaction between the user and the vehicle system, including clicking to start the application software and sliding the page, thereby replacing manual testing behavior to achieve automated testing. To simulate the interactive behavior of starting the application software, the test tool of this embodiment uses a command containing the name of the application software package to be tested to start the application software. The format of the command is am start+the name of the software package to be tested. At the same time, interactive behaviors such as clicking and sliding are simulated through debugging instructions. The format of the debugging instructions used by the test tool in this embodiment is adb shell+interactive behavior.
[0052] This embodiment uses this testing tool to replace traditional manual testing, which can avoid manual errors in the testing process, thereby improving the testing efficiency and greatly saving the labor cost in the testing process.
[0053] Specifically, the image recording module includes a photographic device and image recording software, wherein the photographic device is aligned with the vehicle system screen during the test. In this embodiment, the photographic device selects a portable camera, and the image recording software calls the photographic device to record the screen image based on a python program, thereby realizing frame-by-frame reading of the screen image during the vehicle skills test.
[0054] Furthermore, the python program calls the CV2 library in opencv and records images through the CV2.read method. This method can be used to record images frame by frame. For each frame of the recorded image, the imread method is used to read the test icon corresponding to the reference icon. The reference icon is the icon in the car system that does not change with various scenes, including the home icon and wifi icon that comes with the car system.
[0055] The combined application of the image recording module and the test tool in this embodiment avoids the problem of ignoring abnormal situations due to the failure to record in a timely and comprehensive manner during manual testing.
[0056] Specifically, the image analysis module is used to analyze the recorded images frame by frame to determine whether there is a test icon that matches the reference icon in each frame, and then distinguish abnormal conditions such as a flowery screen and a black screen from the normal screen. At the same time, based on the actual debugging of the preset feature similarity limit value, it is determined whether there is a test icon that matches the reference icon in each frame of the recorded screen, thereby improving the accuracy of the judgment process from the data level.
[0057] Furthermore, the image analysis module in this embodiment reads the icon in the area corresponding to the reference icon position in each frame of the recorded image as a test icon, and compares the test icon with the reference icon. If the two are highly similar, it is determined that there is a test icon corresponding to the reference icon.
[0058] If the test icon exists in every frame of the recorded image, it is judged that no abnormality has occurred; if there are consecutive frames without the test icon, and the number of consecutive frames exceeds the preset value, it is judged that an abnormality has occurred. In order to compare the similarity between the test icon and the reference icon, the image analysis module first converts the images of the test icon and the reference icon into grayscale images through the calcHist method, and then uses normalize to convert the grayscale images of the test icon and the reference icon into a three-dimensional matrix for comparison. Finally, the feature similarity value of the test icon and the reference icon is obtained through the compareHist method, and compared with the preset feature similarity limit value. If it is not lower than the preset feature similarity limit value, that is, the similarity between the two is high, it is judged that there is a test icon that matches the reference icon. If it is lower than the preset feature similarity limit value, that is, the similarity between the two is low, it is judged that there is no test icon that matches the reference icon.
[0059] The feature similarity value is between 0 and 1. The closer the feature similarity value is to 1, the higher the matching degree between the test icon and the reference icon, the higher the possibility that the test icon is consistent with the reference icon, and the lower the possibility that the vehicle system under test has an abnormality at the time corresponding to the frame; the closer the feature similarity value is to 0, the lower the matching degree between the test icon and the reference icon, the lower the possibility that the test icon is consistent with the reference icon, and the higher the possibility that the vehicle system under test has an abnormality at the time corresponding to the frame. Since the feature similarity can only indicate the possibility that the test icon and the reference icon are consistent, an accurate value is needed to help the image analysis module define when to judge that the test icon and the reference icon are consistent and when to judge that the test icon and the reference icon are inconsistent. On this basis, a preset feature similarity boundary value is proposed, and the feature similarity boundary value is also between 0 and 1. Preferably, the preset feature similarity boundary value here is reasonably determined through actual debugging, so that according to the boundary value, it is possible to accurately judge whether the read test icon is consistent with the reference icon.
[0060] Specifically, the storage module is used to store video recordings and vehicle computer logs when an abnormal situation occurs in the vehicle computer system to be tested.
[0061] Furthermore, if the analysis result of the image analysis module shows that an abnormal situation has occurred in the vehicle system to be tested, the storage module records the start and end time of the abnormal situation, and stores the images recorded within the start and end time, and exports the vehicle log for verification by the developer. If there is no abnormal situation, the application software to be tested is replaced to continue testing.
[0062] This module is provided in the present embodiment to complete the storage and reproduction of abnormal conditions in the vehicle system test, and to completely simulate the scenarios in which abnormalities occur for R&D personnel.
[0063] When the system of the present invention is used for testing, the specific steps of the tester are as follows:
[0064] First, a whitelist including the names of the application software packages to be tested is created, and the whitelist is imported into the testing tool.
[0065] Then, the test tool is connected to the vehicle system to be tested, and the portable camera is aimed at the screen of the vehicle system.
[0066] After the preparation is completed, start the test tool and use it to test each application software in the whitelist in the vehicle system to be tested; then the system automatically calls the camera based on the Python program to identify and analyze the screen image to determine whether there is any abnormality. If an abnormality occurs, record the time period when the abnormality occurs, store the corresponding video locally, and export the corresponding vehicle log for verification by the developer.
[0067] From the above steps, we can see that the main work of the tester is to create and import the whitelist, connect the test tool and the vehicle system to be tested, set up and turn on the portable camera, and start the test tool. During the test, the tester does not need to stare at the vehicle system screen all the time. After the test, the corresponding vehicle log and stored video can be directly exported to reproduce the problem.
[0068] To sum up, the present invention provides a method and system for testing a vehicle system using image recognition technology, which is used to complete automated testing of multiple application software in the vehicle system. For abnormal situations that occur during the testing process, the present invention can save them in time and reproduce them for R&D personnel, providing convenience for R&D personnel to discover and solve problems, and saving costs during the vehicle system testing process.
[0069] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.
[0070] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
[0071] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A method for testing a vehicle system using image recognition technology, characterized in that: include: Import a whitelist containing the package names of the application to be tested; Connect to the vehicle system to be tested, and test each application software to be tested in the whitelist in the vehicle system to be tested; Perform image recognition and analysis on the vehicle system screen during the test to determine whether any abnormal situation occurs and record the time period when the abnormal situation occurs.
2. The method for testing a vehicle system using image recognition technology according to claim 1, characterized in that: The testing of each application software to be tested in the white list running in the vehicle system to be tested specifically includes: using an open command to open each application software to be tested in the vehicle system to be tested, and performing simulated interaction through debugging instructions.
3. The method for testing a vehicle system using image recognition technology according to claim 2, characterized in that: The format of the start command includes adding am start before the name of the software package to be tested, and the debugging instruction includes a click instruction adb shell input and a slide instruction adb shell swip.
4. The method for testing a vehicle system using image recognition technology according to claim 1, characterized in that: The method for performing image recognition on the vehicle system screen during the test process specifically includes aiming a photographic device at the vehicle system screen during the test process, and calling the photographic device based on a python program to complete recording and reading of the screen image.
5. The method for testing a vehicle system using image recognition technology according to claim 4, characterized in that: The python program calls the CV2 library in opencv, wherein the method for calling the photographic device to record the screen image includes the CV2.read method, and the method for reading the screen image includes the imread method.
6. The method for testing a vehicle system using image recognition technology according to claim 5, characterized in that: The method for reading the screen image of the vehicle system during the recorded test process specifically includes: for each frame in the recorded image, reading the test icon corresponding to the reference icon position based on the imread method, and the reference icon is an unchanged icon in the vehicle system, including a home icon and a wifi icon.
7. The method for testing a vehicle system using image recognition technology according to claim 6, characterized in that: The method of determining whether an abnormal situation occurs by analyzing an identified image specifically includes: comparing a test icon and a reference icon to determine whether a reference icon exists in each frame of the recorded image; if a reference icon exists in each frame, it is determined that no abnormal situation occurs; if the number of consecutive frames in which no reference icon exists exceeds a preset value, it is determined that an abnormal situation occurs.
8. The method for testing a vehicle system using image recognition technology according to claim 7, characterized in that: The method for comparing the test icon and the reference icon includes: converting the images of the test icon and the reference icon in a certain frame read into grayscale images through the calcHist method; converting the grayscale images of the test icon and the reference icon into a three-dimensional matrix through normalize; obtaining the feature similarity value of the test icon and the reference icon through the compareHist method, and comparing them with the preset feature similarity limit value. If the feature similarity value is not lower than the preset feature similarity limit value, it is judged that the reference icon exists in the corresponding frame, otherwise it is judged that the reference icon does not exist.
9. The method for testing a vehicle system using image recognition technology according to claim 8, characterized in that: The feature similarity value and the preset feature similarity limit value are between 0 and 1, and the closer the feature similarity value is to 1, the higher the similarity between the test icon and the reference icon; The closer the feature similarity value is to 0, the lower the similarity between the test icon and the reference icon.
10. The method for testing a vehicle system using image recognition technology according to claim 1, characterized in that: The recording of the time period when the abnormal situation occurs specifically includes: recording the start and end time when the abnormal situation occurs in the vehicle system to be tested during the test process, and storing the screen images within the start and end time.
11. A vehicle system testing system, characterized in that: It includes a test tool, an image recognition module, an image analysis module and a storage module, wherein: The test tool imports a white list containing the names of the application software packages to be tested, and is used to connect to the vehicle system to be tested, and test each application software to be tested in the white list running in the vehicle system to be tested; The image recognition module is used to record and read screen images; The image analysis module is used to analyze the screen image after recognition to determine whether the vehicle system to be tested has any abnormality; The storage module is used to store video recordings and vehicle computer logs when abnormal situations occur during the test of the vehicle computer system to be tested.