Vehicle central control test method, device, equipment, medium and product

Abstract

The invention relates to the technical field of vehicle software testing, and discloses a vehicle central control testing method, device and equipment, a medium and a product, and the method comprises the steps: obtaining a vehicle central control projection screen image; obtaining and executing a test case to output a test instruction to the vehicle, and enabling the vehicle to execute a test operation included in the test case in the vehicle central control picture according to the test instruction; receiving a test loss message fed back by the vehicle; in response to the test loss message, calling the large model to extract the element meaning of the first target element; positioning a second target element corresponding to the element meaning in the vehicle central control projection screen picture through the large model, and correcting the test case based on the second target element; and continuing to test through the corrected test case. According to the invention, the test accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle software testing, and particularly relates to a testing method, device, equipment, medium and product for a vehicle center console. Background Art

[0002] With the development of cockpit intelligence, the iterative development of in-vehicle infotainment (IVI) system software is getting faster and faster, the project cycle is getting shorter and shorter, and the testing tasks of vehicle center consoles are heavy. Currently, the number of functional test cases for each project is generally more than 20,000. Since a large number of operations (such as clicking, inputting, dragging, etc., operations that interact with the center console) need to be performed on the vehicle center console screen and then the test results are recorded, it conservatively requires more than 200 man-hours to perform a full-functional test. Especially when performing repeated verification tests (such as verifying that a certain software function will not go wrong after a large number of repeated clicks), it will increase the high testing cost. And manual testing is easily affected by human factors, such as misjudgment or omission, which may affect the accuracy of the test results. For projects with frequent software requirement changes, continuously iterating user interfaces or containing complex logics, it is difficult for manual testing to cope efficiently. In recent years, the intelligent cockpit technology has developed very rapidly, and automated testing means have also developed. Some enterprises create functions that simulate various manual operations by writing code, and then implement automated testing through the functions, significantly improving the testing efficiency. However, during the testing process, the screen of the center console may change due to iteration, and the test cases are not updated in time, resulting in the test cases being unable to accurately perform interactive operations, making the test results inaccurate and delaying the testing progress. Summary of the Invention

[0003] In view of this, the present invention provides a testing method, device, equipment, medium and product for a vehicle center console to solve the problem of inaccurate testing.

[0004] In a first aspect, the present invention provides a test method for a vehicle center console, which is applied to a host computer. The host computer is communicatively connected to the vehicle. The method includes: obtaining a vehicle center console screen projection image, which is an image projected on the host computer corresponding to the vehicle center console screen, and the vehicle center console screen is the screen displayed on the vehicle center console; obtaining and executing a test case to output a test instruction to the vehicle, so that the vehicle executes the test operations included in the test case in the vehicle center console screen according to the test instruction; receiving a test loss message fed back by the vehicle, where the test loss message is a message generated when the vehicle fails to find a first target element corresponding to the test operation in the vehicle center console screen; in response to the test loss message, invoking a large model to extract the element meaning of the first target element; positioning, in the vehicle center console screen by the large model, a second target element corresponding to the element meaning, and modifying the test case based on the second target element, where the second target element is a screen element obtained by adjusting the first target element in the vehicle center console screen; and continuing the test with the modified test case.

[0005] According to the above technical means, when performing a vehicle center console test through a test case, if a first target element corresponding to the test operation cannot be found in the vehicle center console screen, for example, if the corresponding control or icon cannot be found and operations such as clicking, dragging, and screenshotting cannot be performed, the AI large model is invoked. Through the image understanding ability and semantic understanding ability of the AI large model, first understand the element meaning of the first target element to be manipulated by the test operation, then analyze and understand the current vehicle center console screen, match a second target element with the same element meaning as the first target element, and then use the second target element as the first target element to continue the test. At the same time, the test case is correspondingly modified according to the second target element. This not only ensures that the test can continue accurately and reliably, solves the problem of test interruption, improves the test accuracy rate, but also automatically modifies the test case according to the fallback function of the AI large model, enabling the vehicle center console screen to adaptively change automatically after the modified test case, reducing the steps of manual and frequent modification of the test case by the user and lowering the labor cost.

[0006] In some alternative embodiments, the steps of creating the test case include: detecting the current operation of the user in the vehicle center console screen projection image; recording the element attribute information of the area corresponding to the current operation, and intercepting the area cut image of the area corresponding to the current operation; extracting the text information in the area cut image; generating a current test instruction based on the current operation and the element attribute information; saving the current test instruction, the area cut image, and the text information as the recording information of the current operation; and sorting the recording information of each step of the user's operation in sequence to obtain the test case.

[0007] According to the above technical means, the present invention provides a method for recording test cases. A communication connection is established between the host computer and the vehicle, and then the screen image of the vehicle's central control is projected onto the host computer screen. The user manually performs test operations on the host computer screen, and at the same time, the recording function is enabled, similar to the video recording function, to record each step of the user's operation, and at the same time, the recorded operations are converted into test cases. Compared with the test case creation scheme of code programming, the test case recording method provided by the present invention significantly improves the speed of test case generation, reduces the difficulty of writing test cases, and lowers the employment threshold for testers. Among them, in order to convert the recorded data into test cases in the form of instructions, on the one hand, it is necessary to record each step of the user's operation action on the screen, and on the other hand, it is also necessary to record the specific position of each step of the operation, so as to extract element attribute information, screenshots, and text at the corresponding position. The element attribute information and operation actions can form specific test instructions to indicate what operation is performed on which element object; the specific meaning of the corresponding operation object is determined by the screenshots and text. Even if the image designer subsequently fine-tunes the screen image of the central control, the AI large model can automatically correct the original test instructions according to the screenshots and text, improving the reliability of the test.

[0008] In some optional embodiments, the host computer and the vehicle are connected through the Android Test Bridge. The recording of the element attribute information of the area corresponding to the current operation includes: creating an element structure tree; sending a command to obtain the user interface layout to the vehicle at a preset period; sending a command to obtain the interface file to the vehicle at a preset period; receiving the interface file fed back by the vehicle, where the interface file is the file sent to the host computer by the vehicle each time it responds to the command to obtain the interface file, and the interface file is generated by the vehicle based on the command to obtain the user interface layout, and the interface file stores the interface element layout information, which is the information extracted from the vehicle's central control screen and used to represent the screen elements in the vehicle's central control screen; parsing the interface file to obtain the interface element layout information; filling the interface element layout information into the element structure tree for display; and searching for the element attribute information from the element structure tree according to the position of the area corresponding to the current operation.

[0009] According to the above technical means, the host computer remotely obtains the interface element layout information of the vehicle's central control screen based on the Android Test Bridge and updates the interface element layout information to the element structure tree in real time. Therefore, when the screen changes, the layout and attributes of each element in the screen are updated to the element structure tree in the first time. The various attribute information in the interface is displayed to the user through the element structure tree, so that the user and the host computer can clearly understand what the operation object corresponding to each test operation of the user is, ensuring the accuracy of test case creation.

[0010] In some alternative embodiments, in response to the test loss message, invoking the large model to extract the element meaning of the first target element includes: analyzing the icons in the regional cut map through the large model to obtain the first element meaning; analyzing the text information through the large model to obtain the second element meaning.

[0011] According to the above technical means, based on the process created from the foregoing test cases, each operated first target element includes two parts of backup information, namely the regional cut map and the text information. Thus, when the first target element is lost, the AI large model understands the meaning of the first target element from both the image and text perspectives, and then finds the second target element with the corresponding meaning from the current vehicle center console screen, further improving the accuracy of matching the second target element.

[0012] In some alternative embodiments, the method of using the large model to locate the second target element corresponding to the element meaning in the vehicle center console projection screen and correcting the test case based on the second target element includes: searching for the target icon that conforms to the first element meaning in the vehicle center console projection screen through the large model; when the target icon is found by searching with the first element meaning, recording the target element attribute information corresponding to the target icon, and using the target element attribute information and the target icon as the second target element; modifying the test instruction in the test case with the target element attribute information; replacing the corresponding regional cut map with the target icon; when the target icon cannot be found by searching with the first element meaning, searching for the target text that conforms to the second element meaning in the vehicle center console projection screen through the large model; determining the target icon based on the area where the target text is located, and returning to the step of recording the target element attribute information corresponding to the target icon.

[0013] According to the above technical means, first, the second target element is matched through the image meaning of the regional cut map. If the image cannot be matched, the text meaning is used for matching. When the corresponding second target element is matched, a new screenshot of the second target element is taken, and then the old screenshot is replaced with the new screenshot. The test instruction in the test case is adjusted with the attribute parameters of the new screenshot area, realizing an automated test case correction process. The dual meaning matching ensures the accuracy of the automated test case correction.

[0014] In some alternative embodiments, obtaining the vehicle center console projection screen includes: sending a screen acquisition message to the vehicle; receiving the vehicle center console projection screen fed back by the vehicle, where the vehicle center console projection screen is an image sent after the vehicle takes screenshots of the center console screen at a preset frame rate, compresses the screenshots, and then removes duplicates based on the similarity between adjacent frame screenshots.

[0015] According to the above technical means, the vehicle screen image is compressed while being acquired, the picture resolution is reduced, and the transmission speed is effectively increased while maintaining the image quality. By using an image comparison algorithm, duplicate pictures are removed, and the frame rate is controlled at about 13 FPS, so that the car machine screen can be displayed in real time without delay and the basic viewing screen mirroring effect can be satisfied.

[0016] In some alternative embodiments, the compressed screenshot includes a heavily compressed area and a lightly compressed area. The heavily compressed area is the area where the vehicle compresses the unimportant area in the screenshot at a first compression ratio, and the lightly compressed area is the area where the vehicle compresses the important area in the screenshot at a second compression ratio, and the first compression ratio is less than the second compression ratio.

[0017] According to the above technical means, for the vehicle center control screen, important areas and unimportant areas are divided, and a compression method of increasing the compression ratio for important areas and decreasing the compression ratio for unimportant areas not only reduces the volume of the screen but also takes into account the screen quality.

[0018] In some alternative embodiments, the method further includes: sending an audio message acquisition request to the vehicle; receiving the vehicle audio fed back by the vehicle through the Android Debug Bridge. The vehicle audio is that the vehicle creates an audio recording object in response to the audio message, reads the vehicle background audio data through the audio recording object, then compresses the read audio data into a playable format file, and sends it to the upper computer through the Android Debug Bridge; transmitting the vehicle audio to an audio playback application through local inter-process communication for playback.

[0019] According to the above technical means, based on the Android Debug Bridge technology, the vehicle audio is also transmitted from the background to the upper computer, and the audio of the vehicle center control is played through the upper computer, so that the accuracy of certain tests can be further verified through the vehicle audio. And the vehicle audio stream is directly transmitted from the background. Compared with audio recording means such as an external microphone, the audio quality is higher and the usage threshold is reduced.

[0020] In a second aspect, the present invention provides a test device for a vehicle center console, which is applied to a host computer. The device includes: a screen mirroring module, configured to obtain a screen mirroring of the vehicle center console, where the screen mirroring of the vehicle center console is a screen projected onto the host computer corresponding to the vehicle center console screen, and the vehicle center console screen is the screen displayed on the vehicle center console; a test case execution module, configured to obtain and execute a test case to output a test instruction to the vehicle, so that the vehicle executes the test operations included in the test case in the vehicle center console screen according to the test instruction; a target loss module, configured to receive a test loss message fed back by the vehicle, where the test loss message is a message generated when the vehicle cannot find a first target element corresponding to the operation in the vehicle center console screen during the execution of the test operation; a lost target analysis module, configured to, in response to the test loss message, call a large model to extract the element meaning of the first target element; a lost target detection module, configured to locate a second target element corresponding to the element meaning in the vehicle center console screen through the large model, and correct the test case based on the second target element, where the second target element is a screen element obtained by adjusting the first target element in the vehicle center console screen; a fallback test module, configured to continue the test through the corrected test case.

[0021] In a third aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to the first aspect or any corresponding embodiment thereof.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method according to the first aspect or any corresponding embodiment thereof.

[0023] In a fifth aspect, the present invention provides a computer program product, including computer instructions, and the computer instructions are used to cause a computer to execute the method according to the first aspect or any corresponding embodiment thereof.

[0024] The technical solution provided by the present invention has the following advantages: According to the above technical means, when performing vehicle center console tests through test cases, if the first target element corresponding to the operation cannot be found in the vehicle center console screen, for example, the corresponding control or icon cannot be found, resulting in the inability to perform operations such as clicking, dragging, and screenshotting, the AI large model is called. Through the image understanding ability and semantic understanding ability of the AI large model, first understand the element meaning of the first target element to be manipulated by the test operation, and then analyze and understand the current vehicle center console screen, match the second target element with the same element meaning as the first target element, and then use the second target element as the first target element to continue the test. At the same time, the test case is corrected accordingly according to the second target element. This not only ensures that the test can continue to be executed accurately and reliably, solves the problem of test interruption, improves the test accuracy rate, but also automatically modifies the test case according to the fallback function of the AI large model, enabling the vehicle center console screen to adaptively change automatically after the modified test case, reducing the steps for users to manually modify the test case frequently and lowering the labor cost.

[0025] According to the above technical means, the present invention provides a method for recording test cases. By establishing a communication connection between the host computer and the vehicle, the screen image of the vehicle center console is projected onto the host computer screen. The user manually performs test operations on the host computer screen, and at the same time, the recording function is enabled, similar to the function of video recording, to record each step of the user's operation, and at the same time, the recorded operations are converted into test cases. Compared with the test case creation scheme based on code programming, the method for recording test cases provided by the present invention significantly improves the speed of generating test cases, reduces the difficulty of writing test cases, and lowers the employment threshold for testers. Among them, in order to convert the recorded data into test cases in the form of instructions, on the one hand, it is necessary to record each step of the user's operation action on the screen, and on the other hand, it is also necessary to record the specific position of each step of the operation, so as to extract element attribute information, screenshots, and text at the corresponding positions. The element attribute information and operation actions can form specific test instructions to indicate what operation is performed on which element object; the specific meaning of the corresponding operation object is determined through screenshots and text. Even if the image designer makes fine adjustments to the screen image of the center console later, the AI large model can automatically correct the original test instructions according to the screenshots and text, improving the reliability of the test.

[0026] (3)Based on the above technical means, the host computer remotely obtains the layout information of the interface elements of the vehicle center control screen from the vehicle based on the Android test bridge, and updates the layout information of the interface elements to the element structure tree in real time. Thus, when the screen changes, the layout and attributes of each element in the screen are updated to the element structure tree immediately. The various attribute information in the interface is displayed to the user through the element structure tree, so that the user and the host computer can clearly understand what the operation object corresponding to each test operation of the user is, ensuring the accuracy of test case creation.

[0027] (4)Based on the above technical means, and based on the process of creating the foregoing test cases, each first target element to be operated includes two parts of fallback information: regional cutout and text information. Thus, when the first target element is lost, the AI large model understands the meaning of the first target element from both the image and text perspectives, and then finds the second target element with the corresponding meaning from the current vehicle center control screen, further improving the accuracy of matching the second target element.

[0028] (5)Based on the above technical means, first match the second target element through the image meaning of the regional cutout. If the image cannot be matched, then use the text meaning for matching. When the corresponding second target element is matched, take a new screenshot of the second target element, and then use the new screenshot to replace the old screenshot. Adjust the test instructions in the test case using the attribute parameters of the new screenshot area, realizing an automated test case correction process. The dual-meaning matching ensures the accuracy of automated test case correction.

[0029] (6)Based on the above technical means, the vehicle screen image is obtained and compressed while reducing the picture resolution, effectively improving the transmission speed while maintaining the image quality. By using an image comparison algorithm, duplicate pictures are removed, and the frame rate is controlled at about 13 FPS. In this way, the car machine screen can be displayed in real time without delay, and the basic screen mirroring viewing effect can be satisfied.

[0030] (7)Based on the above technical means, the vehicle center control screen is divided into important areas and unimportant areas, and the compression ratio is increased for important areas and decreased for unimportant areas, not only reducing the volume of the screen, but also taking into account the screen quality.

[0031] (8)Based on the above technical means and the technical means of the Android debug bridge, the vehicle audio is also transmitted from the background to the host computer, and the audio of the vehicle center control is played through the host computer. Thus, the accuracy of certain tests can be further verified through the vehicle audio. And the vehicle audio stream is directly transmitted from the background. Compared with audio recording means such as external microphones, the audio quality is higher and the usage threshold is reduced. Description of the Drawings

[0032] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0033] Figure 1 is a schematic flowchart of a method for testing a vehicle center console according to an embodiment of the present invention; Figure 2 is a schematic flowchart of creating a test case according to an embodiment of the present invention; Figure 3 is a schematic diagram showing the display effect of an element structure tree according to an embodiment of the present invention; Figure 4 is a schematic flowchart of obtaining interface element layout information according to an embodiment of the present invention; Figure 5 is another schematic flowchart of a method for testing a vehicle center console according to an embodiment of the present invention; Figure 6 is another schematic flowchart of a method for testing a vehicle center console according to an embodiment of the present invention; Figure 7 is a schematic structural diagram of a testing device for a vehicle center console according to an embodiment of the present invention; Figure 8 is a schematic hardware structure diagram of a computer device according to an embodiment of the present invention. Specific Embodiments

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0035] With the development of cockpit intelligence, the iterative development of in-vehicle infotainment (IVI) system software is getting faster and faster, the project cycle is getting shorter and shorter, and the testing tasks of vehicle center consoles are heavy. Currently, the functional test cases for each project are generally more than 20,000. Since a large number of operations (such as clicking, inputting, dragging, etc., operations that interact with the center console) need to be performed on the vehicle center console screen and then record the test results, it conservatively requires more than 200 man-hours to perform a full-functional test. Especially when performing repetitive verification tests (such as verifying that a certain software function will not malfunction after a large number of repeated clicks), it will increase the high testing cost. And manual testing is easily affected by human factors, such as misjudgment or omission, which may affect the accuracy of the test results. For projects with frequent software requirement changes, continuously iterating user interfaces, or containing complex logics, it is difficult for manual testing to cope efficiently. In recent years, the development of intelligent cockpit technology has been very rapid, and automated testing means have also developed. Some enterprises create functions that simulate various manual operations by writing code, and then implement automated testing through the functions, significantly improving the testing efficiency.

[0036] However, during the testing process, there are mainly three major problems in the current technology: 1. The screen of the center console may change due to iteration (for example, the UI designer temporarily modifies the shapes, colors, etc. of some element icons), and the test cases are not updated in time, resulting in the in-vehicle infotainment system not knowing where the element objects to be operated by the test cases are when executing the automated test cases, unable to accurately perform interaction operations, making the test results inaccurate and delaying the test progress.

[0037] 2. The test cases are written entirely in code, which has high requirements for personnel, increasing the recruitment and learning thresholds for testers and being unfavorable to the flexibility of testing.

[0038] 3. Creating test cases by simulating various manual operation functions requires building various software environments on the in-vehicle infotainment system in advance to meet the working requirements of its automated testing software framework, and the deployment of the testing software environment is too complex.

[0039] To solve the above problems, according to an embodiment of the present invention, a testing method for a vehicle center console is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0040] In this embodiment, a testing method for a vehicle center console is provided, which is applied to a host computer, and the host computer is communicatively connected to the vehicle. Figure 1 It is a flowchart of a testing method for a vehicle center console according to an embodiment of the present invention, and this process includes the following steps: Step S101: Obtain the vehicle center console projection screen. The vehicle center console projection screen is the screen projected onto the host computer corresponding to the vehicle center console screen, and the vehicle center console screen is the screen displayed on the vehicle center console. Specifically, the vehicle center console test method provided in the embodiments of the present invention is applied to the host computer, which refers to devices such as personal computers, servers, and tablet computers that can establish a communication connection with the vehicle and include a processor. Thus, the software environment for testing is mainly deployed in the host computer, and test instructions are sent to the vehicle through the host computer to complete remote testing of the vehicle.

[0041] Before the test, the preparation work is to project the vehicle center console screen onto the screen of the host computer to obtain the vehicle center console projection screen. In this embodiment, the vehicle center console screen refers to the screen displayed on the vehicle center console screen, and the vehicle center console projection screen refers to the screen displayed on the host computer, projected by the vehicle, and identical to the vehicle center console screen.

[0042] Step S102: Obtain and execute the test case to output a test instruction to the vehicle, so that the vehicle executes the test operations included in the test case in the vehicle center console screen according to the test instruction.

[0043] In the embodiments of the present invention, a test case refers to a set of information including test inputs, execution conditions, and expected results designed to verify the software functions or requirements of the vehicle center console, used to determine whether the system meets specific requirements. The test case is the core tool for software testing, converting test behaviors into quantifiable and manageable patterns. In the embodiments of the present invention, test cases can be obtained by engineers based on traditional code programming means, or can be created through the test case recording method and graphical programming means provided in the subsequent embodiments of the present invention. The specific creation method will be described in detail in the subsequent embodiments.

[0044] In this step, the host computer obtains the already created test case and outputs a test instruction to the vehicle by executing the test case, so that the vehicle responds to the test instruction and simulates the user to perform corresponding test operations in the vehicle center console screen. For example: click a group of application icons in sequence and record whether each application can be correctly started; open and close a certain page 100 times repeatedly to determine whether a crash problem will occur. Such test operations, as long as they can be used to test the reliability of the vehicle center console, can be included in the test case, and the embodiments of the present invention will not elaborate further.

[0045] Step S103: Receive the test loss message fed back by the vehicle. The test loss message is a message generated when the vehicle cannot find the first target element corresponding to the operation in the vehicle center console screen during the test operation.

[0046] Specifically, in actual applications, in addition to the upgrade and iteration of in-vehicle computer performance, there is often also a design iteration of the user interface to give users a brand-new feeling and brighten their mood. However, after modifying the vehicle's central control screen, since the interface designers and test engineers do not belong to the same department, there is rarely any alignment work for engineering projects, resulting in the test cases not being updated in real time, and further leading to problems such as test failures or test interruptions. For example, the color, ID, name, or shape of a certain screen icon A has been modified, and the test case records clicking on icon A, while also recording the original attributes of icon A, including the original color, original shape, original name, and original ID, etc. Because the element has been modified, it is impossible to match a 100% compliant target through operations such as image matching, ID matching, and name matching. When the test case reaches the step of clicking on icon A, the problem of losing the target occurs, and thus the first target element corresponding to the operation cannot be found.

[0047] Based on this, when the vehicle executes a test instruction and cannot find the first target element corresponding to the operation in the vehicle's central control screen, a test loss message is generated, and at the same time, the test loss message is returned to the host computer. The test loss message includes the lost first target element and the attribute information related to the first target element.

[0048] Step S104, in response to the test loss message, call the large model to extract the element meaning of the first target element.

[0049] Step S105, use the large model to locate the second target element corresponding to the element meaning in the vehicle's central control screen, and correct the test case based on the second target element. The second target element is the screen element obtained after adjusting the first target element in the vehicle's central control screen; Step S106, continue the test with the corrected test case.

[0050] Specifically, after receiving the test loss message, the host computer knows from the test loss message that the first target element in the test case cannot be found. The reason may be that the vehicle's central control screen has been adjusted, while the test case has not been updated synchronously. At this time, the host computer adopts a large model fallback strategy to correct the test case.

[0051] In this embodiment, the large model refers to an AI large model, which is a type of artificial intelligence model with a large number of parameters constructed by artificial neural networks. It is usually pre-trained on a large amount of data through self-supervised learning or semi-supervised learning, and then its performance and capabilities are further optimized through methods such as instruction tuning and human alignment. The large model has characteristics such as a large number of parameters, large training data, and large computing resources, and has the capabilities of solving general tasks, following human instructions, and performing complex reasoning. In the embodiments of the present invention, the large model includes but is not limited to large language models, vision large models, multi-modal large models, and basic science large models.

[0052] In the embodiments of the present invention, the large model is used to determine the element meaning of the first target element in the test case. For example, the function is analyzed based on the icon of the first target element. For example, the meaning of the first target element represents meanings such as window closing, music playing, etc.

[0053] After that, since the central control screen of the vehicle has been projected onto the host computer, the host computer can also use the large model to analyze the vehicle central control screen projection image, determine the specific meanings of different elements in the current image, and determine whether there is a modified element whose meaning matches that of the first target element.

[0054] If a second target element that matches the element meaning is found, it is considered that the second target element is the modified form of the first target element. For example, the meaning represented by a modified icon is window closing, which is the same as the meaning of the first target element. Thus, the host computer can use the second target element to correct the original test case, obtain a new test case, and continue the test based on the new test case.

[0055] According to the above technical means, when performing vehicle central control tests through test cases, if the first target element corresponding to the operation cannot be found in the vehicle central control screen, the AI large model is called. Through the image understanding ability and semantic understanding ability of the AI large model, first understand the element meaning of the first target element to be controlled by the test operation, and then analyze and understand the current vehicle central control screen, match the second target element with the same element meaning as the first target element, and then use the second target element as the first target element to continue the test. At the same time, the test case is corrected accordingly according to the second target element. This not only ensures that the test can continue accurately and reliably, solves the problem of test interruption, improves the test accuracy rate, but also automatically modifies the test case according to the fallback function of the AI large model, enabling the vehicle central control screen to adaptively change automatically after the modified test case, reducing the steps for users to manually modify the test case frequently and reducing the labor cost. In addition, the test tool is installed in the host computer, and the test operation is executed in the host computer. The vehicle head unit only needs to respond to the instructions of the host computer to execute actions, without the need to configure a complex software framework for interaction, reducing the difficulty of deploying the test software environment for the vehicle head unit.

[0056] In some alternative embodiments, the steps of creating a test case include: Step a1, detecting the current operation of the user on the vehicle center console projection screen; Step a2, recording the element attribute information of the area corresponding to the current operation, and capturing the area cut map of the area corresponding to the current operation; Step a3, extracting the text information in the area cut map; Step a4, generating the current test instruction based on the current operation and the element attribute information; Step a5, saving the current test instruction, the area cut map, and the text information as the recording information of the current operation; Step a6, sorting out the recording information of each step of the user's operation in sequence to obtain a test case.

[0057] Specifically, the embodiment of the present invention provides a method for creating a recorded test case, which combines screen recording and use case generation means.

[0058] As Figure 2 shown, first, since the vehicle center console screen of the vehicle has been projected onto the host computer, the user can directly perform relevant operations on the vehicle center console projection screen in the host computer, such as clicking on an icon, dragging a window, etc. The process of recording the user's operation can be achieved by capturing mouse events through the Windows API, which is a specific prior art and will not be elaborated in this embodiment. However, since the vehicle center console projection screen is a projected screen, the user's operations will not receive a response from the host computer. Therefore, it is necessary to let the host computer know which elements and objects the user's interaction operations correspond to on the vehicle center console screen. Thus, the embodiment of the present invention realizes the effect of determining the user operation object by remotely extracting the interface element information of the vehicle center console screen. In this embodiment, the interface element information refers to the attribute information values of different elements in the vehicle center console screen. For example, for each icon that can be clicked and triggered in the interface, its corresponding specific position, ID, name, path, value, etc. are recorded.

[0059] According to the user's operation on the vehicle central control screen projection image on the host computer (for example, the mouse clicks on a certain area), locate the position of the user's operation, and then analyze whether the clicked position of the user includes operable elements according to the interface element information (for example, it is found that there is a triggerable icon in the current mouse click area), so as to record the element attribute information of the area corresponding to the current operation through the interface element information (for example, the ID, name, path, value, etc. of a specific icon in the mouse click area). By detecting each step of the user's operation and combining the element attribute information of the corresponding area, a test instruction (or called a test script) can be automatically generated. For example, the instruction is: click the close window element with ID XXX. Through this instruction, a test can be performed on whether the window can be closed normally.

[0060] During the user's operation, it is also necessary to capture a screenshot of the screen, cut the screenshot of the test click area, save it in each recorded operation, and at the same time convert the text in the picture into text through optical character recognition technology and write it into the recorded information. Finally, the recorded information corresponding to all the user's operations is sorted one by one according to the operation sequence, and a complete test case is integrated.

[0061] Through the technical solution provided by the embodiments of the present invention, each step of the user's operation action on the screen is recorded, and the specific position of each step of the operation is also recorded, so as to extract element attribute information, screenshots and text at the corresponding position. Through the element attribute information and operation actions, specific test instructions can be formed to indicate what operations are performed on which element object, realizing the conversion of recorded data into test cases in the form of instructions. Compared with the test case creation scheme of code programming, the test case recording method provided by the present invention significantly improves the speed of test case generation, reduces the difficulty of writing test cases, and reduces the employment threshold of testers. In addition, screenshots are taken in each step of the operation, and the text in the screenshots is saved at the same time. The saved screenshots and text are used to determine the specific meaning of the corresponding operation object in the area cut map. Even if the image designer makes fine adjustments to the screen of the central control screen later, the AI large model can still recognize the meaning of the originally operable element according to the screenshots and text, and then automatically correct the original test instruction according to the element meaning to improve the reliability of the test.

[0062] Based on the above steps for creating test cases, step S104 includes: Step b1, analyze the icons in the area cut map through the large model to obtain the first element meaning; Step b2, analyze the text information through the large model to obtain the second element meaning.

[0063] Further, step S105 includes: Step c1, search for the target icon that conforms to the first element meaning in the vehicle central control screen projection image through the large model; Step c2, when the target icon is found by searching according to the first element meaning, record the target element attribute information corresponding to the target icon. The target element attribute information and the target icon are used as the second target element; Step c3, modify the test instruction in the test case according to the target element attribute information; Step c4, use the target icon to replace the corresponding regional cutout; Step c5, when the target icon cannot be found by searching according to the first element meaning, use the large model to search for the target text that conforms to the second element meaning in the vehicle center control screen projection image; Step c6, determine the target icon based on the area where the target text is located, and return to the step of recording the target element attribute information corresponding to the target icon.

[0064] According to the above technical means, based on the process created by the foregoing test case, each operated first target element includes two parts of fallback information, namely the regional cutout and the text information. Thus, when the first target element is lost, the AI large model understands the meaning of the first target element from both the image and text perspectives, and then finds the second target element with the corresponding meaning from the current vehicle center control screen, further improving the accuracy of matching the second target element.

[0065] In the specific matching process of the above steps c1 - c5, first match the second target element through the image meaning of the regional cutout. If the image cannot be matched, then use the text meaning for matching. If the second target element cannot be matched in both cases, manual processing is required. As long as the second target element is matched through either the image meaning (the first element meaning) of the regional cutout or the text meaning (the second element meaning) of the text information, the embodiment of the present invention locates the target icon corresponding to the second target element, then takes a new screenshot of the target icon, and uses the new screenshot to replace the old screenshot. At the same time, find the target element attribute information in the element structure tree using the new screenshot area, and adjust the test instruction in the test case using the new attribute parameters, realizing an automated test case correction process. The dual - meaning matching ensures the accuracy of the automated test case correction.

[0066] In some alternative embodiments, the host computer and the vehicle are connected through the Android test bridge. The above step a2 includes: Step d1, create an element structure tree; Step d2, send a command to obtain the user interface layout to the vehicle at a preset period; Step d3, send a command to obtain the interface file to the vehicle at a preset period; Step d4: Receive the interface file fed back by the vehicle. The interface file is the file sent to the host computer by the vehicle each time it responds to the interface file command. The interface file is generated by the vehicle based on the command to obtain the user interface layout. The interface file stores the interface element layout information, which is the information extracted from the vehicle's central control screen and used to represent the screen elements in the vehicle's central control screen. Step d5: Parse the interface file to obtain the interface element layout information. Step d6: Populate the interface element layout information into the element structure tree for display. Step d7: Search for the element attribute information in the element structure tree according to the position of the area corresponding to the current operation.

[0067] Specifically, the embodiment of the present invention provides a specific implementation means for obtaining the element attribute information, which is used to locate what specific element object is operated in each step of the user's operation on the projection screen.

[0068] In the embodiment of the present invention, first, a GUI (Graphical User Interface) development tool is used to create an ElementFrame window (a frame element for organizing and displaying GUI components), which is used to display the interface of the element structure tree. A class named JTreeElement is created to inherit the JTree class to display all the contents of the element structure tree. JTree is a component in the Java Swing library and is used to display tree-structured data in a graphical user interface. Then, a JListElement component is created to inherit JList. JList is a component in the Swing framework and is used to display a list, so as to display the detailed information of the element selected by the user's operation. Through the above creation, a complete element structure tree is obtained.

[0069] In other words, the interface element layout information of the vehicle's Android system is obtained from the in-vehicle device to the host computer. The interface element layout information records the specific positions and attributes such as ID, name, path, and value of each operable object in the vehicle's central control screen. For example Figure 3 As shown, the interface element layout information will be populated in the pre-created element structure tree. When the user clicks on an icon with the mouse, a search is performed in the element structure tree according to the position where the user clicks the mouse to search for the specific attributes corresponding to the icon, and then the specific attributes corresponding to the icon can also be displayed through the JListElement component.

[0070] Such as Figure 4As shown, the interface element layout information is obtained from the vehicle and filled into the element structure tree on the host computer. The specific implementation process is to write a loop logic on the host computer to obtain the list of in-vehicle device connected to the current computer every 1 s. The vehicle is installed with the Android system and switched to the Android test bridge mode. At this time, the vehicle information will be automatically reported to the in-vehicle device list. The host computer can create a subprocess at a preset period (for example, every 1 s), and send the command "adb shell uiautomator dump / sdcard / ui.xml" for obtaining the user interface layout to the vehicle, so that the vehicle runs "adb shell uiautomator dump / sdcard / ui.xml". The vehicle can obtain the hierarchical information of all controls of the current application in the screen and save it in the interface file ui.xml in the sdcard storage device in XML format. Similarly, the host computer can send the command "adb pull / sdcard / ui.xml" for obtaining the interface file to the vehicle at a preset period, so that the vehicle transfers the interface file ui.xml from the in-vehicle sdcard to the host computer. Finally, the host computer obtains the desired interface element layout information by parsing the ui.xml file. In the parsed XML structure, specific UI elements can be found according to the attributes of the elements (such as resource-id, text, class, etc.). In the embodiment of the present invention, by comparing the interface element layout information obtained in the next period with the interface element layout information obtained in the previous period, it is determined whether the current screen has changed. When it is monitored that the in-vehicle display screen has changed, the interface of the element structure tree immediately updates the latest obtained interface element layout information to complete real-time synchronous refresh.

[0071] According to the above technical means, the host computer remotely obtains the interface element layout information of the vehicle center control screen based on the Android test bridge, and updates the interface element layout information to the element structure tree in real time. Therefore, when the screen changes, the layout and attributes of each element in the screen are updated to the element structure tree for the first time, and various attribute information in the interface is displayed to the user through the element structure tree, so that the user and the host computer can clearly understand what the operation object corresponding to each test operation of the user is, ensuring the accuracy of test case creation.

[0072] In some optional embodiments, the steps of creating a test case further include: Step g1, obtaining the vehicle center control screen projection and the element structure tree filled with the interface element layout information; Step g2, obtaining the test method blocks, where different test method blocks are used to map different test functions, and the test functions are used to pre-write the instructions of the test method; Step g3: Receive the user's tile operation command, and arrange the test method tiles in the order of the test process in the use case display area in response to the tile operation command. Step g4: Receive the user's parameter setting command, and fill in the test parameters for the arranged test method tiles in response to the parameter setting command. The parameter setting command is initiated by the user according to the interface element layout information in the element structure tree. Step g5: Convert the arranged test method tiles and test parameters into test cases.

[0073] Specifically, based on the vehicle center console screen mirroring image and the element structure tree obtained in the foregoing steps, the embodiment of the present invention further provides a specific solution for creating test cases through graphical programming.

[0074] Through the mechanism of test method tiles, test methods that can be used by the user are provided. For example, the user can directly call the test method tiles to perform specific operations such as clicking, dragging, and taking screenshots of certain icons on the vehicle center console screen mirroring image.

[0075] In the embodiment of the present invention, common test operations are encapsulated through test method tiles. Different test method tiles are used to map different test functions, and different test functions are used to pre-write instructions for different test methods. From the user's perspective, each test method tile is just a graphical or text name. The user only needs to click different test method tiles in a certain order to sort the test method tiles, thereby generating a corresponding test step script. For example, if the user double-clicks the test method tile representing "dragging" first and then double-clicks the test method tile representing "clicking", a test script that first drags a certain icon and then clicks a certain icon is automatically generated, thereby automatically completing the sorting of the test method. At this time, the corresponding test parameters need to be further filled in to determine which interface elements to operate, so as to complete the creation of the test case.

[0076] In a specific application embodiment, the test method tiles can be generated through the following process: 1. Encapsulate different test methods into callable functions; 2. Create a database; 3. Create a test method table in the database, and fill in the test function names corresponding to the callable functions in the test method table to establish the mapping relationship between the test method table and different test methods; 4. Render the test method table in the tile panel for display.

[0077] Specifically, create a MySQL database, for example, called Case in this embodiment. A test method table is created in the Case database to record the names of different test methods. In addition, the test method table can also include the ID of the test method, creation time, sorting, etc. Implement the corresponding method functions through programming code to obtain callable functions, define the methods with the same name in the test method table, and establish the mapping relationship between the test method table and the callable functions corresponding to different test methods.

[0078] After that, create a ListFrame window for displaying different test method tiles as the tile panel. In addition, the ListFrame window can also create a JList interface that inherits the functions of the JList component (a component in Java for displaying a list of elements) to display the content of different callable functions and explain the test methods corresponding to each callable function. Through the above steps, render the test method table in the tile panel for display, and the creation of the test method tiles can be completed.

[0079] Based on the above technical means, the present invention provides a technical solution for an automated operation test method, and implements the related functions of each test method tile one by one at the backend of the tool. The front-end interface presents the pre-implemented test method tiles to the user in the form of a list. In this way, the development cost of automated script writers is greatly reduced, the quality of automated scripts is improved, and the automated test runs more stably and efficiently.

[0080] Based on the test method tiles provided in the foregoing steps, the user can give tile operation commands such as clicking and dragging on the upper computer screen, and add the test method tiles to the use case display area in the order of operation, so as to achieve graphical test case programming.

[0081] For example, in a specific embodiment, based on the above ListFrame window, the mouseClicked function can be overridden in the JList interface class. When it is determined that the number of mouse clicks is equal to 2, the name of the callable function is passed into the use case display area to display the test method name and parameters. In addition, a custom component JTextFieldSearch can be inherited from the JTextField class to support users to input text to search for related callable functions. At the same time, create a JComboBoxCase component that inherits the JComboBox class to support users to select related callable functions in the form of a drop-down list.

[0082] After starting the test application, the ListFrame window reads the shortcut callable functions and all callable functions from the Case database and renders them on the tile panel. In the embodiment of the present invention, the shortcut callable functions and all callable functions can be displayed on two different pages. Among them, the shortcut callable functions represent the most commonly used test methods by users, such as click, assertion, swipe, input text, etc.; all callable functions are used to display all the automated test methods that users can use. In addition, the embodiment of the present invention also classifies all callable functions. The specific classifications include image operation, assertion, system, CAN, device operation, control click, etc., so as to facilitate users to quickly find the callable functions they need.

[0083] In a specific application embodiment, a window can be formed by creating an EditFrame class that inherits from the JFrame class as a use case display area. Create a component JTextFieldCase that inherits from the JTextField class to display the script text edited by the user. The use case display area can be further designed into 7 areas, namely the view area, the code area, the property area, the use case information area, the pre-stage area, the use case step area, and the post-stage area.

[0084] The view area is the graphical programming area. Users can manually select callable functions and add them to the view area, such as click operation, pause operation, assertion operation, zoom operation, drag operation, swipe operation, screenshot operation, etc. At the same time, it also supports manually filling in the target value and remarking use case information (personnel name, use case level, use case description information, in-vehicle information, affiliated business module). Even those without programming foundation can write automated use case scripts, which reduces the technical threshold requirements for automated testers and improves the automated writing efficiency.

[0085] The code area supports writing automated script use cases in the traditional way of writing code. In this embodiment, a script hint function is added. When writing, it will automatically associate the callable functions that may be needed. Users only need to press Enter or double-click to select the target method to use it easily. At the same time, it also supports customizing the script execution logic, allowing users to write some general methods to reduce duplicate code and making it more convenient and fast to use.

[0086] The property area and the use case area can display some property information and description information about the use case.

[0087] The pre - stage area, use - case step area, and post - stage area are three areas that mainly function during the execution of use cases. The pre - stage area is used to define the pre - requisite steps necessary before the execution of test cases, such as general operations like powering on the machine and checking the status of components. The use - case step area is used to display the completed test cases, which are the final versions of use cases and can be directly executed. The post - stage area is mainly used to display the fallback code edited by the user. That is, regardless of any problems that occur during the test process, it is possible to jump to the post - stage area and execute the fallback code to ensure the normal end of the test - case execution process.

[0088] In summary, another test - case creation technical solution provided by the embodiments of the present invention encapsulates the callable functions corresponding to the automated testing method, including all operations that can be used in the automated script, and displays them in the form of blocks on the panel for easy use at any time. When using, the user expands the block panel to view the callable functions they need, or can also input some relevant words in the search box to associate with the corresponding callable functions. By double - clicking on the specified target callable function, the target callable function will be automatically displayed in the use - case display area. It is also possible to manually input the specified callable function in the use - case display area to write the automated - script use cases. At the same time, the embodiments of the present invention also provide detailed note information and usage - method descriptions for each callable function, and display them through the JList interface. The technical solution provided by the embodiments of the present invention significantly reduces the development cost of automated - script writers, improves the quality of automated scripts, and makes the automated testing run more stably and efficiently.

[0089] According to the test operations the user wants to perform, the user can put the corresponding test - method blocks into the use - case display area by clicking or dragging, and convert them into callable functions. The callable functions in the use - case display area are written into test scripts. However, the callable function only corresponds to the operation, but it is not yet known which element to operate on specifically. At this time, in combination with the prompt information of the aforementioned element structure tree, the user gives the data displayed by the element structure tree as test parameters to the corresponding test - method blocks, that is, fills in the test parameters into the corresponding callable functions to inform what the interface element that the test method should operate on is. Finally, the arranged test - method blocks and test parameters are integrated and converted into test cases. Through the technical solution provided by the present invention, on the one hand, the function of graphical programming is realized, enabling users without programming foundation to write automated - use - case scripts, reducing the technical - threshold requirements for automated testers, and improving the automated - writing efficiency. On the other hand, based on the method of car - machine screen mirroring, the test screen is mirrored to the upper computer for use - case writing, and each element parameter of the car machine is accurately extracted, avoiding installing too many third - party test - software frameworks on the car machine and reducing the deployment complexity of the use - case development environment.

[0090] In some optional implementation manners, the above - mentioned step S101 includes: Step e1: Send a screen acquisition message to the vehicle; Step e2: Receive the vehicle's central control screen mirroring image. The vehicle's central control screen mirroring image is an image that, after the vehicle receives the screen acquisition message, captures screenshots of the central control screen at a preset frame rate, compresses the screenshots, and then removes duplicates based on the similarity between adjacent frame screenshots before sending.

[0091] Specifically, the traditional device screen mirroring solution takes high-frequency screenshots of the original device and continuously provides them to the user. This can cause high screen mirroring latency, slow response, and interface display lag due to the large size and quantity of the pictures. The solution provided by the embodiments of the present invention captures the screen image, compresses it while acquiring, reduces the picture resolution, effectively improves the transmission speed, and solves the problem of interface display lag. For example, the embodiments of the present invention specifically convert the picture into a format with a lower compression ratio (such as compressing it into the JPEG format instead of the PNG or BMP format with a higher compression ratio). When saving the JPEG picture, the color depth of the picture can be further reduced, from 24-bit color to 8-bit, but it cannot be too low otherwise it will cause picture distortion. Preprocessing the picture before compression, such as noise reduction and sharpening, can also improve the visual effect after compression.

[0092] Among them, for the screenshot operation of the vehicle's central control screen, the present invention sends a screen acquisition message to the vehicle through the host computer. The vehicle responds to the sent screen acquisition message by calling the JNI interface of the Android system to instantiate the SurfaceControl object, and then calls the screenshot method therein to implement functions such as background screenshot, full-screen screenshot, and specified area screenshot. This screenshot solution does not require permission from the user and there is no pop-up display. It is powerful and has a simple calling method. The screenshot speed is very fast, with an average time of 20 ms per picture, which is more than 3 times faster than the traditional UiDevice screenshot method.

[0093] In addition, by using the OpenCV image comparison algorithm, duplicate pictures are removed, and the frame rate is controlled at about 13, further reducing the number of redundant pictures. This can not only display the car machine screen in real time without delay but also meet the basic viewing screen mirroring effect. The host computer converts the received pictures into binary blob format files and transfers them to the playback application on the PC desktop through the Socket network interface to play the pictures in real time and restore the actual display effect of the car machine.

[0094] For example, in a specific embodiment, the embodiment of the present invention uses a structural similarity algorithm to first calculate the average brightness of two adjacent frames of images before and after, then calculate the contrast (i.e., standard deviation) of the two images, and then calculate the covariance of the two images according to the image structure (reflecting the structural information). Finally, the similarity between the two is determined by combining the comparison results of the three indicators of brightness, contrast, and covariance. When the similarity of the three indicators of the two pictures reaches 100%, the duplicate and redundant pictures will be removed.

[0095] In addition, in an alternative embodiment, the compressed screenshot includes a heavily compressed area and a lightly compressed area. The heavily compressed area is the area where the vehicle compresses the unimportant area in the screenshot (such as the area without icons) at the first compression rate, and the lightly compressed area (such as the area including icons) is the area where the vehicle compresses the important area in the screenshot at the second compression rate, where the first compression rate is less than the second compression rate. For example, the first compression rate is 50% and the second compression rate is 90%, indicating that the data volume of the area compressed at the first compression rate is smaller. Through the partition compression scheme provided by the embodiment of the present invention, the transmission speed is improved while effectively maintaining the image quality.

[0096] In some alternative embodiments, the vehicle center console testing method provided by the present invention further includes: Step f1, sending an audio message acquisition request to the vehicle; Step f2, receiving the vehicle audio feedback by the Android test bridge. The vehicle audio is created by the vehicle in response to the audio message to create an audio recording object, reads the vehicle background audio data through the audio recording object, and then compresses the read audio data into a playable format file and sends it to the host computer through the Android test bridge; Step f3, transmitting the vehicle audio to the audio playback application through local inter-process communication for playback.

[0097] Specifically, according to the above technical means, the host computer also sends an audio message to the vehicle, so that the vehicle responds to the audio message, and based on the technical means of the Android debug bridge, transmits the vehicle audio from the background to the host computer, and plays the vehicle center console audio through the host computer, so that the accuracy of certain tests can be further verified through the vehicle audio. And the vehicle audio stream is directly transmitted from the background. Compared with audio recording means such as external microphones, the audio quality is higher and the usage threshold is reduced.

[0098] In a specific embodiment, as Figure 5As shown, when the vehicle receives an audio message, the specific implementation process is as follows: First, create an AudioRecord object (audio recording object) on the vehicle side to perform audio recording. AudioRecord is one of the recording APIs provided by Android for reading device audio. Next, create a buffer on the vehicle side to store subsequent audio data. Create a background Service to repeatedly use the read method of AudioRecord to read audio data from the AudioRecord object and load the read audio stream into the buffer until the audio disappears. Then, the Service calls the stop method of AudioRecord to stop recording, and finally calls the release method of the AudioRecord object to release other related resources.

[0099] Next, the Service uses the MediaCodec class provided by the Android system (a component for audio encoding) to compress the audio stream into AAC (Advanced Audio Codec) format audio in real time. Then, by using the commands of the Android test bridge, the vehicle system port is redirected to the host computer, and the vehicle audio is sent to the host computer. After receiving the vehicle audio, the host computer uses local Socket communication (a mechanism for local inter-process communication) to transfer the audio stream to the front-end application of the computer client in real time. The front-end application uses the play method of the MediaPlayer class to parse and play the audio, so as to realize the real-time playback of the vehicle system audio on the host computer.

[0100] In some alternative embodiments, step S102 includes: Step h1, execute the initialization phase of the test case. The initialization phase is the information management phase before the execution of the test case; Step h2, if no exception occurs in the initialization phase, execute the precondition phase. The precondition phase includes the necessary precondition steps before the execution of the test case; Step h3, if no exception occurs in the precondition phase, execute the main phase. The main phase includes the test method of the test case; Step h4, when an exception occurs during the execution of the initialization phase, precondition phase, and main phase, jump to execute the postcondition phase. The postcondition phase includes instructions to end the execution process of the test case normally.

[0101] Specifically, the initialization phase is the information management phase before the execution of test cases. For example, various attributes of test cases, test environment, test hardware information, etc. need to be collected before testing to facilitate subsequent verification of the test report to understand the overall test process and test scenarios. The precondition phase is used to define the necessary pre-steps before the execution of test cases, such as general operations like powering on the machine and checking the component status. The main phase is used to display the test cases that have been created and are the complete versions of the use cases that can be directly executed. The post-phase is mainly used to display the fallback code edited by the user. That is, regardless of any problems that occur during the test process, it can jump to the post-phase area and execute the fallback code to make the test case execution process end normally. It can be used for fallback operations in abnormal situations. Thus, for any problems that occur during the test process, it can automatically jump to the post-phase area and execute the fallback code to make the test case execution process end normally (for example, the fallback code is to end all processes and shut down the machine), thereby significantly improving the reliability of the test and solving the problem of abnormal non-exit.

[0102] For the convenience of understanding the overall solution, as Figure 6 shown, from creating use cases to executing use cases, a complete test process is as follows: I. Preparation phase: 1. Set the in-vehicle computer to the Android Debug Bridge mode.

[0103] 2. Connect the in-vehicle computer cable to the host computer.

[0104] 3. The host computer refreshes the device list and displays the currently connected in-vehicle computer.

[0105] 4. The host computer clicks the screen mirroring button to display the current content of the in-vehicle computer interface in real time.

[0106] 5. The user operates the in-vehicle computer on the host computer and clicks to play audio. At this time, the computer will play the sound of the in-vehicle computer.

[0107] 6. The user views the element structure tree panel on the host computer, and at this time, the element content of the current in-vehicle computer interface will be displayed in real time.

[0108] 7. When the user clicks on the in-vehicle computer interface, the element structure tree will automatically refresh to the current latest element content.

[0109] 8. When the user clicks on a certain position of the vehicle center console screen mirroring image with the mouse, the element structure tree area will display the current area element attribute values in real time.

[0110] II. Use case creation phase: 9. Create a new test case project.

[0111] 10. Record test cases according to user operations.

[0112] III. Use case execution phase: 11. After the recording of the test cases is completed, execute the test cases.

[0113] 12. When operating a certain element, first match through the element attributes. If the match is successful, execute the operation.

[0114] 13. If the element attribute match fails and the target is lost, call the large model to analyze the meaning of the element icon. If the match is successful through the icon meaning, execute the operation.

[0115] 14. If the icon meaning match fails, continue to call the large model to analyze the meaning of the element text. If the match is successful using the text meaning, execute the operation.

[0116] 15. If all matches fail, it means the test case match fails and the test case needs to be regenerated.

[0117] In this embodiment, a test device for a vehicle center console is further provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0118] This embodiment provides a test device for a vehicle center console, as Figure 7 shown, including: A screen mirroring module 601, configured to obtain the screen mirroring of the vehicle center console. The screen mirroring of the vehicle center console is a screen corresponding to the vehicle center console screen and projected on the host computer, and the vehicle center console screen is the screen displayed on the vehicle center console; A test case execution module 602, configured to obtain and execute test cases to output test instructions to the vehicle, so that the vehicle executes the test operations included in the test cases in the vehicle center console screen according to the test instructions; A target loss module 603, configured to receive the test loss message fed back by the vehicle. The test loss message is a message generated when the vehicle cannot find the corresponding first target element for the operation in the vehicle center console screen during the test operation; A lost target analysis module 604, configured to, in response to the test loss message, call the large model to extract the element meaning of the first target element; A lost target detection module 605, configured to locate the second target element corresponding to the element meaning in the vehicle center console screen through the large model and correct the test case based on the second target element. The second target element is a screen element obtained by adjusting the first target element in the vehicle center console screen; A fallback test module 606, configured to continue the test with the corrected test case.

[0119] In an alternative embodiment, a test case is created through the following steps: Detect the current operation of the user on the vehicle's center console screen mirroring screen; record the element attribute information of the area corresponding to the current operation, and capture the area cutout of the area corresponding to the current operation; extract the text information from the area cutout; generate the current test instruction based on the current operation and the element attribute information; save the current test instruction, the area cutout, and the text information as the recording information of the current operation; and organize the recording information of each step of the user's operation in sequence to obtain the test case.

[0120] In an alternative embodiment, the host computer and the vehicle are connected through an Android test bridge, and the element attribute information is obtained through the following steps: Create an element structure tree; send a command to obtain the user interface layout to the vehicle at a preset period; send a command to obtain the interface file to the vehicle at a preset period; receive the interface file fed back by the vehicle, where the interface file is the interface element layout information extracted from the vehicle's center console screen each time the vehicle responds to the command to obtain the user interface layout, and then save the interface element layout information as the interface file, and then send the interface file to the host computer in response to the command to obtain the interface file. The vehicle's center console screen is the screen displayed on the vehicle's center console corresponding to the vehicle's center console screen mirroring screen; parse the interface file to obtain the interface element layout information; fill the interface element layout information into the element structure tree for display; and find the element attribute information from the element structure tree according to the position of the area corresponding to the current operation.

[0121] In an alternative embodiment, the missing target analysis module 604 includes: A graphic meaning analysis unit for analyzing the icons in the area cutout through a large model to obtain the first element meaning; A text meaning analysis unit for analyzing the text information through a large model to obtain the second element meaning.

[0122] In an alternative embodiment, the missing target detection module 605 includes: A similar icon search unit for searching for target icons that conform to the first element meaning in the vehicle's center console screen mirroring screen through a large model; A second element positioning unit for recording the target element attribute information corresponding to the target icon when the target icon is searched through the first element meaning. The target element attribute information and the target icon are used as the second target element; A test instruction correction unit for modifying the test instruction in the test case through the target element attribute information; An icon replacement unit for replacing the corresponding area cutout with the target icon; A similar text search unit for searching for target text that conforms to the second element meaning in the vehicle's center console screen mirroring screen through a large model when the target icon cannot be searched through the first element meaning; Return to the second element positioning unit, which is used to determine the target icon based on the area where the target text is located, and return the step of recording the target element attribute information corresponding to the target icon.

[0123] In some alternative embodiments, the screen mirroring module 601 includes: A screen acquisition instruction unit, which is used to send a screen acquisition message to the vehicle; A compression and duplicate removal screen mirroring unit, which is used to receive the vehicle center console screen mirroring image fed back by the vehicle. The vehicle center console screen mirroring image is an image that the vehicle takes a screenshot of the center console screen at a preset frame rate after receiving the screen acquisition message, compresses the screenshot, and then removes duplicates from the screenshots according to the similarity between adjacent frame screenshots and sends it.

[0124] In some alternative embodiments, the compressed screenshots include a heavily compressed area and a lightly compressed area. The heavily compressed area is the area where the vehicle compresses the unimportant area in the screenshot at a first compression rate, and the lightly compressed area is the area where the vehicle compresses the important area in the screenshot at a second compression rate. The first compression rate is less than the second compression rate.

[0125] In some alternative embodiments, the device further includes: An audio acquisition message unit, which is used to send an audio acquisition message to the vehicle; An audio receiving unit, which is used to receive the vehicle audio fed back by the vehicle through the Android test bridge. The vehicle audio is that the vehicle creates an audio recording object in response to the audio message, reads the vehicle background audio data through the audio recording object, then compresses the read audio data into a playable format file, and sends it to the host computer through the Android test bridge; A playback unit, which is used to transmit the vehicle audio to the audio playback application through local inter-process communication for playback.

[0126] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding above-mentioned embodiments, and will not be repeated here.

[0127] Please refer to Figure 8 , Figure 8 is a schematic structural diagram of a computer device provided by an alternative embodiment of the present invention, as shown in Figure 8As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 8 In the figure, a processor 10 is taken as an example.

[0128] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a generic array logic, or any combination thereof.

[0129] Among them, the memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiments.

[0130] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0132] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0133] Embodiments of the present invention also provide a computer-readable storage medium. The method according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0134] A part of the present invention can be applied as a computer program product, for example, computer program instructions. When executed by a computer, through the operation of the computer, the method and / or technical solution according to the present invention can be called or provided. Those skilled in the art should be able to understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.

[0135] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A test method for a vehicle center console, characterized in that, Applied to the host computer, which has established a communication connection with the vehicle, the method includes: Obtain the vehicle center console projection screen, which is the screen projected on the host computer corresponding to the vehicle center console screen, and the vehicle center console screen is the screen displayed on the vehicle center console; Obtain and execute the test case to output a test instruction to the vehicle, so that the vehicle executes the test operations included in the test case in the vehicle center console screen according to the test instruction; Receive the test loss message fed back by the vehicle, where the test loss message is a message generated when the vehicle cannot find the first target element corresponding to the operation in the vehicle center console screen during the execution of the test operation; In response to the test loss message, call the large model to extract the element meaning of the first target element; Locate the second target element corresponding to the element meaning in the vehicle center console projection screen through the large model, and correct the test case based on the second target element, where the second target element is the screen element obtained by adjusting the first target element in the vehicle center console screen; Continue the test with the corrected test case.

2. The test method according to claim 1, wherein The steps of creating the test case include: Detect the current operation of the user in the vehicle center console projection screen; Record the element attribute information of the area corresponding to the current operation, and capture the area cut map of the area corresponding to the current operation; Extract the text information in the area cut map; Generate the current test instruction based on the current operation and the element attribute information; Save the current test instruction, the area cut map, and the text information as the recording information of the current operation; Sort out the recording information of each step of the user's operation in sequence to obtain the test case.

3. The method according to claim 2, wherein The host computer and the vehicle are connected through the Android test bridge. The recording of the element attribute information of the area corresponding to the current operation includes: Create an element structure tree; Send the command to obtain the user interface layout to the vehicle at a preset period; Send the command to obtain the interface file to the vehicle at a preset period; Receive the interface file fed back by the vehicle, where the interface file is the file sent to the host computer by the vehicle each time in response to the command to obtain the interface file. The interface file is generated by the vehicle based on the command to obtain the user interface layout, and the interface file stores the interface element layout information, which is the information extracted from the vehicle center console screen and used to represent the screen elements in the vehicle center console screen; Parse the interface file to obtain the interface element layout information; Fill the interface element layout information into the element structure tree for display; Find the element attribute information from the element structure tree according to the position of the area corresponding to the current operation.

4. The method according to claim 2, wherein The step of, in response to the test loss message, calling the large model to extract the element meaning of the first target element includes: Analyze the icons in the area cut map through the large model to obtain the first element meaning; Analyze the text information through the large model to obtain the second element meaning.

5. The method according to claim 4, wherein Locating a second target element corresponding to the meaning of the element in the vehicle center console projection screen by means of the large model, and correcting the test case based on the second target element, includes: Searching for a target icon that conforms to the meaning of the first element in the vehicle center console projection screen by means of the large model; When the target icon is found by searching according to the meaning of the first element, recording the target element attribute information corresponding to the target icon, and using the target element attribute information and the target icon as the second target element; Modifying the test instruction in the test case by means of the target element attribute information; Replacing the corresponding regional cut map with the target icon; When the target icon cannot be found by searching according to the meaning of the first element, searching for a target text that conforms to the meaning of the second element in the vehicle center console projection screen by means of the large model; Determining the target icon based on the area where the target text is located, and returning to the step of recording the target element attribute information corresponding to the target icon.

6. The method according to claim 1 or 2, characterized in that, The obtaining of the vehicle center console projection screen includes: Sending a screen acquisition message to the vehicle; Receiving the vehicle center console projection screen fed back by the vehicle, where the vehicle center console projection screen is an image that the vehicle takes a screenshot of the center console screen at a preset frame rate after receiving the screen acquisition message, compresses the screenshot, and then removes duplicates from the screenshots according to the similarity between adjacent frame screenshots and sends it; 7. The method according to claim 6, characterized in that The compressed screenshot includes a heavily compressed area and a lightly compressed area. The heavily compressed area is the area where the vehicle compresses unimportant areas in the screenshot at a first compression rate, and the lightly compressed area is the area where the vehicle compresses important areas in the screenshot at a second compression rate, and the first compression rate is less than the second compression rate.

8. The method according to claim 1, wherein The method further includes: Sending an audio acquisition message to the vehicle; Receiving the vehicle audio fed back by the vehicle through the Android test bridge, where the vehicle audio is that the vehicle creates an audio recording object in response to the audio message, reads the vehicle background audio data through the audio recording object, then compresses the read audio data into a playable format file, and sends it to the host computer through the Android test bridge; Transmitting the vehicle audio to an audio playback application through local inter-process communication for playback.

9. A test device for a vehicle center console, characterized in that, Applied to a host computer, the device includes: A projection module, configured to obtain a vehicle center console projection screen, where the vehicle center console projection screen is a screen projected on the host computer corresponding to the vehicle center console screen, and the vehicle center console screen is the screen displayed on the vehicle center console; A use case execution module, configured to obtain and execute a test case to output a test instruction to the vehicle, so that the vehicle executes the test operations included in the test case in the vehicle center console screen according to the test instruction; A target loss module, configured to receive a test loss message fed back by the vehicle, where the test loss message is a message generated when the vehicle cannot find a first target element corresponding to an operation in the vehicle center console screen when executing the test operation; A lost target analysis module, configured to, in response to the test loss message, call a large model to extract the element meaning of the first target element; The lost target detection module is used to locate the second target element corresponding to the element meaning in the vehicle center control screen through the large model, and correct the test case based on the second target element, where the second target element is a screen element obtained by adjusting the first target element in the vehicle center control screen; The fallback testing module is used to continue testing with the corrected test case.

10. A computer device, characterized in that, Comprising: A memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Computer instructions are stored on the computer-readable storage medium, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Comprising computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 8.

Images