Instruction testing method and apparatus

Abstract

This application discloses an instruction testing method and apparatus applied to an in-vehicle terminal. The in-vehicle terminal is communicatively connected to at least one display device. The method includes: acquiring a first interface image of at least one display device; determining the execution position of a target test instruction in the first interface image of at least one display device, wherein the target test instruction is a preset instruction to be tested; executing a preset operation of the target test instruction at the execution position; and acquiring the execution result of the target test instruction. This method can reduce maintenance costs during the testing process and enables instruction testing in multi-screen scenarios.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and includes, but is not limited to, an instruction testing method and apparatus. Background Technology

[0002] Before the vehicle terminal leaves the factory, in order to ensure that the vehicle terminal correctly executes the instructions given by the user, the vehicle terminal is usually tested during the testing phase, and then debugged based on the test results.

[0003] In related technologies, the technical solution usually involves accessing the interface control tree and determining the position of the control based on its identifier, text, and other relevant information, thereby enabling the testing of instructions. The interface control tree is a data structure in a graphical user interface that represents the hierarchical structure of interface elements. It describes the parent-child relationship and layout hierarchy of all controls (buttons, text boxes, containers, etc.) in an interface.

[0004] However, after the layout of the display interface changes, the preset operation of the test command needs to be changed. For example, if the original requirement was to touch the first preset position, it is changed to touch the second preset position, which requires updating the preset position. This results in a high maintenance cost for the test command, and this type of solution is not suitable for multi-screen scenarios. Summary of the Invention

[0005] In view of this, the instruction testing method and apparatus provided in this application can reduce maintenance costs during the testing process and can implement instruction testing in multi-screen scenarios. The instruction testing method and apparatus provided in this application are implemented as follows: One aspect of this application provides an instruction testing method applied to an in-vehicle terminal, the in-vehicle terminal being communicatively connected to at least one display device, the method comprising: Acquire a first interface image of at least one display device; Determine the execution position of the target test instruction in the first interface image of at least one display device, wherein the target test instruction is a preset test instruction; Execute the preset operation of the target test instruction at the execution location and obtain the execution result of the target test instruction.

[0006] In one embodiment, acquiring a first interface image of at least one display device includes: Identify the target display device that is connected to the vehicle terminal for communication, wherein the target display device is at least one of the display devices that is in a working state; Acquire the first interface image of the target display device.

[0007] In one embodiment, acquiring a first interface image of the target display device includes: Obtain the first interface image of each target display device using the debug bridge command; or... The first interface image of each target display device is obtained by capturing it with the target camera.

[0008] In one embodiment, obtaining the execution location of a target test instruction in a first interface image of at least one display device includes: The first interface image of the target display device and the target test command are input into the target model to obtain the execution position of the target test command. The target model is a model obtained by training with sample interface images and sample test commands.

[0009] In one embodiment, obtaining the execution location of a target test instruction in a first interface image of at least one display device includes: The first interface image of the target display device and the target test command are sent to the target server, which is a server with image recognition and command understanding capabilities. Obtain the execution location of the target test command from the target server.

[0010] In one embodiment, the execution location of the target test instruction is determined based on the target interface image in the first interface image of the target display device and the set of target pixels in the target interface image. The first interface image includes multiple images, and the target interface image is any one of the multiple first interface images.

[0011] In one embodiment, obtaining the execution result of the target test instruction includes: obtaining the content displayed by a second interface image, wherein the second interface image is the target interface image after the target test instruction is executed; and determining whether the target test instruction was successfully executed based on the content displayed by the second interface image.

[0012] In one embodiment, determining whether the target test instruction was successfully executed based on the content displayed in the second interface image includes: If the target information is included in the second interface image, it is determined that the target test instruction was executed successfully. The target information includes at least one of the following: target text, target graphics, and target video. If the target information is not included in the second interface image, it is determined that the target test instruction failed to execute.

[0013] In one embodiment, in the event that the target test instruction fails to execute, the method further includes: If the number of execution failures is less than or equal to a preset threshold, the execution location of the target test instruction is retrieved again and the target test instruction is executed. If the number of failed tests exceeds a preset threshold, a test failure message will be generated.

[0014] Another aspect of the embodiments of this application provides an instruction testing device, applied to an in-vehicle terminal, the in-vehicle terminal being communicatively connected to at least one display device, the device comprising: an image acquisition module, a location acquisition module, and an operation execution module; An image acquisition module is used to acquire a first interface image of at least one display device; The position acquisition module is used to determine the execution position of the target test instruction in the first interface image of at least one display device, wherein the target test instruction is a preset test instruction; The operation execution module is used to execute the preset operation of the target test instruction at the execution location and obtain the execution result of the target test instruction.

[0015] The vehicle terminal provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the method of this application.

[0016] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0017] The instruction testing method and apparatus provided in this application can acquire a first interface image of at least one display device; determine the execution position of a target test instruction in the first interface image of at least one display device, where the target test instruction is a preset instruction to be tested; execute a preset operation of the target test instruction at the execution position; and obtain the execution result of the target test instruction. Acquiring the execution position of the target test instruction in the first interface image of at least one display device allows for determining the position of the control to be operated, and thus enabling the execution of the target test instruction by executing the preset operation of the target test instruction at the execution position. This method of executing the target test instruction by determining the control position is adaptable to different display interfaces. When the display interface is updated, no additional maintenance costs are required. Furthermore, in the case of multiple display devices, the execution position of the target test instruction can be obtained for different display interfaces, thereby enabling the execution of the target test instruction accurately and efficiently. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the application scenario provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the instruction testing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the process for obtaining the first interface image provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the method of obtaining the first interface image provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating another method for obtaining the first interface image provided in the embodiments of this application; Figure 6 This is a schematic diagram of the input and output of the target model provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the interaction between the vehicle terminal and the server provided in the embodiments of this application; Figure 8 This is a schematic diagram of the execution location in the first interface image provided in the embodiments of this application; Figure 9 This is a flowchart illustrating the verification target test instructions provided in the embodiments of this application; Figure 10 This is a logical diagram of the target determination test instruction provided in the embodiments of this application; Figure 11 This is a schematic diagram of the overall process of the instruction testing method provided in the embodiments of this application; Figure 12 This is a schematic diagram of the instruction testing device provided in the embodiments of this application; Figure 13 This is a schematic diagram of the structure of the vehicle-mounted terminal provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0024] Before the vehicle terminal leaves the factory, in order to ensure that the vehicle terminal correctly executes the instructions given by the user, the vehicle terminal is usually tested during the testing phase, and then debugged based on the test results.

[0025] During the testing phase, it is necessary to test the execution of the target test instructions. In order for the vehicle terminal to execute the instructions given by the user normally, the solution adopted in related technologies is usually to access the interface control tree and determine the position of the control based on the control's identifier, text and other relevant information, thereby realizing the test of the instructions.

[0026] Alternatively, a baseline image of the control can be obtained by pre-screening and cropping. Template matching and feature-point-based matching algorithms are then used to find regions with a confidence level greater than a threshold as the control's location, thereby enabling command testing.

[0027] However, both of the above methods have certain problems. For example, since the display interface iterates very quickly and the interface layout may be updated frequently, if the above two methods are used, the operation of the test instructions needs to be changed after the layout of the display interface changes. This results in high maintenance costs during the instruction testing process.

[0028] Furthermore, both the use of interface control trees and cropping screenshots are only applicable to single-display interface scenarios. In the context of in-vehicle terminals, there may be multiple display devices such as central control screens, instrument panel screens, and rear seat screens, resulting in multiple display interfaces. The solutions in the relevant technologies are not suitable for multi-screen scenarios and are not compatible with in-vehicle terminals with multiple display devices.

[0029] To address the aforementioned problems in related technologies, this application provides an instruction testing method, the application scenarios of which will be explained below.

[0030] Figure 1 This is a schematic diagram of the application scenario provided in the embodiments of this application. Please refer to it. Figure 1 In this scenario, the device may include an in-vehicle terminal 110 and multiple display devices 120.

[0031] The vehicle-mounted terminal 110 can communicate with multiple display devices 120.

[0032] It should be noted that the vehicle terminal 110 can be installed inside the vehicle. It can be a fixed vehicle terminal or it can be replaced by other electronic devices. For example, it may include, but is not limited to, mobile phones, wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), tablets, laptops, vehicle terminals, PCs (Personal Computers), etc.

[0033] The function implemented by this method can be achieved by the processor in the vehicle terminal calling the program code. Of course, the program code can be stored in the computer storage medium. It can be seen that the vehicle terminal 110 includes at least a processor and a storage medium.

[0034] Display device 120 can be various screens in the vehicle terminal 110, such as: central control screen, instrument panel screen, passenger screen, rear seat screen, etc. Among them, the central control screen can be an integrated operation screen with the vehicle terminal 110, and users can interact with the vehicle terminal 110 through the operation of the central control screen. The central control screen can display various functional or entertainment content. The instrument panel screen can display basic vehicle information, such as fuel consumption, mileage, current speed, and other real-time basic information. The passenger screen and rear seat screen can be screens for entertainment of other passengers, such as playing games and movies.

[0035] It should be noted that, in the embodiments of this application, the display device 120 can be one type, such as a central control screen, or it can be a combination of any two or more display devices, such as a central control screen, an instrument panel, etc., without any specific limitation.

[0036] Figure 1 The diagram shows a configuration including the four types of display devices described above. In actual implementation, the number of display devices can be fewer or more, and is not limited to this. Figure 1 The quantities shown are as a limit.

[0037] The following section explains, in conjunction with the above application scenarios, one feasible implementation process of the instruction testing method provided in the embodiments of this application.

[0038] Figure 2 This is a flowchart illustrating the instruction testing method provided in the embodiments of this application. Please refer to... Figure 2 The method includes: S210: Acquire a first interface image of at least one display device.

[0039] It should be noted that the subject executing this method can be the aforementioned vehicle-mounted terminal.

[0040] The vehicle terminal can send commands to each display device to send back the corresponding first display interface, or it can use other devices, such as a camera, to take a picture of the screen of the display device and then process the image to obtain the first interface image.

[0041] It should be noted that in this embodiment, there may be one or more display devices. If there is one display device, the vehicle terminal can acquire the first interface image of that one display device; if there are multiple display devices, the vehicle terminal can acquire the first interface image of any one or more of the multiple display devices.

[0042] Optionally, the first interface image can be the image currently displayed on the display device, or it can be the desktop image of the display device. There are no specific restrictions here, and the corresponding settings can be made according to actual needs.

[0043] It should be noted that the aforementioned first interface image may be actively acquired by the vehicle terminal, or it may be actively transmitted to the vehicle terminal after being acquired by other devices such as display devices or cameras; no specific restrictions are imposed here.

[0044] In one embodiment, the vehicle terminal can also communicate with a server, or the server can instruct other devices such as a display device or camera to acquire the first interface image and then send it to the vehicle terminal.

[0045] S220: Determine the execution location of the target test instruction in the first interface image of at least one display device.

[0046] It should be noted that after the vehicle terminal obtains the first interface image, it can obtain the execution location of the target test instruction from the first interface image. The execution location can be determined by the vehicle terminal itself, or it can be determined by other devices, such as the aforementioned server, and then sent to the vehicle terminal. No specific restrictions are made here.

[0047] The target test command is a preset test command. For example, if the music playback function of the vehicle terminal needs to be tested, the target test command can be a music playback command; if the food ordering function of the vehicle terminal needs to be tested, the target test command can be a food delivery ordering command. No specific restrictions are imposed here.

[0048] In actual testing, corresponding target test instructions can be generated based on the specific test cases that need to be tested.

[0049] For each target test instruction, one or more execution locations in the first interface image are required. For example, for a music playback instruction, the location of the music playback application in the first interface image needs to be obtained; for a food delivery order instruction, the location of the food delivery order application in the first interface image needs to be obtained.

[0050] The execution location of the target test command can be determined from the first interface image via the vehicle terminal, or the server can determine the execution location of the target test command from the first interface image and then send it to the vehicle terminal; no specific restrictions are imposed here.

[0051] It should be noted that, as explained above, the first interface image can be the currently displayed image or the desktop image on the display device. If there is at least one display device, there can also be at least one first interface image. If there are multiple first interface images, the execution position of the target test instruction can be determined in any one of the first interface images, or the execution position of the target test instruction can be determined in each of the first interface images. In the subsequent actual execution process, one of the methods can be selected for execution.

[0052] S230: Execute the preset operation of the target test instruction at the execution location and obtain the execution result of the target test instruction.

[0053] It should be noted that after determining the execution location, the preset operation of the target test instruction can be executed. The preset operation can be a single operation or a group of operations. Each operation can be one of several types, such as single click, double click, drag, long press, etc., and a group of operations can be a combination of any two or more of the above operations.

[0054] It should be noted that after obtaining the execution location, the execution location of the target test instruction can be determined during the execution of the target test instruction. For example, if the preset operation is a click operation, the location of the click operation can be determined, and this location is the aforementioned execution location.

[0055] Since the execution location is a position within one of the first interface images, there may be multiple execution locations if there are multiple first interface images. You can select one of the interface images to execute the preset operation of the target test instruction at the execution location.

[0056] The instruction testing method provided in this application embodiment can acquire a first interface image of at least one display device; determine the execution position of a target test instruction in the first interface image of at least one display device, where the target test instruction is a preset instruction to be tested; execute a preset operation of the target test instruction at the execution position; and obtain the execution result of the target test instruction. Acquiring the execution position of the target test instruction in the first interface image of at least one display device allows for determining the position of the control to be operated, and thus enabling the execution of the target test instruction by executing the preset operation of the target test instruction at the execution position. This method of executing the target test instruction by determining the position of the control is adaptable to different display interfaces. When the display interface is updated, no additional maintenance cost is required. Furthermore, in the case of multiple display devices, the execution position of the target test instruction can be obtained for different display interfaces, thereby enabling the execution of the target test instruction accurately and efficiently.

[0057] The following explains one feasible implementation process for obtaining the first interface image provided in the embodiments of this application.

[0058] Figure 3 This is a schematic diagram of the process for obtaining the first interface image provided in the embodiments of this application. Please refer to... Figure 3 Acquiring a first interface image of at least one display device, including: S310: Identify the target display device that will communicate with the vehicle terminal.

[0059] It should be noted that although there may be multiple display devices in a vehicle, some display devices may be damaged or in a dormant state during actual implementation. In such cases, it is impossible to operate these display devices. Therefore, the target display device can be identified first.

[0060] The target display device is at least one display device that is in operation.

[0061] In one embodiment, the display device can be determined to be in a working state by various means such as whether the display device is powered on or whether the display device can communicate, thereby obtaining the target display device.

[0062] S320: Acquire the first interface image of the target display device.

[0063] It should be noted that after the target display device is determined, the vehicle terminal can acquire the first interface image of the target display device. The target display interface can be one or more display devices.

[0064] If there is a single display device, one first interface image can be obtained; if there are multiple display devices, a corresponding number of first interface images can be obtained. No specific limitation is made here.

[0065] The instruction testing method provided in this application embodiment can determine a target display device that is communicatively connected to an in-vehicle terminal; and acquire a first interface image of the target display device. The target display device is at least one display device that is in a working state. By determining the target display device, one or more display devices in a working state can be identified, and thus the first interface images of these display devices can be acquired, avoiding the problem of first interface image acquisition failure.

[0066] There are several ways to acquire the first interface image. Two of these methods will be explained below.

[0067] In one embodiment, acquiring a first interface image of the target display device includes the following two methods: Method 1: Obtain the first interface image of each target display device through the debug bridge command.

[0068] Method 2: Obtain the first interface image of each target display device by taking a picture with the target camera.

[0069] For the first method mentioned above, the first interface image can be obtained based on the interaction between the target display device and the vehicle terminal. For the second method mentioned above, the first interface image can be obtained based on the interaction between the target display device and the target camera.

[0070] Figure 4 This is a schematic diagram illustrating the method for obtaining the first interface image provided in this application embodiment. Please refer to... Figure 4 , Figure 4 The method shown is the same as method one mentioned above.

[0071] Among them, the debug bridge command can be, for example, the Android Debug Bridge (ADB) command. This command enables interaction between the vehicle terminal and the target display device. The target display device can take a screenshot and obtain a first interface image, which can then be sent to the vehicle terminal.

[0072] It should be noted that during this process, the vehicle terminal can send a debugging bridge command to each target display device. After receiving the debugging bridge command, the target display device can perform the corresponding steps according to the content indicated by the command. In this process, the content indicated by the command may be, for example, taking a screenshot and sending it back.

[0073] The above method allows the vehicle terminal to obtain the first interface image sent by the target display device.

[0074] Figure 5 This is a schematic diagram illustrating another method for obtaining the first interface image provided in this application embodiment. Please refer to... Figure 5 , Figure 5 The acquisition method shown is the second method mentioned above.

[0075] The target camera can be a high-definition camera installed in the vehicle. The scene recognition capability of the high-definition camera can determine the position of the target display device in the vehicle and take pictures.

[0076] It should be noted that the captured images may include the first interface image. The target camera can send the captured images to the vehicle terminal, and the vehicle terminal can perform image processing on the captured images to obtain the first interface image.

[0077] Alternatively, the target camera can perform image processing on the captured image to obtain the first interface image, and then the target display device can send the first interface image to the vehicle terminal, so that the vehicle terminal can obtain the aforementioned first interface image.

[0078] In the process of processing the captured images, the first interface image can be determined from the captured images using image recognition technology.

[0079] Image recognition technology can identify the shape of a screen interface. Once identified, the first interface image can be obtained from the captured image by cropping, adjusting, or other methods.

[0080] In one embodiment, if neither the target camera nor the vehicle terminal has image processing capabilities, the captured image can be sent to the server, which will then perform the image recognition processing described above to obtain the first interface image. After obtaining the first interface image, the server can send it to the vehicle terminal.

[0081] The following explains one feasible implementation process for determining the execution location of the target test instruction provided in the embodiments of this application.

[0082] In one embodiment, obtaining the execution position of a target test instruction in a first interface image of at least one display device includes: inputting the first interface image of the target display device and the target test instruction into a target model to obtain the execution position of the target test instruction, wherein the target model is a model obtained by training with sample interface images and sample test instructions.

[0083] Figure 6 This is a schematic diagram of the input and output of the target model provided in the embodiments of this application. Please refer to... Figure 6 It should be noted that the input of the target model can be the target test instruction and the first interface image, and the output of the target model can be the execution position of the target test instruction, which is the execution position in the first interface image.

[0084] When there are multiple target display devices, the first interface image of each target display device can be input into the model to obtain the execution position of the target test instruction corresponding to each first interface image.

[0085] In one embodiment, the target model can be a neural network model, an AI model, or a large language model, etc., without specific limitations, and can be selected according to actual needs.

[0086] The target model can be placed inside the vehicle terminal. After the vehicle terminal obtains the first interface image, it can input the first interface image and the target test command through the target model to obtain the execution location of the target test command.

[0087] In the instruction testing method provided in this application embodiment, a first interface image of the target display device and the target test instruction can be input into a target model to obtain the execution position of the target test instruction. Recognition through the target model ensures the accuracy of the execution position of the target test instruction, thereby improving the accuracy of the test of the target test instruction.

[0088] The following explains another feasible implementation process for determining the execution location of the target test instruction provided in the embodiments of this application.

[0089] In one embodiment, obtaining the execution location of a target test instruction from a first interface image of at least one display device includes: sending the first interface image of the target display device and the target test instruction to a target server, wherein the target server is a server with image recognition capability and instruction understanding capability; and obtaining the execution location of the target test instruction from the target server.

[0090] Figure 7 This is a schematic diagram illustrating the interaction between the vehicle-mounted terminal and the server provided in the embodiments of this application. Please refer to... Figure 7It should be noted that the target server can be any server that can communicate and interact with the vehicle terminal as described above. This server can be equipped with various functions, such as image recognition and target model deployment, etc., without any specific restrictions.

[0091] In one embodiment, the target model can be placed in the target server. During the process of determining the execution location of the target test instruction, the vehicle terminal can send the target test instruction and the first interface image to the target server. The target server can obtain the execution location of the target test instruction through the target model and can send the execution location of the target test instruction to the vehicle terminal, so that the vehicle terminal can obtain the execution location of the target test instruction.

[0092] In the instruction testing method provided in this application embodiment, a first interface image of the target display device and a target test instruction can be sent to a target server, which is a server with image recognition and instruction understanding capabilities; the execution position of the target test instruction is obtained from the target server. By having the target server perform the corresponding position determination process, the computational pressure on the vehicle terminal can be reduced, making it applicable to various types of vehicle terminals, avoiding excessive computation time for the vehicle terminal, and improving the efficiency of determining the execution position of the target test instruction.

[0093] Either of the two methods mentioned above can be used to determine the result during the actual implementation process.

[0094] For example: If the target test instruction is a music playback instruction, the icon of the music playback software in the first interface image can be identified in the above manner, and then the position of the music playback software icon in the first interface image can be used as the execution position of the target test instruction.

[0095] The aforementioned execution location can be represented by a region in the first interface image.

[0096] The following is a specific interface diagram to explain the scope of the execution location and the display method in the first interface image.

[0097] Figure 8 This is a schematic diagram of the execution location in the first interface image provided in the embodiments of this application. Please refer to... Figure 8 The execution location of the target test instruction is determined based on the target interface image in the first interface image of the target display device and the set of target pixels in the target interface image. The first interface image includes multiple images, and the target interface image is any one of the multiple first interface images.

[0098] The target interface image is one of the first interface images. If there is only one first interface image, then the target interface image is the first interface image. If there are multiple first interface images, then the target interface image can be any one of the first interface images.

[0099] The execution location of the target test instruction may include the target interface image determined from the first interface image and the set of target pixels in the target interface image.

[0100] It should be noted that the target pixel set can be a set of multiple pixels. A pixel can be represented by a two-dimensional coordinate. In actual implementation, the execution location can be a rectangular area in the target interface image, and the set of all pixels in the rectangular area is the aforementioned target pixel set.

[0101] Optionally, the rectangular area is just one example. In actual implementation, it can also be set to a circular or elliptical area, without specific restrictions.

[0102] Example: The shape of the execution area can be set according to the shape of the corresponding control.

[0103] Once the execution location is determined, the target test instruction can be executed at that location.

[0104] The execution process can involve the vehicle terminal sending an ADB command to the corresponding target display device, which is the debug bridge command explained above. It should be noted that the content of this debug bridge command may differ from the content of the aforementioned debug bridge command. This debug bridge command can instruct the target display device to perform a preset operation, such as simulating user clicks, drags, or other operations on the target display device. This operation can be performed at the specified execution location.

[0105] It should be noted that after executing the above target test command, it is also possible to test whether the target test command was executed successfully. The specific implementation process is as follows: Figure 9 This is a flowchart illustrating the verification target test instructions provided in the embodiments of this application. Please refer to... Figure 9 To obtain the execution results of the target test command, including: S910: Obtain the content displayed by the second interface image.

[0106] The second interface image is the target interface image after the target test command is executed.

[0107] It should be noted that after the preset operation of the above target test instruction is completed at the execution location, the interface image of the target display device can be acquired again, which is the second interface image.

[0108] For example, when displaying a target interface image, a target test instruction can be executed. After the target test instruction is executed, the target interface image is changed to a second interface image, so that the second interface image can be acquired in the corresponding target display device.

[0109] The method for acquiring the second interface image is the same as that for acquiring the first interface image. It can be obtained through the debug bridge command or by taking a picture with the target camera, which will not be repeated here.

[0110] S920: Determine whether the target test command was executed successfully based on the content displayed in the second interface image.

[0111] It should be noted that after obtaining the second interface image, the content displayed on the second interface image can be obtained through image recognition. If the vehicle terminal has image recognition capabilities, the displayed content can be recognized through the vehicle terminal; if the vehicle terminal does not have image recognition capabilities, the content displayed on the second interface image can be recognized through the target server.

[0112] In one embodiment, the success of the target test instruction can be determined based on the recognition result, i.e., the content displayed in the second interface image.

[0113] For example, if the content displayed on the second interface image includes specific information, it can be determined that the target test instruction was executed successfully; or, if the content displayed on the second interface image matches a preset template, it can be determined that the target test instruction was executed successfully. No specific limitations are imposed here, and multiple methods can be set to determine whether execution was successful.

[0114] The instruction testing method provided in this application embodiment can acquire a second interface image of the target display device where the execution location is located; based on the content displayed in the second interface image, it can be determined whether the target test instruction was successfully executed. By acquiring the second interface image, the execution result of the target test instruction can be accurately determined, thereby enabling better testing of the target test instruction and improving the accuracy of the test.

[0115] The following explains one possible method of determination: Figure 10 This is a logical diagram of the target determination test instruction provided in the embodiments of this application. Please refer to... Figure 10 Based on the content displayed in the second interface image, determine whether the target test command was executed successfully, including: S1010: If the target information is included in the second interface image, it is determined that the target test instruction was executed successfully.

[0116] It should be noted that the content of the second interface image can be obtained through image recognition. If the image contains target information, it can be determined that the target test command was executed successfully.

[0117] The target information includes at least one of the following: target text, target graphics, and target video.

[0118] It should be noted that the target text, target graphic, and target video can all be preset content. For target text, it can be determined by whether the text in the second interface image is the same as the target text; for target graphic and target video, it can be determined by comparing the degree of matching between the graphic and the video.

[0119] The above determination method can be used to determine whether the second interface image contains target information.

[0120] If any of the above target information is included, then the target test instruction can be confirmed to have been executed successfully.

[0121] S1020: If the target information is not included in the second interface image, it is determined that the target test instruction execution failed.

[0122] Correspondingly, if the second interface image does not contain any of the above target information, it can be determined that the target test instruction failed to execute.

[0123] The instruction testing method provided in this application embodiment can determine that the target test instruction was successfully executed if the second interface image includes target information, and determine that the target test instruction failed to execute if the second interface image does not include target information. The presence or absence of target information in the second interface image allows for quick and accurate determination of the execution result of the target test instruction, thereby improving the testing efficiency and accuracy of the target test instruction.

[0124] It should be noted that if the target test command is successfully tested, the corresponding test result can be recorded, and other test commands can be tested; if the target test command fails, it can be retested.

[0125] In one embodiment, if the target test instruction fails to execute, the method further includes: if the number of execution failures is less than or equal to a preset threshold, re-acquiring the execution position of the target test instruction and executing the target test instruction; if the number of execution failures exceeds the preset threshold, generating a test failure message.

[0126] The preset number of times threshold can be set according to actual needs, for example, it can be 3 times.

[0127] If the number of execution failures is less than or equal to the preset threshold, the target test instruction can be re-executed. The execution location of the target test instruction can be retrieved again and the target test instruction can be executed. If successful, the test is considered successful; if it fails, the execution failure count is increased by one.

[0128] If the number of failed executions exceeds a preset threshold, a test failure message can be generated to alert testers that the target test instruction may be abnormal and requires manual investigation.

[0129] The following explains the overall implementation process of the instruction testing method provided in the embodiments of this application.

[0130] Figure 11 This is a schematic diagram of the overall process of the instruction testing method provided in the embodiments of this application. Please refer to... Figure 11 The method includes the following four stages: Phase 1: Acquire the first interface image of the target display device.

[0131] Phase Two: Determine the execution location of the target test instruction in the first interface image.

[0132] Phase 3: Execute the preset operation of the target test instruction at the execution position of the first interface image of the target display device.

[0133] Phase 4: Determine whether the target test command was executed successfully based on the second interface image of the target display device.

[0134] For the first stage described above, automated commands (such as the aforementioned debug bridge command) or deployed high-definition industrial cameras can be used to simultaneously capture screenshots or take specific photos of the interfaces of multiple display devices within the vehicle's cabin. This allows the onboard terminal to acquire the first interface image.

[0135] For Phase Two above, the first interface image acquired in Phase One, along with keywords from the target test instructions (such as the setting icon and play button), can be input into the target model (such as a multimodal large language model). This target model can then determine the UI controls in the first interface image, for example, the position of the music playback control, and obtain the execution location within the first interface image. This execution location can be an identifier of an interface image (such as the target interface image) and a collection of pixels (such as a rectangular area in the target interface image), thus yielding the execution location.

[0136] In the third stage described above, user interaction can be simulated to execute the target test instructions.

[0137] For stage four above, images can be acquired again in a similar manner to stage one to obtain the second interface image, and image analysis can be performed on the second interface image. The image analysis process can use another image recognition model or the AI ​​model used in stage two above to determine whether the target test instruction was executed successfully.

[0138] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0139] Based on the foregoing embodiments, this application provides an instruction testing device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0140] Figure 12 This is a schematic diagram of the instruction testing device provided in the embodiments of this application. Please refer to... Figure 12 In another aspect of the embodiments of this application, an instruction testing device is also provided, which is applied to an in-vehicle terminal. The in-vehicle terminal is communicatively connected to at least one display device. The device includes: an image acquisition module 1210, a position acquisition module 1220, and an operation execution module 1230. Image acquisition module 1210 is used to acquire a first interface image of at least one display device; The position acquisition module 1220 is used to determine the execution position of the target test instruction in the first interface image of at least one display device, wherein the target test instruction is a preset test instruction; The operation execution module 1230 is used to execute the preset operation of the target test instruction at the execution location and obtain the execution result of the target test instruction.

[0141] In one embodiment, the image acquisition module 1210 is specifically used to determine a target display device that is communicatively connected to the vehicle terminal, wherein the target display device is a display device that is in a working state among at least one display device; and to acquire a first interface image of the target display device.

[0142] In one embodiment, the image acquisition module 1210 is specifically used to acquire a first interface image of each target display device through a debug bridge command; or, to acquire a first interface image of each target display device by taking a picture with a target camera.

[0143] In one embodiment, the position acquisition module 1220 is specifically used to input the first interface image of the target display device and the target test command into the target model to obtain the execution position of the target test command. The target model is a model obtained after training with sample interface images and sample test commands.

[0144] In one embodiment, the location acquisition module 1220 is specifically used to send the first interface image of the target display device and the target test instruction to the target server, the target server being a server with image recognition capability and instruction understanding capability; and to obtain the execution location of the target test instruction from the target server.

[0145] In one embodiment, the execution position of the target test instruction in the device is determined based on the target interface image in the first interface image of the target display device and the set of target pixels in the target interface image. The first interface image includes multiple images, and the target interface image is any one of the multiple first interface images.

[0146] In one embodiment, the image acquisition module 1210 is further configured to acquire the content displayed by the second interface image, wherein the second interface image is the target interface image after the target test instruction is executed; and determine whether the target test instruction has been successfully executed based on the content displayed by the second interface image.

[0147] In one embodiment, the image acquisition module 1210 is further configured to determine that the target test instruction was successfully executed if the second interface image includes target information, wherein the target information includes at least one of the following: target text, target graphics, and target video; and to determine that the target test instruction failed to execute if the second interface image does not include target information.

[0148] In one embodiment, if the execution of the target test instruction fails, the operation execution module 1230 is further configured to: if the number of execution failures is less than or equal to a preset threshold, re-acquire the execution position of the target test instruction and execute the target test instruction; if the number of execution failures is greater than the preset threshold, generate a test failure prompt message.

[0149] The instruction testing apparatus provided in this application embodiment can acquire a first interface image of at least one display device; determine the execution position of a target test instruction in the first interface image of at least one display device, wherein the target test instruction is a preset instruction to be tested; execute a preset operation of the target test instruction at the execution position; and obtain the execution result of the target test instruction. Acquiring the execution position of the target test instruction in the first interface image of at least one display device allows for determining the position of the control to be operated, and thus executing the preset operation of the target test instruction at the execution position to achieve the execution of the target test instruction. This method of determining the control position to execute the target test instruction is adaptable to different display interfaces. When the display interface is updated, no additional maintenance cost is required. Furthermore, in the case of multiple display devices, the execution position of the target test instruction can be obtained for different display interfaces, thereby achieving the execution of the target test instruction. This allows for accurate and efficient testing of the target test instruction.

[0150] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0151] It should be noted that, in the embodiments of this application... Figure 12 The module division of the instruction testing device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit from two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0152] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0153] Figure 13 This is a schematic diagram of the structure of the vehicle-mounted terminal provided in the embodiments of this application. Please refer to... Figure 13 This application provides a vehicle-mounted terminal, the internal structure of which can be shown in the following diagram. Figure 13 As shown, the vehicle-mounted terminal includes a processor 1320, a memory, and a network interface 1340 connected via a system bus 1310. The processor 1320 provides computing and control capabilities. The memory includes a non-volatile storage medium 1331 and internal memory 1332. The non-volatile storage medium 1331 stores the operating system, computer programs, and a database. The internal memory 1332 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 1331. The database is used to store data. The network interface 1340 is used to communicate with external terminals via a network connection. When the computer program is executed by the processor 1320, it implements the aforementioned methods.

[0154] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0155] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0156] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the vehicle terminal to which the present application is applied. A specific vehicle terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, the instruction testing apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 13 The device operates on the vehicle-mounted terminal shown. The memory of the vehicle-mounted terminal can store the various program modules that make up the above-described device. The computer program composed of the various program modules causes the processor to execute the steps of the methods in the various embodiments of this application described in this specification.

[0158] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0159] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0160] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0163] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0165] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0166] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0167] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0168] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0169] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0170] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for testing instructions, characterized in that, Applied to an in-vehicle terminal, wherein the in-vehicle terminal is communicatively connected to at least one display device, the method includes: Obtain a first interface image of the at least one display device; Determine the execution position of the target test instruction in the first interface image of the at least one display device, wherein the target test instruction is a preset test instruction; The preset operation of the target test instruction is executed at the execution location, and the execution result of the target test instruction is obtained.

2. The method according to claim 1, characterized in that, The step of acquiring the first interface image of the at least one display device includes: Identify a target display device that is communicatively connected to the vehicle terminal, wherein the target display device is one of the at least one display devices that is in a working state; Obtain the first interface image of the target display device.

3. The method according to claim 2, characterized in that, The step of acquiring the first interface image of the target display device includes: Obtain the first interface image of each target display device using the debug bridge command; or... The first interface image of each target display device is obtained by capturing it with the target camera.

4. The method according to claim 2, characterized in that, Obtaining the execution location of the target test instruction in the first interface image of the at least one display device includes: The first interface image of the target display device and the target test command are input into the target model to obtain the execution position of the target test command. The target model is a model obtained by training with sample interface images and sample test commands.

5. The method according to claim 2, characterized in that, Obtaining the execution location of the target test instruction in the first interface image of the at least one display device includes: The first interface image of the target display device and the target test command are sent to the target server, which is a server with image recognition and command understanding capabilities. Obtain the execution location of the target test instruction from the target server.

6. The method according to claim 4 or 5, characterized in that, The execution location of the target test instruction is determined based on the target interface image in the first interface image of the target display device and the set of target pixels in the target interface image. The first interface image includes multiple images, and the target interface image is any one of the multiple first interface images.

7. The method according to claim 6, characterized in that, The step of obtaining the execution result of the target test instruction includes: Obtain the content displayed in the second interface image, where the second interface image is the target interface image after executing the target test command; Based on the content displayed in the second interface image, determine whether the target test instruction was executed successfully.

8. The method according to claim 7, characterized in that, The step of determining whether the target test instruction was successfully executed based on the content displayed in the second interface image includes: If the second interface image includes target information, it is determined that the target test instruction was executed successfully. The target information includes at least one of the following: target text, target graphics, and target video. If the target information is not included in the second interface image, it is determined that the target test instruction failed to execute.

9. The method according to claim 8, characterized in that, In the event that the target test instruction fails to execute, the method further includes: If the number of execution failures is less than or equal to a preset threshold, the execution location of the target test instruction is retrieved again and the target test instruction is executed. If the number of failed tests exceeds a preset threshold, a test failure message will be generated.

10. An instruction testing device, characterized in that, The device is applied to an in-vehicle terminal, which is communicatively connected to at least one display device. The device includes: an image acquisition module, a location acquisition module, and an operation execution module. The image acquisition module is used to acquire a first interface image of the at least one display device; The position acquisition module is used to determine the execution position of the target test instruction in the first interface image of the at least one display device, wherein the target test instruction is a preset test instruction; The operation execution module is used to execute a preset operation of the target test instruction at the execution location and obtain the execution result of the target test instruction.

Images