Automated testing method, apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application disclose an automatic testing method, device, equipment and storage medium. The method comprises: determining a target control and a target state corresponding to the target control based on a to-be-tested scene; obtaining an original state of the target control in a current visual interface, and determining an automatic operation instruction of the target control based on the original state and the target state; adjusting the state of the target control based on the automatic control instruction, obtaining an adjusted visual interface, and extracting an actual state corresponding to the target control in the adjusted visual interface; and if the actual state matches the target state, determining that the test of the target control is passed. Embodiments of the present application can ensure the consistency and accuracy of each test by precisely controlling the test conditions and the execution process, thereby improving the reliability of the test result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication system testing, and in particular to an automatic testing method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the development of science and technology, computer vision technology and automatic testing systems have been widely applied in various fields. In the field of automotive electronic devices, automatic control of CANoe Panel (a visual control set) for testing is a common testing method. This method mainly simulates user operations to test the functions and performance of automotive electronic devices. However, this method requires a large amount of manual operation, is inefficient, and is prone to errors. Therefore, there is an urgent need in the prior art for a method capable of automatic and effective automatic testing to meet the needs of actual testing applications. SUMMARY

[0003] To solve the above technical problems, embodiments of the present application provide an automatic testing method and device, an electronic device, a computer readable storage medium, and a computer program product.

[0004] According to an aspect of an embodiment of the present application, an automatic testing method is provided, including: determining a target control and a target state corresponding to the target control based on a to-be-tested scene; obtaining an original state of the target control in a current visual interface, and determining an automatic operation instruction of the target control based on the original state and the target state; adjusting the state of the target control based on the automatic control instruction to obtain an adjusted visual interface, and extracting an actual state corresponding to the target control in the adjusted visual interface; and determining that the test of the target control is passed if the actual state matches the target state.

[0005] According to an aspect of an embodiment of the present application, the method further includes: performing image detection on the visual interface based on a target detection model to obtain an original detection result; and determining the original state of the target control based on the detection result if the confidence of the original detection result reaches a preset confidence threshold, wherein the original state includes an original center point coordinate, an original bounding box, and an original rotation angle of the target control.

[0006] According to an aspect of some embodiments of the present application, the method further includes: determining a target center point coordinate, a target bounding box, and a target rotation angle corresponding to the target control based on the to-be-tested scene; determining a plurality of operation values based on the target center point coordinate, the target bounding box, and the target rotation angle and the original center point coordinate, the original bounding box, and the original rotation angle; generating a plurality of operations corresponding to the driving level operation based on the plurality of operation values, and generating the automation operation instruction based on the plurality of operations.

[0007] According to an aspect of some embodiments of the present application, the method further includes: performing image detection on the adjusted visual interface based on the target detection model to obtain an adjusted detection result; and determining an actual state corresponding to the target control based on the adjusted detection result if a confidence of the adjusted detection result reaches a preset confidence threshold.

[0008] According to an aspect of some embodiments of the present application, the method further includes: obtaining a control image set in the visual interface, and labeling a label corresponding to the control based on a position of the control on the visual interface and a function of the control to obtain a center point coordinate, a bounding box, and a rotation angle corresponding to the control; generating a control image label set based on the control image set and the label corresponding to the control, wherein the control image label set includes the center point coordinate, the bounding box, and the rotation angle corresponding to the control; and training the target detection model based on the control image label set.

[0009] According to an aspect of some embodiments of the present application, the method further includes: parsing the to-be-tested scene to obtain a key function corresponding to the to-be-tested scene; determining a control label corresponding to the key function to find a matching target control based on the control label.

[0010] According to an aspect of some embodiments of the present application, the method further includes: determining an expected result corresponding to the to-be-tested scene; obtaining an operation result corresponding to the target control after the automation control instruction is executed; and determining that the target control passes the test if the operation result matches the expected result.

[0011] According to an aspect of some embodiments of the present application, an automated testing device is provided, comprising: a determining module configured to determine a target control and a target state corresponding to the target control based on a testing scenario; an obtaining module configured to obtain an original state of the target control in a current visual interface, and determine an automated operation instruction of the target control based on the original state and the target state; an adjusting module configured to adjust the original state based on the automated operation instruction to obtain an adjusted visual interface, and extract an actual state corresponding to the target control in the adjusted visual interface; and a judging module configured to determine that the target control passes the test if the actual state matches the target state.

[0012] According to an aspect of some embodiments of the present application, an electronic device is provided, comprising: one or more processors; and a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the automated testing method as described above. According to an aspect of some embodiments of the present application, a computer-readable storage medium is provided, which stores computer-readable instructions, which, when executed by a processor of a computer, cause the computer to perform the automated testing method as described above.

[0013] According to an aspect of some embodiments of the present application, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the steps of the automated testing method as described above.

[0014] In the technical solutions provided in the embodiments of the present application, by determining a target control and a target state corresponding to the target control based on a testing scenario, then obtaining an original state of the target control in a real-time visual interface, and further determining an automated operation instruction of the target control based on the original state and the target state, the automated testing can greatly reduce the time and labor cost of manual testing. Through the pre-defined target control and target state, and the automatically generated and executed operation instruction, the testing process can be quickly and accurately executed, significantly improving the testing efficiency. Then, the state of the target control is adjusted based on the automated control instruction to obtain an adjusted visual interface, and then the actual state corresponding to the target control in the adjusted visual interface is extracted. If the actual state matches the target state, it is determined that the target control passes the test. The automated testing reduces errors caused by human factors, such as missing test points and misjudging test results. By precisely controlling the testing conditions and the execution process, the consistency and accuracy of each test can be ensured, thereby improving the reliability of the test results.

[0015] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application. It is apparent that the drawing in the following description is only some embodiments of the application, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0017] Figure 1 is a schematic diagram of a visualization interface of an automated test according to an exemplary embodiment of the application;

[0018] Figure 2 is a schematic diagram of an implementation environment for performing an automated test according to an exemplary embodiment of the application;

[0019] Figure 3 is a flowchart of an automated test method according to an exemplary embodiment of the application;

[0020] Figure 4 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0021] Figure 5 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0022] Figure 6 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0023] Figure 7 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0024] Figure 8 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0025] Figure 9 is a flowchart of an automated test method according to another exemplary embodiment of the application;

[0026] Figure 10 is a schematic diagram of a brief flow of an automated test in an exemplary application scenario;

[0027] Figure 11 is a block diagram of an automated test device according to an exemplary embodiment of the application;

[0028] Figure 12 shows a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the application. DETAILED DESCRIPTION

[0029] Detailed description of example embodiments will now be given with reference to the accompanying drawings. The description below relates to the drawings, in which like reference numerals refer to like elements or features. It will be evident from this description that various example embodiments can be realized in various forms. The following description, in connection with the examples described, provides a full explanation of the devices, methods, and techniques within the scope of the present disclosure, but are not meant to limit the present disclosure to these specific examples.

[0030] The block diagrams in the drawings show only the functionality of the example embodiments and do not imply that the described functionality must be implemented in a specific order. Further, the block diagrams do not imply that the described functionality must be implemented in every example embodiment. The examples described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, functional blocks of the devices can be implemented in a programmable logic device, discrete hardware, or any combination thereof. Program code means or program instructions applied to a general-purpose computer or a special-purpose computer set the computer in operation in combination with the computer program code means or program instructions for implementation of the functions described herein and specified in a computer implementation. The general-purpose or special-purpose computer receives structure from the computer program code means or program instructions, and that computer program code means or program instructions specifically structure the computer to carry out or to induce the computer to carry out the functions described herein.

[0031] The flow diagrams depicted herein are examples only and are not meant to limit the disclosed embodiments to the specific sequences or orders of steps. Rather, the steps can be performed in any order, or in parallel, or in any combination, and the order or sequence of steps can vary from one example to another. Further, some steps can be omitted, some steps can be combined, and some steps can be performed in a different order or at a different time.

[0032] In the present disclosure, "multiple" refers to two or more. The "and / or" description of the associated objects represents that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. The character " / " generally represents that the front and rear associated objects are a "or" relationship.

[0033] First of all, it needs to be explained that CANoe Panel is a very powerful function in CANoe software, which allows users to create custom graphical user interfaces (GUI) for interacting with simulated or actual running network environments. This interface can simulate various physical controls such as buttons, switches, knobs, instrument panels, etc., and directly map the operations of these controls to signals in network messages (such as CAN, LIN, FlexRay, etc.). Through CANoe Panel, testers can easily simulate user operations, send control commands to ECUs (Electronic Control Units) or other network nodes, and observe the system's response. This interactive way greatly simplifies the testing process, enabling testers to more intuitively understand the behavior of the system and quickly identify potential problems.

[0034] In CANoe, Panel is a graphical user interface (GUI) that allows users to interact with the simulated or actual running ECU (Electronic Control Unit) in an intuitive way. Panel can be designed to simulate various switches, buttons, indicator lights, instrument panels, etc., which can be directly mapped to signals in network messages. In this way, testers can simulate user operations, send control commands, and observe the system's response.

[0035] Please refer to Figure Figure 1 , Figure 1 is a schematic diagram of the visualization interface shown in an exemplary embodiment of the present application, which can be seen Figure 1 including controls, and the corresponding center point coordinates, frame height, width, and rotation angle of the space. Among them, "control picture" may refer to the icon or form of the control) The content is very rich, these controls are used to create graphical user interfaces, and there are Boolean switches, which are usually used to represent the on-off state of the switch. They can have various styles, including button type, sliding type, etc., and may be accompanied by changes in icons or colors when clicked to indicate state changes. Buttons, standard button controls, are used to trigger certain operations or commands. The button usually displays text or icons to indicate its function. And text boxes that allow users to input data or display data. Users can enter numbers, text, etc. in them.

[0036] Figure 2 is a schematic diagram of the implementation environment for automated testing shown in an exemplary embodiment of the present application, as shown in Figure 2, in the automated testing scenario, the server 220 first determines the test scene, and then the server 220 determines at least one target control corresponding to the test scene and the target state corresponding to the at least one target control from the test scene. Then the server 220 obtains the original state of the target control on the visualization interface of the intelligent terminal 210, and then can determine the automated operation instruction of the target control according to the original state and the target state, and then adjust the state of the target control based on the automated control instruction, obtain the adjusted visualization interface, and extract the actual state of the target control in the adjusted visualization interface; If the actual state matches the target state, it is determined that the test of the target control is passed.

[0037] Among them, Figure 2 The intelligent terminal 210 shown in the figure can be any terminal device that supports installation of navigation map software, such as a smartphone, a car computer, a tablet computer, a notebook computer, or a wearable device, but is not limited thereto. Figure 2The server 220 shown can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like, and is not limited herein. The intelligent terminal 210 can communicate with the server 220 through a wireless network such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), 5G (fifth generation mobile information technology), and the like, and is not limited herein.

[0038] With the development of science and technology, computer vision technology and automated test systems have been widely applied in various fields. In the field of automotive electronic devices, it is a common test method to automatically control CANoe Panel (a visual control set) for testing. This method mainly simulates user operations to test the functions and performance of automotive electronic devices. However, this method requires a large amount of manual operation, is inefficient, and is prone to errors. Therefore, there is an urgent need in the prior art for a method that can automatically and effectively perform automated testing to meet the needs of actual testing applications.

[0039] The above-mentioned problems have universal applicability. To solve these problems, embodiments of the present application propose an automated testing method, an automated testing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0040] As shown in Figure 3 In an exemplary embodiment, the automated testing method includes at least steps S310 to S340, which are described in detail as follows:

[0041] Step S310: determining a target control and a target state corresponding to the target control based on a to-be-tested scenario.

[0042] Specifically, a test scenario is a series of steps and conditions that describe how a user interacts with a system in a specific testing environment. A well-defined test scenario should include: scenario name: a brief description of the theme or purpose of the test scenario; preconditions: conditions that must be met before executing the test scenario, such as user login status, system configuration, etc.; steps: detailed list of user actions, including interactions with interface controls; input data: data that needs to be input during the test, such as text, files, commands, etc.; expected results: the expected output or system response for each step or the entire scenario. After the test scenario is defined, the key controls involved in the scenario need to be analyzed. Controls are interface elements that users interact with application programs, such as buttons, text boxes, lists, etc.

[0043] For example, based on the CANoe Panel test scenario, test the unlocking and locking of the vehicle door, test the opening and closing of the vehicle lights, etc. Send the unlock or lock signal through the CANoe Panel to verify whether the vehicle door can correctly respond and feedback the status or control the vehicle light switch through the CANoe Panel to check whether the lighting and extinguishing of the vehicle lights (such as low beam, high beam, turn signal, etc.) meet the expectations. Specifically, the target controls involved in the test of the vehicle door unlocking and locking can include the unlock button, the lock button, the status indicator light, etc., and the target controls involved in the test of the vehicle light opening and closing include the high beam switch and the low beam switch. Then, according to the test scenario to be tested, determine the target state corresponding to the target control, then create or open an existing CANoe project, configure the relevant DBC file and network environment, map the Panel control to the signal in the DBC file, and ensure that the control operation can correctly send the corresponding signal to the vehicle ECU. Connect the CANoe device on the real vehicle to ensure normal communication. Operate the control through the Panel interface and observe whether the vehicle response meets the expectations.

[0044] Step S320, obtaining the original state of the target control in the current visual interface, and determining the automation operation instruction of the target control based on the original state and the target state.

[0045] Specifically, it is necessary to be able to access and read the current state of the target control in the current visual interface (such as CANoe Panel or other GUI interface). This can usually be achieved in the following ways: using a graphical user interface automation tool: such as Selenium (mainly for Web interface), PyAutoGUI (for desktop applications) or a CANoe-specific automation tool (if available); directly through an interface or API: if CANoe provides APIs to access the state of Panel controls, these APIs can be directly called to obtain the state; screen reading technology: for GUIs that do not support direct API access, screen capture and image recognition techniques can be used to identify the state of the control.

[0046] The acquired original state is compared with the target state to determine whether an automation operation needs to be performed to change the state of the control, where the target state is the state you want the control to reach during testing or control. This is usually predefined and part of the test script or automation control logic. If the original state does not match the target state, corresponding automation operation instructions need to be generated to change the state of the control. These instructions can include: click: if the control is a button, a mouse click may need to be simulated to activate it; input text: if the control is a text box, specific text may need to be entered; send signal: in the context of CANoe, if the control is associated with an ECU signal, a corresponding CAN signal may need to be sent through a CAPL script or other mechanism; change attribute value: if the state of the control is controlled by changing the value of one of its attributes (such as the checked state of a checkbox), the corresponding method or function needs to be found to change this attribute value.

[0047] Step S330, based on the automation control instructions, adjust the state of the target control, obtain the adjusted visual interface, and extract the actual state of the target control in the adjusted visual interface.

[0048] Specifically, first, the automation control instructions generated in the previous embodiments need to be executed. These instructions can include simulating user operations (such as clicking, inputting, etc.), or directly sending signals to the system through API. In the context of CANoe, this usually involves using CAPL scripts or other mechanisms to send CAN signals to the vehicle ECU, thereby triggering a change in the state of the control. After executing the control instructions, some time is needed to respond and update the control state on the interface. Therefore, appropriate waiting mechanisms need to be added in the automation test script. This can be a simple fixed time delay, or a more intelligent waiting condition, such as checking whether a certain state variable has changed or a certain event has been triggered. Once the waiting mechanism confirms that the state has changed, the adjusted visual interface can be obtained.

[0049] In automated testing, this usually means capturing a screenshot of the current GUI, or using GUI automation tools to read the current state of controls on the interface. From the adjusted visualization interface, you need to extract the actual state of the target control. This can be achieved in multiple ways, including but not limited to: screen image analysis: using image recognition techniques to analyze the screenshot, determine the state of the control (such as whether the button is pressed, the indicator light is on, etc.); GUI automation tools: use GUI automation tools like Selenium, PyAutoGUI to query the current attribute value of the control; CANoe API or CAPL script: if CANoe provides the corresponding API or can access the control state through CAPL script, directly call these interfaces to obtain the state information.

[0050] Step S340, if the actual state matches the target state, determine that the test of the target control is passed.

[0051] Specifically, when the actual state matches the target state, it can be determined that the test for the target control has passed. In the automated testing framework, once the test step is executed and the necessary state change is waited for, the actual state of the target control is extracted and compared with the preset target state. If they are consistent, the test is considered successful; if they are not consistent, the test fails and further analysis and debugging may be needed to find out the problem. For tools like CANoe, you can implement this process by writing CAPL scripts. CAPL scripts can simulate user input, send CAN messages, read ECU responses, perform conditional judgments, and record test results, etc. Through carefully designed CAPL scripts, you can automatically execute complex test scenarios to verify whether the behavior of various controls and systems in the vehicle network meets expectations. When the test passes, the test success information is usually recorded, and the results may be reported to relevant stakeholders. If the test fails, the reasons and context information for the failure need to be recorded in detail for subsequent problem identification and repair work.

[0052] In the embodiment, by determining the target control and the target state corresponding to the target control for the to-be-tested scene, then obtaining the original state of the target control corresponding to the target control in the real-time visual interface, and then determining the automatic operation instruction of the target control according to the original state and the target state of the target control, the automatic test can greatly reduce the time and labor cost of manual test. Through the pre-defined target control and target state, and the automatically generated and executed operation instruction, the test process can be quickly and accurately executed, which significantly improves the test efficiency. Then the state of the target control is adjusted through the automatic control instruction to obtain the adjusted visual interface, and then the actual state corresponding to the target control in the adjusted visual interface is extracted. When the actual state matches the target state, it can be determined that the test of the target control is passed. The automatic test reduces errors caused by human factors, such as missing test points and misjudging test results. By accurately controlling the test conditions and the execution process, the consistency and accuracy of each test can be ensured, thereby improving the reliability of the test results.

[0053] Further, based on the above embodiments, please refer to Figure 4 In one of the example embodiments provided in the present application, the specific implementation process of the automatic test method can further include steps S410 and S420, which are described in detail as follows:

[0054] Step S410: performing image detection on the visual interface based on a target detection model to obtain an original detection result.

[0055] Step S420: if the confidence of the original detection result reaches a pre-set confidence threshold, determining the original state of the target control based on the detection result, and the original state includes the original center point coordinate, the original bounding box and the original rotation angle of the target control.

[0056] It should be noted that the target detection model can be a yolov5 model, wherein the yolov5 model is a computer vision pattern designed object detection task, which is an upgraded version of the YOLO series and is famous for its real-time object detection capability.

[0057] Specifically, before inputting the image into the object detection model, some preprocessing steps may be needed, such as adjusting image size, normalization, color space conversion, etc., to ensure that the model can efficiently and accurately process the input image. The preprocessed image is detected using a pre-trained or custom object detection model. The model identifies the target objects (in this context, target controls) in the image and outputs a bounding box and a corresponding confidence score for each detected object. The bounding box usually represents the position and size of the target control in the image, while the confidence score represents the model's confidence in the detection result. For each detected target control, check if its confidence score meets a pre-set confidence threshold. This threshold is an adjustable parameter that balances the accuracy and recall rate of detection. If the confidence score is higher than the threshold, the detection result is considered reliable,

[0058] Specifically, the original state of the target control can be extracted from the current visualization interface by the object detection model, which includes: original center point coordinates: calculate the center point coordinates of the bounding box, which can be obtained by taking the average of the left top corner and right bottom corner coordinates; original edge box: directly use the bounding box output by the model as the original edge box of the target control. The bounding box is usually composed of four coordinates (left top corner x, y and right bottom corner x, y), or equivalently, represented by center point coordinates, width and height; original rotation angle: if the target control is rotated in the image and the detection model supports the detection of rotated objects, the rotation angle can be directly obtained from the model output. However, if the model does not support rotation detection, it may be necessary to estimate the rotation angle by other methods (such as image processing techniques). In some cases, if the rotation of the target control has little effect on subsequent processing, the rotation angle can be ignored.

[0059] In this embodiment, the object detection model can accurately identify the position, shape and orientation of the control, greatly improving the accuracy and efficiency of the test. By accurately identifying the control state of the user interface, automated testing can detect problems in interface layout, control function, etc. earlier.

[0060] Further, based on the above embodiments, please refer to Figure 5 In one of the example embodiments provided in this application, the specific implementation process of the above-mentioned automated testing method can further include steps S510 to S530, which are described in detail as follows:

[0061] Step S510: determining the target center point coordinates, target edge box and target rotation angle corresponding to the target control based on the scene to be tested.

[0062] Specifically, the scene to be tested needs to be analyzed in detail to understand the interface elements, layout of controls, and interaction between controls, which helps to determine which controls are the target controls of interest and the target center point coordinates, target frame, and target rotation angle corresponding to the target state of the target controls.

[0063] Step S520, based on the target center point coordinates, target frame, and target rotation angle and the original center point coordinates, original frame, and original rotation angle, determine a plurality of operation values.

[0064] Specifically, the movement distance and direction of the center point of the target control relative to the center point of the original control are calculated to obtain the corresponding operation value, and the operation value can be a two-dimensional vector representing the movement amount in the X and Y axis directions (i.e. target center point coordinates minus original center point coordinates). This vector indicates the direction and distance of the control movement. The change in the frame size of the target control relative to the frame size of the original control is evaluated. Operation value: usually includes two values, one representing the change in frame width (target width minus original width), and the other representing the change in frame height (target height minus original height). These values can be positive (indicating expansion), negative (indicating contraction), or zero (indicating no change). The operation value of the rotation angle. Measure the change in the rotation angle of the target control relative to the original control. Operation value: an angle value representing the change from the original rotation angle to the target rotation angle (target rotation angle minus original rotation angle). This value can be positive (indicating clockwise rotation), negative (indicating counterclockwise rotation), or zero (indicating no rotation change). Note that when dealing with angles, their periodicity (e.g. 360 degrees equals 0 degrees) may need to be considered, but in actual operation, only small angle changes are usually concerned.

[0065] And these operation values collectively describe the geometric transformation and property changes of the target control relative to its original state. They can be used in various scenarios, such as state verification in automated testing, iterative tracking of UI design, or user interaction analysis in dynamic interfaces. By comparing these operation values, we can quickly identify the specific changes of the controls on the interface and make corresponding processing or decisions accordingly.

[0066] Step S530, based on the plurality of operation values, generate a plurality of operations corresponding to the driving level operations, and based on the plurality of operations, generate an automated operation instruction.

[0067] Specifically, analyze the plurality of operation values obtained from the control state comparison, including the offset of the center point coordinates, the change of the frame size, and the change of the rotation angle. Map the parsed operation values to specific driving level operations. This usually involves calling the UI framework or automated testing tool API.

[0068] Center point coordinate offset: May be converted into a mouse movement operation, moving the mouse pointer to the new center point position of the target control.

[0069] Border size change: May need to simulate size changes in combination with specific UI element interactions (such as scroll bars, dividing lines, etc.), or in some cases, this may directly reflect changes in content without explicit operations. However, if the size is changed by dragging the edge of the control, it can be converted into a simulated mouse drag operation.

[0070] Rotation angle change: In standard web or desktop applications, control rotation is usually not the result of direct user operation, but driven by internal application logic or animation effects. However, in some specific testing environments (such as game testing, graphical interface testing), it may be necessary to simulate keyboard input (such as specific shortcut keys), use the rotation function provided by the testing tool, or call the underlying graphics API to achieve it.

[0071] Then, according to the mapped driving level operation, generate specific automation operation instructions. These instructions will guide the automation testing tool to perform the corresponding operation. Use the API provided by the testing framework or tool to write code or scripts that will perform the above mouse movement, click, drag, or keyboard input operations. Set the necessary waiting time or checkpoints to ensure the correct order and dependencies between operations.

[0072] In this embodiment, by accurately calculating the center point coordinates, border, and rotation angle of the target control, we can ensure that the test operation is directly performed on the target control, avoiding misoperation of other controls. The method of generating automation operation instructions based on the state changes of the target control has significant beneficial effects in improving test accuracy, efficiency, flexibility, and reducing maintenance costs, etc.

[0073] Further, based on the above embodiments, please refer to Figure 6 In one of the example embodiments provided in this application, the specific implementation process of the above-mentioned automation testing method can further include steps S610 and S620, which are described in detail as follows:

[0074] Step S610: Image detection based on the target detection model to obtain the adjusted detection result.

[0075] Specifically, as described in the above embodiments, the trained target detection model can be used to detect the target control and the corresponding state of the target control in the adjusted visual interface, and the image detection of the adjusted visual interface can be performed to obtain the detection result corresponding to the target control in the adjusted visual interface.

[0076] Step S620, if the confidence of the adjusted detection result reaches the preset confidence threshold, determine the actual state of the target control based on the adjusted detection result.

[0077] Specifically, the target control is initially detected by a certain detection method (such as image recognition, OCR technology, UI automation test framework, etc.) to obtain its current state (such as whether visible, enabled, selected, etc.). On the adjusted detection result, confidence evaluation is performed. Confidence is an indicator of the reliability of the detection result, usually based on a series of statistical data and algorithms. The level of confidence reflects the degree of consistency between the detection result and the actual state. In order to ensure the reliability of the detection result, a confidence threshold will be preset. This threshold is determined according to the actual demand and application scenario, usually a value between 0 and 1 (for example, 0.9 represents 90% confidence). If the confidence of the adjusted detection result reaches the preset confidence threshold, then the detection result can be considered reliable enough to determine the actual state of the target control. In this case, the actual state of the target control will be updated or confirmed based on the adjusted detection result.

[0078] In this embodiment, the adjusted visual interface is detected by the target detection model, and the actual state of the target control is determined according to the confidence of the adjusted detection result, which has significant beneficial effects in improving the accuracy, real-time, reliability and stability of the test, while reducing the test cost and improving the test efficiency.

[0079] Further, based on the above embodiments, please refer to Figure 7 In one of the example embodiments provided in the present application, the specific implementation process of the above-mentioned automatic test method can further include steps S710 to S730, which are described in detail as follows:

[0080] Step S710, obtain the control image set in the visual interface, and label based on the position of the control on the visual interface and the function of the control to obtain the label corresponding to the control.

[0081] Specifically, a control image training set in a to-be-trained panel control panel can be acquired. First, a large number of control images in the CANoe Panel control panel need to be collected, such as a large number of control images in the Picture Box, Switch / Indicator, LEDControl, etc. These images will be used as a training set to train the target detection model. Then, a label tool is used to label the images to determine the meaning of the images and the position of the control on the visual interface, the function of the control. For example, a label tool (such as rolabelimg) can be used to label the images in the training set, and a label needs to be added to each control to determine its position and function in the image.

[0082] In step S720, a control image label set is generated based on the control image set and the labels corresponding to the controls. The control image label set includes the center point coordinates, the bounding box, and the rotation angle of the control.

[0083] First, if the control images are of different sizes, the control images need to be normalized to obtain a uniform size. For regular-shaped controls (such as rectangles and circles), the center point coordinates can be calculated from the boundary coordinates of the control. For example, the center point coordinates of a rectangular control can be calculated from the boundary coordinates to obtain the midline point coordinates. If the control shape is irregular, more complex image processing methods may be needed to estimate the center point, such as the centroid calculation method based on the contour. Then, the bounding box is usually defined by the outermost boundary coordinates of the control. For rectangular controls, the corresponding boundary coordinates can be directly determined. For non-rectangular controls, edge detection algorithms (such as the Canny edge detector) or contour detection algorithms (such as the Hough transform) may be needed to identify the bounding box. Moreover, if the control image is rotated, the rotation angle needs to be detected and recorded. This can be achieved by calculating the angle between the principal axis of the control and the image coordinate axis, or if the control image is generated from an original image rotated at a known angle, the rotation angle can be directly obtained from the generation process. Then, the control image label set can be generated according to the labels corresponding to each control image and the center point coordinates, the bounding box, and the rotation angle of the control.

[0084] In step S730, a target detection model is trained based on the control image label set.

[0085] Specifically, first, the control image label set is divided into a training set, a validation set, and a test set. Generally, the training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameter tuning), and the test set is used to evaluate the final performance of the model. In addition, in order to improve the generalization ability of the model, data augmentation can be performed on the training set, such as rotation, scaling, cropping, flipping, etc. Then, according to the specific requirements (such as real-time requirements, accuracy requirements, computational resource limitations, etc.), a suitable target detection model is selected. Common target detection models include models based on convolutional neural networks (CNN), such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Faster R-CNN, etc. Using a deep learning framework (such as TensorFlow, PyTorch, etc.), load the preprocessed control image and corresponding label data. According to the selected target detection model architecture, define the model in the deep learning framework. This usually involves setting parameters such as the number of network layers, convolution kernel size, activation function, etc. Set various parameters during the training process, such as learning rate, batch size, training epochs, etc. Input the training data into the model, calculate the loss value through forward propagation, and then update the model parameters through backpropagation. Repeat this process until the preset training epochs are reached or other stopping conditions are met.

[0086] For example, taking the yolov5 model as an example, first, since yolov5 uses a specific data format (usually COCO or VOC format, but yolov5 has its own data organization method). You need to convert the control image label set into a format that yolov5 can recognize. This usually involves creating a text file (such as a.txt file) containing image paths, class IDs, and bounding box coordinates. Divide the dataset into a training set, a validation set, and an optional test set. yolov5 uses configuration files (such as yolov5s.yaml, yolov5m.yaml, etc.) to define the model architecture, training parameters, etc. You need to modify these configuration files according to your dataset and task requirements. In particular, you need to specify the number of classes, the path of the training / validation dataset, etc. Then, use the training script (such as train.py) provided by yolov5 to train the model. You need to specify the configuration file and dataset path as command line parameters. The training process will output training logs, including loss values, accuracy, etc. During the training process, yolov5 will automatically evaluate the performance of the model on the validation set. In some embodiments, the test set can also be used to further verify the accuracy and robustness of the model after training is completed, and then the yolov5 model is obtained.

[0087] In the embodiment, through the process of generating a control image label set based on the control image set and its label, and training the target detection model according to the control image label set, remarkable beneficial effects are brought in improving model accuracy, accelerating the training process, supporting complex interface detection, reducing labor costs, and improving test efficiency.

[0088] Further, based on the above embodiment, please refer to Figure 8 In one of the example embodiments provided in the present application, the specific implementation process of the above-mentioned automatic test method can further include steps S810 and S820, which are described in detail as follows:

[0089] Step S810: Analyze the test scene to obtain the corresponding key function of the test scene;

[0090] Step S820: Determine the corresponding control label based on the key function, and find the matching target control based on the control label.

[0091] Specifically, by analyzing the test scene, user behavior, input data, expected results, etc. can be understood. The scene description is converted into a business process diagram or step list, and the operation and input and output of each step are clearly defined. Then find the corresponding user interface (UI) control for each key function, these controls are the direct target of the test operation (such as click, input, selection, etc.). Analyze the mapping relationship between the control and the function: according to the UI design of the application program, analyze which controls are directly related to the extracted key functions. Determine the control label: define a unique label or identifier for these controls. These labels can be the ID, class name, text content or other attributes of the control, depending on the type of control and the capabilities of the test framework. Write control lookup logic: based on the control label, write code or script to locate these controls in the UI. This usually involves using the API provided by the test framework to search for elements with specific properties.

[0092] In the embodiment, through the process of analyzing the test scene, determining the control label corresponding to the key function, and finding the matching target control, remarkable beneficial effects are brought in improving the pertinence, automation, comprehensiveness, depth, reducing the cost of the test, and improving the flexibility and scalability of the test.

[0093] Further, based on the above embodiment, please refer to Figure 9 In one of the example embodiments provided in the present application, the specific implementation process of the above-mentioned automatic test method can further include steps S910 to S930, which are described in detail as follows:

[0094] Step S910: Determine the expected result corresponding to the test scene;

[0095] Step S920, after the execution of the automation control instruction, the operation result corresponding to the target control is obtained;

[0096] Step S930, if the operation result matches the expected result, it is determined that the test of the target control is passed.

[0097] Specifically, based on the scene requirements, the results that should be observed after the test execution are determined. This may include changes in the UI interface, updates of data, response status of the system, etc. The expected results are clearly recorded in the test plan for verification after the test execution. After the script execution, the operation result of the target control is captured through appropriate means (such as API call, UI element check, log analysis, etc.). This may involve obtaining the attribute value of the interface element, checking the data change in the database, verifying the record in the system log, etc. The actual operation result is compared with the expected result set previously using assertion or similar mechanism. This usually involves string comparison, numerical value comparison, state check, etc. According to the comparison result, it is evaluated whether the test is passed. If the operation result completely matches the expected result, it is considered that the test of the target control is passed; otherwise, it is considered that the test fails.

[0098] In the present embodiment, through the process of determining the expected result, comparing the operation result with the expected result to judge whether the test is passed, significant beneficial effects are brought in improving the reliability and effectiveness of the test, supporting the automation test, enhancing the feedback and iteration capability of the test, improving the product quality and user satisfaction, etc.

[0099] Figure 10is a brief flowchart of the automatic testing in an example application scenario. In the application scenario shown in 10, the scene to be tested is parsed to obtain the key function corresponding to the scene to be tested; the control label corresponding to the key function is determined to find the target control and the target state corresponding to the target control based on the control label, obtain the original state of the target control in the current visual interface, and the original state includes the original center point coordinate, the original border and the original rotation angle of the target control. The target center point coordinate, the target border and the target rotation angle corresponding to the target control are determined based on the scene to be tested; the operation value is determined based on the target center point coordinate, the target border and the target rotation angle and the original center point coordinate, the original border and the original rotation angle. Based on the operation value, a plurality of operations corresponding to the driving level operation are generated, and the automatic operation instruction is generated based on the plurality of operations. The state of the target control is adjusted based on the automatic control instruction to obtain the adjusted visual interface, and the actual state corresponding to the target control in the adjusted visual interface is extracted; the expected result corresponding to the scene to be tested is determined; after the execution of the automatic control instruction, the operation result corresponding to the target control is obtained; if the operation result matches the expected result, the test of the target control is determined to pass. For detailed implementation process, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0100] Figure 11 is a block diagram of a road condition refreshing device according to an example embodiment of the present application. The device can be applied to the implementation environment shown in Figure 2 , and is specifically configured in the controller 220. The device can also be applied to other example implementation environments, and is specifically configured in other devices, and the implementation environment to which the device is applied is not limited in the present embodiment.

[0101] As shown in Figure 11 , the example automatic testing device includes a determination module 1110 configured to determine the target control and the target state corresponding to the target control based on the scene to be tested; an acquisition module 1120 configured to acquire the original state of the target control in the current visual interface, and determine the automatic operation instruction of the target control based on the original state and the target state; an adjustment module 1130 configured to adjust the state of the target control based on the automatic control instruction to obtain the adjusted visual interface, and extract the actual state corresponding to the target control in the adjusted visual interface; and a judgment module 1140 configured to determine that the test of the target control passes if the actual state matches the target state.

[0102] According to an aspect of some embodiments of the present application, the obtaining module 1120 is further configured to perform image detection on the visual interface based on the target detection model to obtain an original detection result; and if a confidence level of the original detection result is higher than a preset confidence threshold, determine an original state of the target control based on the detection result, the original state including an original center point coordinate, an original bounding box, and an original rotation angle of the target control.

[0103] According to an aspect of some embodiments of the present application, the obtaining module 1120 is further configured to determine a target center point coordinate, a target bounding box, and a target rotation angle corresponding to the target control based on the to-be-tested scene; determine a plurality of operation values based on the target center point coordinate, the target bounding box, and the target rotation angle and the original center point coordinate, the original bounding box, and the original rotation angle; generate a plurality of operations corresponding to the drive-level operation based on the plurality of operation values, and generate the automation operation instruction based on the plurality of operations.

[0104] According to an aspect of some embodiments of the present application, the adjusting module 1130 is further configured to perform image detection on the adjusted visual interface based on the target detection model to obtain an adjusted detection result; and if a confidence level of the adjusted detection result is higher than a preset confidence threshold, determine an actual state of the target control based on the adjusted detection result.

[0105] According to an aspect of some embodiments of the present application, the automation testing apparatus further includes a training module configured to obtain a control image set in the visual interface, and label based on positions of the controls on the visual interface and functions of the controls to obtain labels corresponding to the controls; generate a control image label set based on the control image set and the labels corresponding to the controls, the control image label set including center point coordinates, bounding boxes, and rotation angles corresponding to the controls; and train the target detection model based on the control image label set.

[0106] According to an aspect of some embodiments of the present application, the determining module 1110 is further configured to analyze the to-be-tested scene to obtain a key function corresponding to the to-be-tested scene; determine a control label corresponding to the key function to find a matching target control based on the control label.

[0107] According to an aspect of some embodiments of the present application, the determining module 1110 is further configured to analyze the to-be-tested scene to obtain a key function corresponding to the to-be-tested scene; determine a control label corresponding to the key function to find a matching target control based on the control label.

[0108] It should be noted that the automatic test device provided in the above embodiments and the automatic test method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The automatic test device provided in the above embodiments can be divided into different functional modules to complete the above-described functions in actual application, i.e., the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions, and this is not limited herein.

[0109] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the automatic test method provided in each of the above embodiments.

[0110] Figure 12 The structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 12 The computer system 1200 of the electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0111] As Figure 12 shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1202 or programs loaded from a storage portion 1208 into a random access memory (RAM) 1203, such as performing the methods in the above embodiments. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0112] The following components are connected to the I / O interface 1205: an input part 1206 including a keyboard, a mouse, etc.; an output part 1207 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 1208 including a hard disk, etc.; and a communication part 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable media 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed in the storage part 1208 as necessary.

[0113] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 1209, and / or installed from the removable media 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0114] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0115] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems, which perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0116] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0117] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the automatic test method as described above. The computer readable storage medium can be included in the electronic device as described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0118] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the automatic test method provided in each of the above embodiments.

[0119] The above is only a preferred exemplary embodiment of the present application, and is not intended to limit the implementation of the present application. Those skilled in the art can easily make corresponding modifications or variations according to the main concept and spirit of the present application, and the protection scope of the present application should be subject to the protection scope required by the claims.

Claims

1. An automated testing method, characterized by, The method comprises: determining a target control and a target state corresponding to the target control based on a to-be-tested scene; obtaining an original state of the target control in a current visual interface, and determining an automation operation instruction of the target control based on the original state and the target state; adjusting the state of the target control based on the automation operation instruction to obtain an adjusted visual interface, and extracting an actual state corresponding to the target control in the adjusted visual interface; if the actual state matches the target state, determining that the test of the target control is passed; the method further comprises: performing image detection on the visual interface based on a target detection model to obtain an original detection result; if the confidence of the original detection result reaches a preset confidence threshold, determining the original state of the target control based on the detection result, wherein the original state comprises an original center point coordinate, an original bounding box and an original rotation angle of the target control; determining a target center point coordinate, a target bounding box and a target rotation angle corresponding to the target control based on the to-be-tested scene; determining a plurality of operation values based on the target center point coordinate, the target bounding box and the target rotation angle and the original center point coordinate, the original bounding box and the original rotation angle; generating a plurality of operations corresponding to a driving level operation based on the plurality of operation values, and generating the automation operation instruction based on the plurality of operations.

2. The method of claim 1, wherein, The method further comprises: performing image detection on the adjusted visual interface based on the target detection model to obtain an adjusted detection result; if the confidence of the adjusted detection result reaches a preset confidence threshold, determining the actual state corresponding to the target control based on the adjusted detection result.

3. The method of claim 1, wherein, The method further comprises: obtaining a control image set in the visual interface, and labeling based on the position of the control on the visual interface and the function of the control to obtain a label corresponding to the control; generating a control image label set based on the control image set and the label corresponding to the control, wherein the control image label set comprises a center point coordinate, a bounding box and a rotation angle corresponding to the control; training the target detection model based on the control image label set.

4. The method of claim 1, wherein, The method further comprises: parsing the to-be-tested scene to obtain a key function corresponding to the to-be-tested scene; determining a corresponding control label based on the key function, and searching for a matching target control based on the control label.

5. The method of claim 1, wherein, The method further comprises: determining an expected result corresponding to the to-be-tested scene; obtaining an operation result corresponding to the target control after the automation operation instruction is executed; if the operation result matches the expected result, determining that the test of the target control is passed.

6. An automated test device characterized by, The device comprises: a determination module configured to determine a target control and a target state corresponding to the target control based on a to-be-tested scene; The acquisition module is configured to acquire an original state of the target control in the current visual interface, and determine an automation operation instruction of the target control based on the original state and the target state; and configured to perform image detection on the visual interface based on a target detection model to obtain an original detection result; if a confidence of the original detection result reaches a preset confidence threshold, determine the original state of the target control based on the detection result, the original state including an original center point coordinate, an original bounding box, and an original rotation angle of the target control; and configured to determine a target center point coordinate, a target bounding box, and a target rotation angle corresponding to the target control based on the to-be-tested scene; determine a plurality of operation values based on the target center point coordinate, the target bounding box, and the target rotation angle and the original center point coordinate, the original bounding box, and the original rotation angle; generate a plurality of operations corresponding to a drive-level operation based on the plurality of operation values, and generate the automation operation instruction based on the plurality of operations; The adjustment module is configured to adjust the state of the target control based on the automation operation instruction, obtain an adjusted visual interface, and extract an actual state of the target control in the adjusted visual interface; The judgment module is configured to determine that the test of the target control is passed if the actual state matches the target state.

7. An electronic device, comprising: The electronic device includes: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the automation test method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, a computer readable instruction stored thereon, which, when executed by a processor of a computer, causes the computer to perform the automation test method of any one of claims 1 to 5.

Images