Cross-device automated testing method and related devices

Abstract

This application discloses a cross-device automated testing method and related equipment, relating to the field of testing technology. The method includes: constructing a multi-dimensional UI feature map based on first UI element information of a first device; if a test command for a second device is received, matching the second UI element information of the second device based on the multi-dimensional UI feature map to determine the display position information of the second UI element corresponding to the test command on the second device; instantiating the test command into a target operation command based on the display position information; executing the target operation command through the second device and verifying the response interface of the second device to determine the test result for the second device. This application, by constructing a multi-dimensional UI feature map and matching and instantiating UI elements and commands for different devices based on it, can improve the universality of automated testing in multi-device scenarios, the accuracy of element location, the efficiency of script maintenance, and the reliability of test results.

Description

Technical Field

[0001] This application relates to the field of testing technology, and more specifically, to a cross-device automated testing method and related equipment. Background Technology

[0002] With the continuous expansion of software system scale and the increasing diversification of terminal device forms, automated testing technology has become an important means to ensure software quality and stability. Especially in application scenarios such as in-vehicle systems, smart terminals, and embedded devices, the same application often needs to adapt to various models, resolutions, and system configurations of devices, which places higher demands on the versatility and stability of automated testing.

[0003] However, existing automated testing solutions typically rely on fixed coordinates, resource identifiers, or single attributes to locate user interface (UI) elements. When test scripts are migrated from one device to another, the display position and appearance of UI elements can easily change due to differences in screen size, resolution, aspect ratio, or interface layout. This can cause test instructions to fail to accurately target the elements, requiring frequent adjustments or even rewriting of the test scripts, resulting in high maintenance costs and low efficiency. In other words, related technologies generally suffer from the technical problems of strong dependence on specific devices, insufficient cross-device adaptability, and high script maintenance costs. Summary of the Invention

[0004] The summary section of this application introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this application is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0005] The cross-device automated testing method and related equipment provided in this application can construct a multi-dimensional feature map of UI and match and instantiate UI elements of different devices based on it. Under the premise of ensuring that the operation results are verifiable, it can decouple the test instructions from the specific devices, thereby improving the universality of automated testing in multi-device scenarios, the accuracy of element positioning, the efficiency of script maintenance, and the reliability of test results.

[0006] In a first aspect, this application provides a cross-device automated testing method, comprising: constructing a multi-dimensional UI feature map based on first UI element information of a first device; if a test instruction for a second device is received, matching second UI element information of the second device based on the multi-dimensional UI feature map to determine the display position information of the second UI element corresponding to the test instruction on the second device; instantiating the test instruction into a target operation instruction based on the display position information; executing the target operation instruction through the second device and verifying the response interface of the second device to determine the test result for the second device.

[0007] In some implementations, constructing a multi-dimensional UI feature map based on the first UI element information of the first device includes: extracting first multi-dimensional features of the first UI element of the first device based on a first interface screenshot of the first device, wherein the first multi-dimensional features include geometric features, visual features, and semantic features; generating a first feature identifier corresponding to the first UI element based on the first multi-dimensional features; associating the first feature identifier with the first functional semantic of the corresponding first UI element, and storing it as a reference node in the multi-dimensional UI feature map.

[0008] In some implementations, extracting the first multi-dimensional features of the first UI element of the first device based on the first interface screenshot of the first device includes: performing target detection on the first interface screenshot to locate the region image of the first UI element; determining the geometric features based on the contour information of the region image and the relative positional relationship between the contour information and preset interface anchor points; processing the region image through a preset visual feature extraction model to obtain the visual features representing color and texture distribution information; extracting the text information contained in the region image through an optical character recognition tool, and semantically embedding the text information based on a natural language processing model to obtain the semantic features.

[0009] In some implementations, the step of matching the second UI element information of the second device based on the UI multi-dimensional feature map to determine the display position information of the second UI element corresponding to the test command on the second device includes: parsing the test command to obtain the second functional semantics of the second UI element; extracting the second multi-dimensional features of the second UI element and generating a second feature identifier corresponding to the second multi-dimensional features; calculating the similarity between the second multi-dimensional features and the first feature identifier of the target reference node with the second functional semantics in the UI multi-dimensional feature map to obtain the target similarity; if the target similarity is greater than a preset similarity threshold, then the second UI element is used as an instance node of the target reference node, and a mapping relationship is established with the target reference node; based on the mapping relationship and the screen parameters of the second device, the display position information of the second UI element on the second device is determined.

[0010] In some implementations, instantiating the test instruction into a target operation instruction based on the display position information includes: determining the operation type and initial operation parameters to be executed based on the test instruction; converting the initial operation parameters into instantiated operation parameters adapted to the screen coordinate system of the second device based on the display position information; and generating the target operation instruction based on the operation type and the instantiated operation parameters.

[0011] In some implementations, verifying the response interface of the second device to determine the test result for the second device includes: performing a structural comparison between a screenshot of the second interface of the response interface and a benchmark reference image corresponding to the test instruction to obtain a first verification result; searching for the expected third UI element in the screenshot of the second interface based on the UI multi-dimensional feature map to obtain a second verification result; and determining whether the test instruction was successfully executed based on the first verification result and the second verification result, and generating the test result.

[0012] In some implementations, determining whether the test instruction was successfully executed based on the first verification result and the second verification result includes: if the structural similarity indicated by the first verification result is greater than a first preset threshold and the matching confidence of the second verification result is greater than a second preset threshold, then the test instruction is determined to have been successfully executed; if the structural similarity indicated by the first verification result is less than or equal to the first preset threshold, or the matching confidence of the second verification result is less than or equal to the second preset threshold, then the test instruction is determined to have failed to execute.

[0013] Secondly, this application also provides a cross-device automated testing apparatus, comprising: a map construction unit, configured to construct a multi-dimensional UI feature map based on first UI element information of a first device; a position matching unit, configured to, upon receiving a test instruction for a second device, match second UI element information of the second device based on the multi-dimensional UI feature map to determine the display position information of the second UI element corresponding to the test instruction on the second device; an instruction instantiation unit, configured to instantiate the test instruction into a target operation instruction based on the display position information; and a test verification unit, configured to execute the target operation instruction through the second device and verify the response interface of the second device to determine the test result for the second device.

[0014] Thirdly, this application also provides an electronic device, including: a memory and a processor, wherein the processor is configured to implement the steps of the cross-device automated testing method described in the first aspect when executing a computer program stored in the memory.

[0015] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions or a computer program, which, when executed by a processor, implement the steps of the cross-device automated testing method described in the first aspect.

[0016] Fifthly, this application also provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the steps of the cross-device automated testing method provided in the embodiments of this application.

[0017] In summary, this application constructs a multi-dimensional UI feature map based on the first UI element information of the first device, so that test commands no longer depend on the fixed coordinates or unique identifiers of a specific device. When testing a second device, the feature map is used to match the second UI element information and determine its display position information, thereby enabling the same test command to adapt to the UI differences of different devices and improving the versatility of automated testing in multi-device scenarios. By matching the UI elements of the second device with the feature map before determining the display position information, compared to directly using static coordinates or single attributes for element positioning, it can still accurately locate the target corresponding to the test command even when there are differences in device UI layout, resolution, or display form. UI elements are used to reduce the probability of test failures due to UI changes. Test instructions exist only in an abstract form before execution, and are instantiated only after the display position information of the corresponding UI elements on the second device is determined. This avoids writing or repeatedly modifying test scripts for different devices. When the test device changes, there is no need to redesign the test instructions; only element matching needs to be performed again to complete the test, which improves test maintenance efficiency. After the target operation instruction is executed on the second device, the response interface of the second device is verified to determine the test result. This means that the test process includes not only operation execution but also result verification, which helps to promptly identify situations where the operation does not take effect or the interface does not meet expectations, thereby improving the accuracy and reliability of automated test results. In summary, the cross-device automated testing method provided in this application constructs a multi-dimensional UI feature map and matches and instantiates UI elements and instructions on different devices based on it. While ensuring the verifiability of operation results, it can decouple test instructions from specific devices, thereby improving the universality of automated testing in multi-device scenarios, the accuracy of element location, the efficiency of script maintenance, and the reliability of test results. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a cross-device automated testing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the composition structure of a cross-device automated testing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The terms used in the specification, claims, and drawings of this application, such as "first," "second," "third," "fourth," etc. (if any), are used to distinguish similar objects and not to describe a specific order or sequence. Therefore, it is to be understood that these terms can be used interchangeably where appropriate, allowing the described embodiments to be used in different orders, unless specifically required by the illustrations or description. Furthermore, the terms "is" and "has," and any variations thereof, are intended to cover, non-exclusively, all possible constituent elements. For example, a process, method, system, product, or apparatus comprising several steps or units is not necessarily limited to the steps or units explicitly listed, but may also include other steps or units not explicitly listed, or steps or units inherent to the process, method, product, or apparatus.

[0020] In this application, a "module" or "unit" refers to a computer program or part of a computer program that has a specific function and works in conjunction with other related parts to achieve a predetermined goal. These modules or units can be implemented by software, hardware (e.g., processing circuitry or memory), or a combination of both. One or more processors or memories can implement one or more modules or units. Furthermore, each module or unit can also be part of a larger module or unit.

[0021] The technical solutions of this application will be described in detail below with reference to the accompanying drawings of the embodiments. It should be noted that the described embodiments are only a part of this application, and not all embodiments. In the following description, the "some embodiments" mentioned are only a subset of all possible embodiments, which may be the same or different subsets, and different embodiments can be combined with each other without conflict.

[0022] Figure 1 This is a flowchart illustrating a cross-device automated testing method provided in an embodiment of this application. For example, see [link to example]. Figure 1 The cross-device automated testing method provided in this application embodiment may include the following steps 101 to 104: Step 101: Construct a multi-dimensional feature map of the UI based on the first UI element information of the first device.

[0023] In some examples, the first device is the benchmark device used in cross-device automated testing. The first UI element information of the first device is the basis for constructing a multi-dimensional UI feature map, and it needs to be representative and able to cover the UI scenarios of the function to be tested. A device with mainstream in-vehicle intelligent operating system (Android In-Vehicle Infotainment, IVI) and screen parameters (resolution, size) that are common in the industry can be selected from the list of in-vehicle devices to be tested as the first device. For example, the first device can be the central control device of a 2025 mid-size sedan of a certain brand. This device is equipped with a customized version of Android 12 in-vehicle system, with a screen resolution of 1920×1080, covering navigation, entertainment, vehicle control and other functional modules to be tested. The first UI element information refers to the attributes and association information of all user interface elements (i.e., first UI elements) related to the function under test on the first device. It is the data source for constructing the feature map and covers various information related to the appearance, structure, and function of the elements. The screen capture tool of the vehicle device can be used to collect the interface image (such as the navigation start interface or music playback interface) corresponding to the function under test on the first device, and extract the relevant information of all valid UI elements (such as buttons, input boxes, icons, text boxes, etc.) in the image. For example, it can be the relevant information of elements such as the "Start Navigation" button, "Destination Input Box", and "Route Preference Selection Icon" in the navigation function interface of the above-mentioned central control device, including the presentation form, position association and function direction of the elements in the interface. A UI multi-dimensional feature map integrates the multi-dimensional features of the first UI element, establishes the association between features and the element's primary functional semantics, and forms a structured, reusable feature set. It serves as a carrier for cross-device UI element matching. Based on the collected information of the first UI element, it can extract the multi-dimensional features of each element, bind the features to the element's primary functional semantics, and organize all the bound data into a structured map according to a preset map data structure, storing it in a database for subsequent matching and retrieval. For example, a UI multi-dimensional feature map can integrate the multi-dimensional features of the "Start Navigation" button, bind it to the button's primary functional semantics of "Trigger Navigation Route Planning," and simultaneously associate it with the features and semantics of related elements such as the "Destination Input Box" and the "Route Preference Selection Icon," forming a structured map containing the feature associations of all UI elements in the navigation function module.

[0024] By implementing step 101, a multi-dimensional feature map of UI is constructed based on the first UI element information of the first device. This allows UI elements to no longer be described by a single coordinate or identifier attribute, but to be uniformly represented by multi-dimensional features. This provides a stable and universal reference basis for subsequent UI element matching between different devices, thereby improving the applicability of the automated testing solution in a multi-device environment.

[0025] Step 102: If a test instruction for the second device is received, the second UI element information of the second device is matched based on the multi-dimensional feature map of the UI to determine the display position information of the second UI element corresponding to the test instruction on the second device.

[0026] In some examples, the second device is the target in-vehicle device to be verified during cross-device automated testing. Together with the first device, it forms a device combination for cross-device testing. Its device model, screen parameters, and in-vehicle system version may differ from the first device, serving as the actual execution vehicle for test instructions. For example, it could be the central control device of a 2026 compact SUV from a certain brand. This device runs a customized version of the Android 13 in-vehicle intelligent operating system, has a screen resolution of 2560×1440, and supports navigation, entertainment, vehicle control, and other functions to be tested that are consistent with the first device. Test instructions are operational commands that trigger the second device to execute specific functions to be tested. These test instructions are written based on functional semantics, are not bound to the specific device's hardware parameters or UI layout, and are device-independent. They can be input by testers through an automated testing platform or automatically generated and sent to the execution module based on preset test cases. For example, they could be functional semantics-based commands such as "trigger navigation function," "click the music play button," or "enter destination address," without specifying specific operation coordinates or device-specific identifiers. The second UI element information consists of the attributes and association information of all user interface elements related to the function under test corresponding to the test command on the second device. It covers the element's presentation form, structural features, and functional orientation, and is data matched with the multi-dimensional feature map of UI. The interface image of the function under test corresponding to the test command on the second device can be collected in real time using an in-vehicle screen capture tool, and then the relevant information of all valid UI elements (such as buttons, icons, input boxes, etc.) can be extracted from the image. For example, the relevant information of elements such as the "Navigation Start" button, "Address Input Box", and "Confirmation Icon" in the navigation function interface of the above-mentioned compact SUV central control device can be obtained, including the appearance of the elements, their relative position in the interface, and their functional association.

[0027] The second UI element corresponding to the test command is a specific UI element on the second device that can respond to the test command and implement the function corresponding to the command. Its functional semantics are consistent with the requirements of the test command. The functional semantics of the test command can be parsed, and the UI element that matches the semantics can be selected by combining the functional orientation of the second UI element information. For example, if the test command is "trigger navigation function", the corresponding second UI element is the "navigation start" button in the navigation interface of the second device; if the test command is "play music", the corresponding second UI element is the "music play" icon in the entertainment interface. The display position information is the precise position data of the second UI element corresponding to the test command on the screen of the second device. This data is generated based on the screen coordinate system of the second device and can directly support the instantiation and execution of the test command. The specific position data of the UI element in the screen coordinate system can be calculated by combining the screen parameters of the second device (such as resolution and screen size) and the relative position relationship of the UI element in the interface. For example, the display position information of the "navigation start" button in the above-mentioned compact SUV central control device can be the coordinates (x:1200, y:900) in the screen coordinate system. This coordinate precisely corresponds to the actual position of the button on the 2560×1440 resolution screen.

[0028] By implementing step 102, when a test instruction for the second device is received, the UI element information of the second device is matched using the constructed UI multi-dimensional feature map, and the display position information of the corresponding UI element on the second device is determined. This allows the test process to adapt to the differences in interface layout, resolution, or display form of different devices, improves the accuracy of target UI element positioning, and reduces the risk of test failure due to device differences.

[0029] Step 103: Based on the display location information, instantiate the test instruction into the target operation instruction.

[0030] In some examples, instantiation is the process of transforming abstract, device-independent test instructions into executable instructions adapted to the specific hardware parameters (such as screen coordinates and operation logic) of the second device. This allows for the filling of specific operation parameters specific to the second device into the abstract instructions, achieving precise adaptation between the test logic and the target device. Based on the display position information determined in step 102, combined with the screen characteristics of the second device, parameter injection and format conversion can be performed on the test instructions. For example, the abstract "trigger navigation function" test instruction can be filled with the specific coordinates and operation method of the second device's "navigation start" button, transforming it into an executable operation instruction adapted to that device, thus completing instantiation. The target operation instruction is the specific operation instruction generated after instantiation that can be directly executed on the second device. It includes a clear operation type, a precise execution location, and operation parameters adapted to the second device, meeting the requirements of the second device's automated execution framework.

[0031] By implementing step 103, the abstract test instructions are instantiated into target operation instructions that can be directly executed on the second device based on the determined display position information. This allows the test instructions to be flexibly adapted to specific devices while maintaining device independence, avoiding the need to write or repeatedly modify test scripts for different devices. This effectively reduces the maintenance cost of test scripts and improves test execution efficiency.

[0032] Step 104: Execute the target operation command through the second device and verify the response interface of the second device to determine the test results for the second device.

[0033] In some examples, the process of executing a target operation command through a second device and verifying the response interface of the second device involves first driving the second device to complete the specific operation according to the requirements of the target operation command, then collecting the response interface after the operation, comparing the response interface with the standard interface information corresponding to the first device in the UI multi-dimensional feature map, and determining whether the operation execution effect meets expectations. Alternatively, a target operation command can be issued through an automation execution framework compatible with the second device (such as UIAutomator). After the command is executed, a screen capture tool is triggered to collect the response interface, and then benchmark data in the UI multi-dimensional feature map is called for consistency verification. For example, a target operation command can be issued to the central control device of a 2026 compact SUV of a certain brand via the UI Automator framework to click the coordinates (1200, 900) to start navigation. After the device executes the command, the navigation startup interface is collected as the response interface, and then this interface is compared with the standard navigation startup interface features of the first device in the UI multi-dimensional feature map to complete the verification process.

[0034] The test results for the second device are based on the verification results of the response interface, and are the final conclusion on whether the second device has successfully implemented the function corresponding to the test command. This result directly reflects the effectiveness of the cross-device test and can include two types of conclusions: "test passed" or "test failed". The conclusion can be automatically determined and output based on the verification comparison results between the response interface and the benchmark information. If the response interface is consistent with the benchmark information, it is considered "test passed"; otherwise, it is considered "test failed". For example, if the navigation startup response interface of the second device is consistent with the standard interface features in the UI multi-dimensional feature map, it can be determined that "test passed", indicating that the navigation function of the second device can be triggered normally; if the response interface does not show the navigation startup state (such as still staying on the original interface or displaying an error message), it is determined that "test failed".

[0035] By implementing step 104, the target operation command is executed on the second device and its response interface is verified. This means that the testing process includes not only operation execution but also verification of the execution results. This helps to promptly identify situations where the operation is ineffective or the interface state does not meet expectations, thereby improving the accuracy and reliability of automated test results.

[0036] In summary, this application embodiment constructs a multi-dimensional UI feature map based on the first UI element information of the first device, so that test instructions no longer depend on the fixed coordinates or unique identifiers of a specific device. When testing is performed on a second device, the feature map is used to match the second UI element information and determine its display position information. This allows the same test instruction to adapt to the UI differences of different devices, improving the versatility of automated testing in multi-device scenarios. By matching the UI elements of the second device with the feature map before determining the display position information, compared to directly using static coordinates or single attributes for element positioning, it can still accurately locate the target corresponding to the test instruction even when there are differences in device UI layout, resolution, or display form. By labeling UI elements, the probability of test failure due to UI changes is reduced. Test instructions exist only in an abstract form before execution, and are instantiated only after the display position information of the corresponding UI elements on the second device is determined. This avoids writing or repeatedly modifying test scripts for different devices. When the test device changes, there is no need to redesign the test instructions; only element matching needs to be performed again to complete the test, which improves test maintenance efficiency. After the target operation instruction is executed on the second device, the response interface of the second device is verified to determine the test result. This means that the test process includes not only operation execution but also result verification, which helps to promptly identify situations where the operation does not take effect or the interface does not meet expectations, thereby improving the accuracy and reliability of automated test results. In summary, the cross-device automated testing method provided in this application constructs a multi-dimensional UI feature map and matches and instantiates UI elements and instructions on different devices based on it. While ensuring that the operation results are verifiable, it can decouple test instructions from specific devices, thereby improving the universality of automated testing in multi-device scenarios, the accuracy of element positioning, the efficiency of script maintenance, and the reliability of test results.

[0037] In some embodiments, the aforementioned step 101 may include: extracting a first multi-dimensional feature of a first UI element of the first device based on a first interface screenshot of the first device, wherein the first multi-dimensional feature includes geometric features, visual features and semantic features; generating a first feature identifier corresponding to the first UI element based on the first multi-dimensional feature; associating the first feature identifier with the first functional semantic of the corresponding first UI element, and storing it as a reference node in the UI multi-dimensional feature map.

[0038] In some examples, the first screenshot is the user interface image corresponding to the functional module under test on the first device. It serves as the basic data source for extracting information on the first UI elements and multi-dimensional features, and can fully present the distribution, shape, and relationships of all UI elements in the interface. For example, the central control device of a 2025 mid-size sedan of a certain brand, equipped with an Android 12 in-vehicle intelligent operating system, captures a 1920×1080 resolution interface image containing elements such as the "Start Navigation" button and the "Destination Input Box" after the navigation function is activated. The first multi-dimensional features are a set of features extracted from the first UI elements, covering structural, appearance, and functional attributes. They can include geometric features, visual features, and semantic features, and serve as the basis for constructing unique identifiers for UI elements. Targeted analysis can be performed on the first UI elements in the first screenshot, and the three types of features can be extracted and integrated to form a multi-dimensional feature set. Geometric features are attributes corresponding to the spatial structure and position of the first UI element. They reflect the element's morphological regularity and positional relationship with other elements, possessing stability for cross-device adaptation. Image contour recognition technology can be used to analyze the regional boundaries, size ratios, and relative positional relationships of the first UI element with screen edges or anchor points. For example, the geometric features of the "Start Navigation" button could include its dimensions of 300 pixels wide and 150 pixels high, its relative position to the bottom edge of the screen (200 pixels away), and its proximity to the "Destination Input Box." Visual features are attributes corresponding to the appearance of the first UI element. They can capture subtle visual information such as the element's color, texture, and brightness, reflecting the element's visual essence. Image feature extraction technology can be used to process the regional image of the first UI element and extract feature data that can characterize visual attributes. For example, the visual features of the "Start Navigation" button could include the color distribution of red RGB values ​​in the range of (255,0,0)-(255,50,50) and the texture features of its matte surface. Semantic features are the functional meanings carried by the first UI element. They are obtained through text information mining or functional association analysis and can overcome the limitations of visual or structural differences. The text content within the first UI element can be extracted using optical character recognition tools, and then the semantics of the text can be analyzed using natural language processing technology. For example, the semantic features of the "Start Navigation" button can be obtained by extracting the "Start Navigation" text on the button and processing it with natural language to obtain the semantic meaning of "start navigation path calculation and guidance".

[0039] The first feature identifier is a standardized data carrier that uniquely identifies any UI element, generated based on the first multi-dimensional features of the first UI element of the first device. It possesses device independence and uniqueness. A preset feature fusion algorithm can be used to integrate geometric, visual, and semantic features to generate a unified format identifier. For example, the first feature identifier for the "Start Navigation" button can be a 64-dimensional high-dimensional vector (e.g., [0.12, 0.35, ..., 0.78]) generated by a weighted fusion algorithm from its geometric, visual, and semantic feature vectors. The process of generating the first feature identifier corresponding to the first UI element based on the first multi-dimensional features involves using the geometric, visual, and semantic features of the first UI element as input, and generating standardized data that uniquely identifies the element through a standardized feature fusion and encoding process. For example, the three types of features can be first normalized, then weighted and summed according to preset weights (e.g., geometric features 0.3, visual features 0.4, semantic features 0.3), and finally converted into a unified format identifier through an encoding algorithm.

[0040] The first functional semantics refers to the specific functional effect achieved after the first UI element is operated. It is the value attribute of the first UI element and directly corresponds to the user's operation intention. The functional direction of each first UI element can be determined by combining the functional design document of the first device, the operation logic of the UI element, and the effect verification after actual operation. For example, the first functional semantics of the "Start Navigation" button can be "After receiving the user's click operation, trigger the in-vehicle navigation system to perform route planning and display the planned navigation path"; the first functional semantics of the "Music Pause" icon can be "After receiving the user's click operation, stop the playback of the current audio file".

[0041] A baseline node is a basic data unit in the multi-dimensional feature graph of a UI that stores information about the first UI element. It serves as a benchmark reference carrier for subsequent cross-device element matching and contains the association between the first feature identifier and the first functional semantics. According to the preset data structure of the graph, the associated first feature identifier and the first functional semantics can be encapsulated into a structured data unit, namely a baseline node. For example, the baseline node corresponding to the "Start Navigation" button can contain its first feature identifier in the form of a 64-dimensional vector and the first functional semantics of "trigger navigation path planning and display". The two are encapsulated into the baseline node of the graph in the form of key-value pairs.

[0042] Through the implementation of the above embodiments, geometric features, visual features, and semantic features of UI elements are extracted from the screenshot of the interface of the first device, and feature identifiers associated with the first functional semantics are generated and stored in the UI multi-dimensional feature map as reference nodes. This enables the in-vehicle UI elements to be uniformly represented in a way that is independent of the specific device implementation. It can maintain the consistency of the first functional semantics of the elements under different in-vehicle systems or different UI styles, providing a stable reference for subsequent cross-device matching and fundamentally reducing the problem of difficulty in automated testing and adaptation caused by the high degree of customization of in-vehicle systems.

[0043] In some embodiments, the aforementioned extraction of the first multi-dimensional features of the first UI element of the first device based on the first interface screenshot of the first device may include: performing target detection on the first interface screenshot to locate the region image of the first UI element; determining geometric features based on the contour information of the region image and the relative positional relationship between the contour information and the preset interface anchor points; processing the region image through a preset visual feature extraction model to obtain visual features representing color and texture distribution information; extracting text information contained in the region image through an optical character recognition tool, and semantically embedding the text information based on a natural language processing model to obtain semantic features.

[0044] In some examples, object detection is a technical process that uses a specific algorithm model to analyze a first interface screenshot, accurately identify and locate all the first UI elements to be tested, and is a preliminary step for extracting the first multi-dimensional features. A preset lightweight object detection model can be integrated to perform inference calculations on the input first interface screenshot and output the position coordinates of the UI elements. For example, the MobileNet-SSD (Single Shot MultiBox Detector) model can be used to detect the 1920×1080 resolution interface screenshot of the first device navigation function and locate the rectangular area coordinates of the first UI elements such as the "Start Navigation" button and the "Destination Input Box". A region image is a partial image cropped from a screenshot of the first interface after the position coordinates of the first UI element are determined by object detection. It contains only the first UI element and serves as a direct data source for subsequent feature extraction. Based on the element rectangle coordinates output by object detection, such as the top left corner (x1, y1) and bottom right corner (x2, y2), the first interface screenshot can be precisely cropped to generate an independent partial image file. For example, it can be a 300×150 pixel partial image cropped from the first interface screenshot of the navigation function based on the coordinates of the "Start Navigation" button (x1:800, y1:600, x2:1100, y2:750). This image only contains the complete shape of the "Start Navigation" button.

[0045] Contour information refers to the boundary shape and size data of the region image of the first UI element, reflecting the basic morphological characteristics of the element. Edge detection algorithms such as the Canny algorithm can be used to process the UI element region image, extract the coordinates of the boundary contour points of the element, and then calculate the size and shape parameters of the contour. The relative positional relationship between the contour information and the preset interface anchor points is the spatial positional association data between the contour boundary of the UI element and the preset fixed reference points (anchor points) in the first interface, which is the basis for determining geometric features. Anchor points of the first interface (such as the four corners of the screen, the bottom edge of the status bar, etc.) can be preset, and the straight-line distance and orientation relationship between the key nodes of the UI element contour (such as the center point, the top left corner point) and the anchor points can be calculated. The process of determining geometric features based on the contour information of the region image and the relative positional relationship between the contour information and the preset interface anchor points is a process of integrating the contour size data of the UI element region image and the positional data of the contour relative to the anchor points to form a set of geometric features that can characterize the spatial attributes of the element. Parameters such as aspect ratio and contour shape type can be extracted from the contour information first, and then combined with distance and orientation data from the relative positional relationship to jointly constitute geometric features. For example, the 2:1 aspect ratio, rectangular contour type, and 200-pixel vertical distance from the bottom anchor point of the "Start Navigation" button can be integrated to form the complete geometric features of the button.

[0046] The preset visual feature extraction model is a pre-trained algorithm model integrated into the system for extracting visual attributes of UI elements. It is lightweight and high-performance, and can adapt to the resource requirements of in-vehicle testing scenarios. It can be a lightweight convolutional neural network (CNN) model pre-installed with in-vehicle UI scene data for fine-tuning. For example, it can be a lightweight convolutional neural network model that has been trained and fine-tuned with massive in-vehicle UI element images and can quickly extract visual features such as color and texture. The process of processing a region image using a preset visual feature extraction model to obtain visual features representing color and texture distribution information involves inputting the region image of the first UI element into the preset visual feature extraction model, performing operations such as convolution and pooling, and outputting high-dimensional vector data that can quantify the color distribution pattern and texture mode of the image. The region image can be scaled to a standard size (e.g., 224×224 pixels) according to the model requirements, input into the model for forward inference, and the output vector of the last fully connected layer of the model is extracted as the visual feature. For example, the 300×150 pixel region image of the "Start Navigation" button can be scaled to 224×224 pixels, input into the MobileNet model, and output a 128-dimensional high-dimensional vector, which accurately represents the color distribution of the button's red gradient and the texture information of its frosted texture.

[0047] Optical Character Recognition (OCR) tools are software tools that can recognize printed or handwritten text in images and convert it into editable text data. They are used to extract text information from UI element area images. The text information contained in the area image is all the visible text content presented in the UI element area image, which is the basic data for mining the semantic features of elements. For example, using the Tesseract optical character recognition tool to recognize the area image of the "Start Navigation" button, the output text string "Start Navigation" can be generated; recognizing the area image of the "Destination Input Box" can output the prompt text "Please enter your destination". Natural Language Processing (NLP) models are algorithmic models that can perform semantic analysis and encoding on text information, converting it into quantized semantic vectors, to mine the functional meaning behind the text. For example, a lightweight model using Bidirectional Encoder Representations from Transformers (BERT) can be adapted to the semantic extraction needs of in-vehicle UI text. Semantic embedding is the process of transforming textual information into high-dimensional vectors using a natural language processing (NLP) model. The transformed vectors reflect the semantic relationships between the texts in a vector space, with semantically similar texts having closer corresponding vectors. For example, inputting the text "Start Navigation" into a lightweight BERT model and encoding it into a 768-dimensional semantic vector results in a vector that is spatially close to the semantic vector of the text "Start Navigation." The process of extracting textual information from a region image using optical character recognition (OCR) tools and then semantically embedding this information using a NLP model to obtain semantic features involves first extracting visible text from the UI element region image using an OCR tool, then inputting the text into an NLP model for semantic encoding, ultimately generating quantified features that represent the element's primary semantic function.

[0048] Through the implementation of the above embodiments, target detection is performed on the screenshot of the vehicle interface, and UI element features are extracted from multiple dimensions such as outline and relative position relationship, color and texture distribution, and text semantics. The constructed multi-dimensional features can simultaneously reflect the spatial structure characteristics, visual appearance characteristics, and primary functional semantic characteristics of UI elements in the vehicle interface. This can effectively cope with UI changes caused by resolution changes, layout adjustments, or language differences in the vehicle system, improve the recognizability and stability of UI elements in cross-device scenarios, and thus enhance the robustness of the overall automated testing.

[0049] In some embodiments, the aforementioned matching of the second UI element information of the second device based on the UI multi-dimensional feature map to determine the display position information of the second UI element corresponding to the test command on the second device may include: parsing the test command to obtain the second functional semantics of the second UI element; extracting the second multi-dimensional features of the second UI element and generating a second feature identifier corresponding to the second multi-dimensional features; calculating the similarity between the second multi-dimensional features and the first feature identifier of the target reference node with the second functional semantics in the UI multi-dimensional feature map to obtain the target similarity; if the target similarity is greater than a preset similarity threshold, then the second UI element is used as an instance node of the target reference node, and a mapping relationship is established with the target reference node; based on the mapping relationship and the screen parameters of the second device, the display position information of the second UI element on the second device is determined.

[0050] In some examples, parsing is a technical process of semantic analysis and core information extraction from received abstract test instructions. This clarifies the functional requirements corresponding to the test instructions, providing direction for subsequent UI element matching. Functional requirements can be extracted from test instructions through keyword extraction, syntax analysis, and other logic. For example, parsing the test instruction "trigger navigation function" can extract the functional requirement of "start navigation path planning," clarifying the UI element that needs to be matched with this function. The second functional semantics is the functional meaning that the UI element to be matched on the second device should possess, obtained after parsing the test instruction. It is essentially consistent with the functional semantics of the UI element on the first device and serves as the basis for cross-device element matching. The corresponding functional semantics can be directly mapped from the parsing of the test instructions. For example, the second functional semantics of "start navigation path planning and display" obtained after parsing the "trigger navigation function" instruction completely corresponds to the functional semantics of the "start navigation" button on the first device.

[0051] The second multi-dimensional feature is a comprehensive feature set extracted from the second UI element of the second device, encompassing geometric, visual, and semantic features. Its constituent dimensions are consistent with the first multi-dimensional feature, and it is used for similarity comparison with the map baseline features. The same feature extraction process as the first device can be used to extract the three types of features from the regional image of the second UI element and then integrate them. The second feature identifier is a standardized data carrier uniquely identifying the element, generated based on the second multi-dimensional feature of the second UI element. Its generation logic is completely consistent with the first feature identifier, ensuring compatibility for cross-device feature comparison. The generation algorithm of the first feature identifier can be used to normalize, weightedly fuse, and encode the second multi-dimensional feature. For example, it can be a 64-dimensional high-dimensional vector form of the second feature identifier generated by fusing the three types of feature vectors of the "Navigation Start" button according to preset weights and then encoding them using Base64.

[0052] The target baseline node is the baseline node of the first UI element in the UI multi-dimensional feature map that perfectly matches the second functional semantics. It serves as a reference benchmark for similarity comparison of the second UI element. Based on the second functional semantics, semantic retrieval can be performed in the UI multi-dimensional feature map to filter out baseline nodes with consistent functional semantics. For example, based on the second functional semantics of "start navigation route planning and display," the baseline node corresponding to the "Start Navigation" button on the first device can be retrieved in the map; this node is the target baseline node. Target similarity is the degree of matching between the second feature identifier and the first feature identifier of the target baseline node. It is an indicator for determining whether the second UI element and the first UI element are the same functional element. Algorithms such as cosine similarity or Euclidean distance can be used to quantify the second and first feature identifiers, outputting a value between 0 and 1. For example, the second feature identifier of the "Navigation Start" button and the first feature identifier of the "Start Navigation" button can be calculated using a cosine similarity algorithm, resulting in a target similarity value of 0.95. The preset similarity threshold is a pre-set critical value used to determine whether there is a mapping relationship between the second UI element and the target baseline node. This threshold can be optimized through in-vehicle UI scenario testing to balance recognition accuracy and adaptation flexibility. It can be preset with a fixed threshold and support fine-tuning as needed in the early stage of graph construction based on a large number of in-vehicle cross-device test samples. For example, the preset similarity threshold can be 0.9. When the target similarity is greater than 0.9, the two elements are determined to be the same functional element.

[0053] Instance nodes are the data carriers of the second UI element in the UI multi-dimensional feature map. They have a clear functional mapping relationship with the target baseline node and are an extension and supplement to the baseline node in the map. They are used to store the feature and device association information of the second device UI element. When the target similarity is greater than a preset threshold, a structured data unit containing the second feature identifier and the second device information can be automatically created and associated with the target baseline node as an instance node. For example, the instance node corresponding to the "Navigation Start" button can contain its 64-dimensional vector feature identifier, the second device model and screen parameters, and be associated with the target baseline node of the "Start Navigation" button.

[0054] The process of using a second UI element as an instance node of the target baseline node and establishing a mapping relationship with it involves binding the instance node of the second UI element to the target baseline node after confirming that the functions of the second UI element and the target baseline node are consistent, forming a "baseline node-instance node" association structure to realize the cross-device feature migration process. A bidirectional association between instance nodes and target baseline nodes can be established in the graph through key-value pairs, node pointers, etc., and the mapping relationship can be stored in the database. For example, the instance node of the "Navigation Start" button and the target baseline node of the "Start Navigation" button can be bound to each other through a functional semantic key to form a clear mapping relationship in the graph and record the functional consistency association between the two elements.

[0055] The process of determining the display position information of the second UI element on the second device based on the mapping relationship and the screen parameters of the second device is a process of calculating the precise position of the second UI element in its screen coordinate system based on the mapping relationship between the instance node and the target reference node and the screen hardware parameters of the second device. It can be done by first confirming the functional attributes of the second UI element through the mapping relationship, then calling the screen parameters of the second device (resolution, size, coordinate system rules), and combining the relative position data of the element in the interface to calculate the absolute display position. For example, after confirming that the "Navigation Start" button and the "Start Navigation" button have the same function through the mapping relationship, the resolution parameter of the second device (2560×1440) can be obtained, and combined with the relative position of the button in the interface of the second device, its coordinates in the screen coordinate system (x:1200, y:900) can be calculated, which is the display position information.

[0056] Through the implementation of the above embodiments, the test instructions are semantically parsed, and the multi-dimensional features of UI elements in the second device are matched with the reference nodes with the same first functional semantics in the feature map. This enables accurate identification and mapping of UI elements with the same function but different forms of expression between different vehicle devices. It can automatically establish the correspondence between UI elements in the case of vehicle model change, screen specification change or system upgrade, avoid manual repositioning of elements, and thus improve the adaptation efficiency and success rate of in-vehicle cross-device automated testing.

[0057] In some embodiments, step 103 may include: determining the operation type and initial operation parameters to be executed based on the test instruction; converting the initial operation parameters into instantiated operation parameters adapted to the screen coordinate system of the second device based on the display position information; and generating a target operation instruction based on the operation type and the instantiated operation parameters.

[0058] In some examples, the operation type is the specific interactive action category corresponding to the test instruction, which is the execution logic of the target operation instruction and clarifies the operation form on the second UI element. Semantic parsing can be performed on abstract test instructions to extract action keywords; for example, the "click" operation type extracted from the "trigger navigation function" test instruction, the "text input" operation type extracted from the "enter destination address" instruction, and the "swipe" operation type extracted from the "refresh song list" instruction. Initial operation parameters are abstract parameters corresponding to the operation type that are not bound to a specific device. They only contain the function's direction information and do not involve device-specific hardware or coordinate data. When parsing the test instruction, the function-related parameters associated with the operation type can be extracted simultaneously; for example, the initial operation parameters for the "trigger navigation function" instruction can be "target element: UI element corresponding to the navigation start function," and the initial operation parameters for the "enter destination address" instruction can be "target element: destination input box; input content: XX street XX number."

[0059] The screen coordinate system of the second device is the coordinate system used by the second device screen to locate the position of UI elements. It clarifies the coordinate origin, coordinate axis direction, and the mapping relationship between coordinate values ​​and actual screen pixels, and serves as the basis for instantiating operation parameters. The screen coordinate system rules and parameters preset by the device system can be read through the communication interface with the second device (such as ADB, Android Debug Bridge). For example, it can be the screen coordinate system of the central control device of a certain brand's 2026 compact SUV, with the upper left corner of the screen as the origin, the horizontal rightward direction as the positive X-axis, and the vertical downward direction as the positive Y-axis. The coordinate values ​​are in pixels and are adapted to a screen resolution of 2560×1440. Instantiated operation parameters are specific executable parameters that are adapted to the screen coordinate system and hardware characteristics of the second device after the initial operation parameters are transformed. They include concrete information such as precise position and value. The initial operation parameters can be concretely transformed based on the display position information and the screen coordinate system rules of the second device. For example, the initial operation parameters of "target element: navigation start button" can be transformed into coordinate parameters (x:1200, y:900) in the screen coordinate system of the second device. These parameters are the instantiated operation parameters.

[0060] The process of transforming initial operation parameters into instantiated operation parameters adapted to the screen coordinate system of the second device, based on the display position information, is centered on the display position information of the second UI element determined in step 102. Combined with the screen coordinate system rules of the second device, the abstract initial operation parameters are transformed into specific, executable device-specific parameters. The function pointer (such as the target element) in the initial operation parameters can be associated with the display position information first, and then the position information can be transformed into the corresponding coordinate parameters according to the coordinate system mapping rules of the second device to form instantiated operation parameters. For example, the initial operation parameters of "target element: navigation start button" can be associated with the display position information (x:1200, y:900) and transformed into instantiated operation parameters of "coordinates (1200, 900)" according to the screen coordinate system rules of the second device to ensure that the parameters are adapted to the screen positioning logic of the device.

[0061] The process of generating a target operation instruction based on the operation type and instantiated operation parameters is a flow that integrates the action of the operation type with the specific information of the instantiated operation parameters, and generates an instruction that can be directly executed on the second device according to the automation execution framework format compatible with the second device. It can automatically concatenate and format the instruction based on the operation type, instantiated operation parameters and preset execution framework syntax. For example, it can generate the target operation instruction "UIAutomator click --x1200--y900" by integrating the operation type of "click" and the instantiated operation parameter of "coordinates (1200, 900)" according to the syntax requirements of the UI Automator automation execution framework.

[0062] Through the implementation of the above embodiments, after determining the display position information of UI elements on the second device, the abstract test instructions are then transformed into target operation instructions that adapt to the screen coordinate system of the second device. This allows the test instructions to maintain high abstraction and reusability while flexibly adapting to the screen parameters of different vehicle devices. This avoids writing test scripts separately for different vehicle models or different screen specifications, thereby reducing the script maintenance cost of vehicle automation testing in a multi-device environment and improving test execution efficiency.

[0063] In some embodiments, the aforementioned verification of the response interface of the second device to determine the test result for the second device may include: performing a structural comparison between a screenshot of the second interface of the response interface and a benchmark reference image corresponding to the test instruction to obtain a first verification result; searching for the expected third UI element in the screenshot of the second interface based on a multi-dimensional UI feature map to obtain a second verification result; and determining whether the test instruction was successfully executed based on the first verification result and the second verification result, and generating a test result.

[0064] In some examples, the second interface screenshot is an image file corresponding to the response interface displayed on the screen after the second device executes the target operation command. It completely records the interface state after the operation and serves as a data source for verifying the operation effect. A vehicle screen capture tool can be used to capture the current screen display content of the second device in real time after the target operation command is executed, generating a high-definition image. For example, for the central control device of a 2026 compact SUV of a certain brand, which is equipped with an Android 13 version vehicle intelligent operating system and has a screen resolution of 2560×1440, after executing the target operation command "enter XX Road XX Number in the destination input box", the captured interface image includes the address display after the input is completed and the "start navigation" button is active.

[0065] The benchmark reference image is a standard response interface image of the first device corresponding to the test command, stored in the UI multi-dimensional feature map. It serves as the benchmark for measuring whether the response effect of the second device meets expectations, possessing functional semantic consistency and visual standardization. When constructing the UI multi-dimensional feature map, a screenshot of the standard response interface generated after the first device executes the same functional test command can be simultaneously collected and stored in the map as benchmark data. For example, after the first device (the central control device of a 2025 mid-size sedan of a certain brand) performs the operation of "entering an address into the destination input box", a 1920×1080 resolution standard interface image containing a standard address display and the "start navigation" button being normally activated can be captured. This image is the benchmark reference image.

[0066] The first verification result is the quantitative matching result obtained after structurally comparing the second interface screenshot with the baseline reference image. It is used to characterize the degree of consistency between the two in terms of visual structure and layout distribution. A structural similarity comparison algorithm, such as the Structural Similarity Index Measure (SSIM) algorithm, can be used to quantitatively calculate the pixel distribution, interface layout, and relative positions of elements between the second interface screenshot and the baseline reference image, and output a similarity value between 0 and 1. For example, if the second interface screenshot and the baseline reference image are compared using the SSIM algorithm and a similarity value of 0.92 is obtained, this value is the first verification result, reflecting that the two have a high degree of consistency in visual structure.

[0067] The expected third UI element is a specific UI element that should appear in the second device's response interface after the target operation is executed, based on the functional logic of the test command and the baseline information in the UI multi-dimensional feature map. Its functional semantics are directly related to the expected effect of the test command. Based on the functional logic of the test command and the standard UI elements that appear in the first device after executing the command in the UI multi-dimensional feature map, the corresponding element that should appear in the second device's response interface can be determined. For example, the expected UI elements such as "route planning list" and "navigation mode selection button" after executing the command "enter address in destination input box" are the third UI elements.

[0068] The second verification result is the quantitative matching result obtained after searching for the expected third UI element in the second interface screenshot. It is used to characterize whether the third UI element exists and the accuracy of the matching. The UI multi-dimensional feature map can be called to extract the baseline features of the third UI element, and feature matching retrieval can be performed in the second interface screenshot to output a matching confidence value between 0 and 1. For example, if we search for the third UI element "route planning list" in the second interface screenshot, the matching confidence value of 0.96 can be obtained through feature matching. This value is the second verification result, indicating that the element exists accurately in the response interface.

[0069] Two types of verification results can be preset with acceptance criteria. If both results meet the criteria, the test is deemed "passed"; if either result fails, the test is deemed "failed," and the corresponding test result is automatically generated. For example, the acceptance criteria can be preset as a similarity of ≥0.85 for the first verification result and a matching confidence of ≥0.9 for the second verification result. If both the first verification result (0.92) and the second verification result (0.96) meet the criteria, the test is deemed "passed"; if the first verification result (0.80) fails to meet the criteria, the test is deemed "failed."

[0070] Through the implementation of the above embodiments, after executing the target operation command, the response interface of the vehicle device is verified by both interface structure comparison and element search based on UI multi-dimensional feature map. This dual verification mechanism can effectively reduce the risk of misjudgment caused by misidentification or occasional anomalies in the complex interface environment of the vehicle, improve the accuracy and reliability of automated test results, and meet the requirements of the vehicle system for high stability testing.

[0071] In some embodiments, the aforementioned determination of whether the test instruction was executed successfully based on the first verification result and the second verification result may include: if the structural similarity indicated by the first verification result is greater than a first preset threshold and the matching confidence of the second verification result is greater than a second preset threshold, then the test instruction is determined to be executed successfully; if the structural similarity indicated by the first verification result is less than or equal to the first preset threshold, or the matching confidence of the second verification result is less than or equal to the second preset threshold, then the test instruction is determined to be executed unsuccessfully.

[0072] In some examples, structural similarity is a quantitative indicator of the degree of fit between the second interface screenshot and the baseline reference image in terms of visual structure, layout distribution, relative position of elements, and pixel distribution patterns. It is the representation data of the first verification result, with a value ranging from 0 to 1. The closer the value is to 1, the higher the structural consistency. The quantitative result can be calculated by performing a structural similarity algorithm to compare the second interface screenshot and the baseline reference image pixel by region and matching the layout features. For example, after performing an address input operation on the central control device of a 2026 compact SUV of a certain brand, the structural similarity of its second interface screenshot and the baseline reference image of the first device is calculated by a structural similarity algorithm and a structural similarity of 0.92 is obtained, indicating that the interface structures of the two are highly consistent. The first preset threshold is a pre-set critical value used to determine whether the structural similarity is qualified. This threshold has been calibrated by a large number of in-vehicle cross-device test samples and can balance the strictness of structural matching with the tolerance of cross-device visual differences. During the initialization stage, a fixed critical value can be determined and stored through statistical analysis based on the structural difference data of in-vehicle UI interfaces of different models and different screen parameters, and can be fine-tuned as needed. For example, the first preset threshold can be 0.85. When the structural similarity is greater than 0.85, the structural matching between the second interface screenshot and the benchmark reference image is deemed qualified.

[0073] Match confidence is a quantitative indicator of the reliability of the match between the target element searched in the second screenshot and the baseline features of the third UI element in the multi-dimensional feature map of the UI. It is the representation data of the second verification result, with a value ranging from 0 to 1. The closer the value is to 1, the more accurate the element match. Feature similarity matching algorithms (such as cosine similarity algorithm) can be used to compare and calculate the features of the candidate elements extracted from the second screenshot with the baseline features of the third UI element, and output the quantitative result. For example, if we search for the third UI element "route planning list" in the second screenshot, the feature matching algorithm calculates a match confidence of 0.96, indicating that the candidate element and the baseline features are highly consistent and the matching result is reliable. The second preset threshold is a pre-set critical value used to determine whether the matching confidence level is qualified. This threshold is optimized for the feature stability and cross-device differences of in-vehicle UI elements to ensure the accuracy of functional element recognition. It can be a fixed critical value calibrated and stored through iterative testing based on massive feature matching sample data of in-vehicle UI elements when constructing a multi-dimensional feature map of UI. For example, the second preset threshold can be preset to 0.9. When the matching confidence level is greater than 0.9, it is determined that the third UI element is successfully and reliably matched in the second interface screenshot.

[0074] When both conditions of "qualified interface structure matching" and "reliable functional element matching" are met, the target operation instruction is deemed to have achieved the expected functional effect. This logic takes into account both visual structural consistency and functional semantic integrity to ensure the accuracy of test results. If either condition of "qualified interface structure matching" or "reliable functional element matching" is not met, the target operation instruction is deemed to have failed to achieve the expected functional effect. This logic can promptly identify situations where the interface structure is abnormal or functional elements are missing.

[0075] By implementing the above embodiments, thresholds are set for interface structure similarity and UI element matching confidence, and the success of test command execution is determined based on the comprehensive result of the two. This enables vehicle automation testing to have clear and quantifiable judgment criteria, avoids the uncertainty caused by a single verification condition, and can still output reliable test results stably even when there are local differences or dynamic changes in the vehicle system interface. This further improves the practicality and engineering reliability of cross-device automation testing in vehicle scenarios.

[0076] Furthermore, as an implementation of the foregoing method embodiments, this application also provides a cross-device automated testing apparatus for implementing the foregoing method embodiments. This apparatus embodiment corresponds to the foregoing method embodiments. For ease of reading, this cross-device automated testing apparatus embodiment will not repeat the details of the foregoing method embodiments one by one, but it should be understood that the apparatus in this application embodiment can correspondingly implement all the contents of the foregoing method embodiments. For example... Figure 2As shown, the cross-device automated testing device 20 includes: a graph construction unit 201, a position matching unit 202, an instruction instantiation unit 203, and a test verification unit 204. The graph construction unit 201 is used to construct a multi-dimensional UI feature graph based on the first UI element information of the first device. The position matching unit 202, if a test instruction for the second device is received, matches the second UI element information of the second device based on the multi-dimensional UI feature graph to determine the display position information of the second UI element corresponding to the test instruction on the second device. The instruction instantiation unit 203 is used to instantiate the test instruction into a target operation instruction based on the display position information. The test verification unit 204 is used to execute the target operation instruction through the second device and verify the response interface of the second device to determine the test result for the second device.

[0077] In some embodiments, the graph construction unit 201 is further configured to extract a first multi-dimensional feature of a first UI element of the first device based on a first interface screenshot of the first device, wherein the first multi-dimensional feature includes geometric features, visual features and semantic features; generate a first feature identifier corresponding to the first UI element based on the first multi-dimensional feature; associate the first feature identifier with the first functional semantic of the corresponding first UI element, and store it as a reference node in the UI multi-dimensional feature graph.

[0078] In some embodiments, the map construction unit 201 is further configured to perform target detection on the first interface screenshot to locate the region image of the first UI element; determine geometric features based on the contour information of the region image and the relative positional relationship between the contour information and the preset interface anchor point; process the region image through a preset visual feature extraction model to obtain visual features representing color and texture distribution information; extract the text information contained in the region image through an optical character recognition tool, and semantically embed the text information based on a natural language processing model to obtain semantic features.

[0079] In some embodiments, the position matching unit 202 is further configured to parse and process the test instructions to obtain the second functional semantics of the second UI element; extract the second multi-dimensional features of the second UI element and generate a second feature identifier corresponding to the second multi-dimensional features; calculate the similarity between the second multi-dimensional features and the first feature identifier of the target reference node with the second functional semantics in the UI multi-dimensional feature map to obtain the target similarity; if the target similarity is greater than a preset similarity threshold, then the second UI element is used as an instance node of the target reference node and a mapping relationship is established with the target reference node; based on the mapping relationship and the screen parameters of the second device, the display position information of the second UI element on the second device is determined.

[0080] In some embodiments, the instruction instantiation unit 203 is further configured to determine the operation type and initial operation parameters to be executed based on the test instruction; convert the initial operation parameters into instantiated operation parameters adapted to the screen coordinate system of the second device based on the display position information; and generate a target operation instruction based on the operation type and instantiated operation parameters.

[0081] In some embodiments, the test verification unit 204 is further configured to perform a structural comparison between the second interface screenshot of the response interface and the benchmark reference image corresponding to the test instruction to obtain a first verification result; search for the expected third UI element in the second interface screenshot based on the UI multi-dimensional feature map to obtain a second verification result; and determine whether the test instruction was successfully executed based on the first verification result and the second verification result, and generate a test result.

[0082] In some embodiments, the test verification unit 204 is further configured to determine that the test instruction was executed successfully if the structural similarity indicated by the first verification result is greater than the first preset threshold and the matching confidence of the second verification result is greater than the second preset threshold; and to determine that the test instruction was executed unsuccessfully if the structural similarity indicated by the first verification result is less than or equal to the first preset threshold, or the matching confidence of the second verification result is less than or equal to the second preset threshold.

[0083] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs that, when executed by a processor, will cause the processor to perform any step of the cross-device automated testing method provided in this application.

[0084] In some embodiments, the computer-readable storage medium may be a random access memory (RAM), a read-only memory (ROM), flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or it may be a variety of devices that include one or any combination of the above-mentioned memories.

[0085] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0086] In some embodiments, computer-executable instructions may, but do not necessarily, correspond to files in a file system, and may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0087] In some embodiments, computer-executable instructions may be deployed to execute on an electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0088] like Figure 3 As shown, this application also provides an electronic device 30, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements any step of the above-described cross-device automated testing method.

[0089] This application also provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform any step of the cross-device automated testing method described above.

[0090] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A cross-device automated testing method, characterized in that, include: Based on the first UI element information of the first device, construct a multi-dimensional UI feature map; If a test instruction is received for the second device, the second UI element information of the second device is matched based on the UI multi-dimensional feature map to determine the display position information of the second UI element corresponding to the test instruction on the second device. Based on the display location information, the test instruction is instantiated into a target operation instruction; The target operation command is executed through the second device, and the response interface of the second device is verified to determine the test results for the second device.

2. The cross-device automated testing method according to claim 1, characterized in that, The construction of a multi-dimensional UI feature map based on the first UI element information of the first device includes: Based on the first interface screenshot of the first device, the first multi-dimensional features of the first UI element of the first device are extracted, wherein the first multi-dimensional features include geometric features, visual features and semantic features. Based on the first multi-dimensional feature, a first feature identifier corresponding to the first UI element is generated; The first feature identifier is associated with the first functional semantic of the corresponding first UI element, and stored as a reference node in the UI multi-dimensional feature map.

3. The cross-device automated testing method according to claim 2, characterized in that, The step of extracting the first multi-dimensional features of the first UI element of the first device based on the first interface screenshot of the first device includes: Perform target detection on the first interface screenshot to locate the region image of the first UI element; The geometric features are determined based on the contour information of the region image and the relative positional relationship between the contour information and the preset interface anchor points; The region image is processed by a preset visual feature extraction model to obtain the visual features that characterize color and texture distribution information; The text information contained in the region image is extracted using an optical character recognition tool, and the text information is semantically embedded based on a natural language processing model to obtain the semantic features.

4. The cross-device automated testing method according to claim 1, characterized in that, The step of matching the second UI element information of the second device based on the UI multi-dimensional feature map to determine the display position information of the second UI element corresponding to the test command on the second device includes: The test instructions are parsed and processed to obtain the second functional semantics of the second UI element; Extract the second multi-dimensional feature of the second UI element and generate a second feature identifier corresponding to the second multi-dimensional feature; The similarity between the second multi-dimensional feature and the first feature identifier of the target baseline node with the second functional semantic in the UI multi-dimensional feature map is calculated to obtain the target similarity. If the target similarity is greater than a preset similarity threshold, then the second UI element is used as an instance node of the target reference node, and a mapping relationship is established between it and the target reference node; Based on the mapping relationship and the screen parameters of the second device, the display position information of the second UI element on the second device is determined.

5. The cross-device automated testing method according to claim 1, characterized in that, The step of instantiating the test instruction into a target operation instruction based on the display position information includes: The type of operation to be executed and the initial operation parameters are determined based on the test instructions; Based on the display position information, the initial operation parameters are converted into instantiated operation parameters adapted to the screen coordinate system of the second device; The target operation instruction is generated based on the operation type and the instantiated operation parameters.

6. The cross-device automated testing method according to claim 1, characterized in that, The verification of the response interface of the second device to determine the test results for the second device includes: The second interface screenshot of the response interface is structurally compared with the benchmark reference image corresponding to the test command to obtain the first verification result. Based on the UI multi-dimensional feature map, the expected third UI element is searched in the second interface screenshot to obtain the second verification result; Based on the first verification result and the second verification result, it is determined whether the test instruction was executed successfully, and the test result is generated.

7. The cross-device automated testing method according to claim 6, characterized in that, The step of determining whether the test instruction was successfully executed based on the first verification result and the second verification result includes: If the structural similarity indicated by the first verification result is greater than the first preset threshold, and the matching confidence of the second verification result is greater than the second preset threshold, then the test instruction is determined to have been executed successfully. If the structural similarity indicated by the first verification result is less than or equal to the first preset threshold, or the matching confidence of the second verification result is less than or equal to the second preset threshold, then the test instruction is determined to have failed to execute.

8. A cross-device automated testing device, characterized in that, include: The graph construction unit is used to construct a multi-dimensional feature graph of the UI based on the first UI element information of the first device; The location matching unit is used to, if a test instruction for the second device is received, match the second UI element information of the second device based on the UI multi-dimensional feature map, and determine the display position information of the second UI element corresponding to the test instruction on the second device. The instruction instantiation unit is used to instantiate the test instruction into a target operation instruction based on the display position information; The test and verification unit is used to execute the target operation command through the second device and verify the response interface of the second device to determine the test results for the second device.

9. An electronic device, comprising: A memory and a processor, characterized in that the processor, when executing a computer program stored in the memory, implements the steps of the cross-device automated testing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by a processor, the steps of the cross-device automated testing method as described in any one of claims 1 to 7 are implemented.

Images