Control tree construction method and related device

Abstract

A control tree construction method and related device, the control tree construction method comprises: acquiring a user interface image of an IVI system; acquiring spatial position features, category attribute features, text semantic features and microscopic feedback state features of a plurality of discrete controls from the user interface image of the IVI system; and constructing a control tree comprising hierarchical nesting relationships between the plurality of controls based on the spatial position features, category attribute features, text semantic features and microscopic feedback state features of the plurality of discrete controls. The technical solution of the embodiment of the present application can implement construction of a control tree for an IVI system that does not support or provide a control tree service.

Description

Technical Field

[0001] This invention relates to the field of automated testing, and in particular to a control tree construction method and related equipment. Background Technology

[0002] In-Vehicle Infotainment (IVI) systems, commonly known as car infotainment systems, are integrated in-vehicle information processing systems developed based on automotive-grade processors and connected to the vehicle via the vehicle bus. They are now a core component of smart cockpits, undertaking human-vehicle interaction and vehicle networking service functions.

[0003] The user interface (UI) of an IVI system, as the core interface for human-vehicle interaction, determines the user experience and driving safety. With the intelligent development of IVI systems, the demand for automated testing of their user interface (UI) is increasing. The core of automated testing lies in accurately identifying and locating controls (UI elements), which is usually achieved by obtaining the control tree of the IVI system.

[0004] Therefore, how to construct a control tree in an IVI system that does not support or provide control tree services has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The problem solved by the embodiments of the present invention is to provide a control tree construction method and related equipment, which can realize the construction of control trees for IVI systems that do not support or provide control tree services.

[0006] To address the above problems, embodiments of the present invention provide a control tree construction method, including: Obtain the user interface image of the IVI system; The spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls are obtained from the user interface image of the IVI system. Based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, a control tree is constructed that includes the hierarchical nesting relationship between the multiple controls.

[0007] Optionally, the step of obtaining spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system includes: A pre-trained UI shape detection model is used to perform UI shape detection and category attribute recognition on the user interface image of the IVI system, and information on the control prediction boxes and category attributes present in the user interface image of the IVI system is obtained. The pre-trained text semantic recognition model is used to perform text semantic recognition processing on the user interface image of the IVI system to obtain information about text boxes containing text content in the user interface image of the IVI system. The micro-state recognition model obtained in the pre-trained process is used to perform micro-feedback state recognition processing on the user interface image of the IVI system to obtain information on the micro-feedback state of the controls present in the user interface image of the IVI system. Multimodal fusion processing is performed on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the acquired controls to obtain the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls.

[0008] Optionally, for the same user interface image, the multimodal fusion processing of the acquired spatial location features, category attribute features, text semantic features, and micro-feedback state features of the controls includes at least one of the following: If the center point of the text box output by the text semantic recognition model is located within the control prediction box output by the UI shape detection model, or if the preset intersection-union ratio of the center point of the text box output by the text semantic recognition model and the control prediction box output by the UI shape detection model is greater than the preset intersection-union ratio threshold, then the text content in the text box output by the text semantic recognition model is used as the text content attribute of the control in the control prediction box output by the UI shape detection model. If the text box output by the text semantic recognition model is within the predetermined extension tolerance of the control prediction box output by the UI shape detection model, and the main projection axis of the text box output by the text semantic recognition model is aligned with the control prediction box output by the UI shape detection model, then the text content in the text box output by the text semantic recognition model is used as the label attribute or descriptor attribute of the control in the control prediction box output by the UI shape detection model. If the UI shape detection model does not detect a control at a spatial location, and the text semantic recognition model detects a text box with the interactive word "OK" semantics at the control location at the spatial location with a first detection confidence score higher than the preset detection confidence score threshold, then a virtual text button is added to the spatial location as a virtual control at the spatial location. If the category attribute features of the control output by the UI shape detection model are that the detection confidence of both the switch and the button is lower than the preset second confidence threshold, and the micro state recognition model identifies that the control has a binary micro feedback state feature of "selected / unselected", or the semantic text recognition model identifies that the control prediction box has the word "on / off" on one side, then the control is determined to be a switch control.

[0009] Optionally, the construction of a control tree, which includes hierarchical nesting relationships between multiple controls, based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, includes: Create a root node, which is located at a predetermined position in the full-screen view of the user interface; Obtain a control set of the plurality of discrete controls, the control set including information on the coordinates of the control prediction boxes of the plurality of discrete controls; Based on the coordinate information of the multiple discrete control prediction boxes in the control set, the spatial envelope relationship between the multiple discrete controls is obtained; Based on the spatial envelope relationship between the multiple discrete controls, a control tree is constructed with the root node as the unique top-level node.

[0010] Optionally, obtaining the spatial envelope relationship between the plurality of discrete controls based on the coordinate information of the plurality of discrete control prediction boxes in the control set includes: For any two controls in the control set, calculate the area of ​​the two control prediction boxes and the area of ​​the overlapping area of ​​the two control prediction boxes based on the coordinates of the two control prediction boxes respectively. The control with the smaller predicted box area of ​​the two controls is designated as the first control, and the control with the larger predicted box area of ​​the two controls is designated as the second control. Calculate the area ratio between the predicted area of ​​the first control and the overlapping area between the predicted areas of the first control and the second control, and use this ratio as the spatial envelope ratio of the first control and the second control.

[0011] Optionally, constructing a control tree with the root node as the unique top-level node based on the spatial envelope relationship between the plurality of discrete controls includes: If the spatial envelope ratio is greater than a preset envelope ratio threshold, then the second control is used as the envelope parent node of the first control; the envelope parent node of each control in the control set is obtained, and corresponding envelope parent node sets are formed respectively; If there is only one envelope parent node in the set of envelope parent nodes, then the unique envelope parent node in the set of envelope parent nodes shall be used as the parent node of the corresponding control. If there are multiple envelope parent nodes in the set of envelope parent nodes, the envelope parent node with the smallest area of ​​the control prediction box is selected from the set of envelope parent nodes and used as the parent node of the corresponding control.

[0012] Optionally, if the two controls partially overlap and their spatial envelope ratio is less than or equal to the envelope ratio threshold, the step of constructing a control tree with the root node as the unique top-level node based on the spatial envelope relationship between the plurality of discrete controls includes: The stacking relationship between the two controls is determined based on their category priority in a preset category priority order.

[0013] Optionally, after determining the parent node of each control in the control set, constructing a control tree with the root node as the unique top-level node further includes: Get a sequence of three or more sibling nodes with the same parent node that share the same category attribute; Create a minimum bounding box that includes a sequence of three or more sibling nodes with the same parent node and the same class attribute, as a virtual envelope box; Generate a virtual control object that corresponds to the virtual envelope and is not natively visible in the IVI system, and assign the virtual control object the corresponding category attribute; The virtual control object is used as the parent node of the sibling node sequence with the same category attribute in the virtual envelope, and the virtual control object is used as the child node of the parent node of the sibling node sequence with the same category attribute in the virtual envelope.

[0014] Optionally, obtaining a sequence of three or more sibling nodes with the same category attribute under the same parent node includes: For any two adjacent and non-intersecting sibling child nodes under the same parent node, if the deviation between the center points of the control prediction boxes of the two adjacent and non-intersecting sibling child nodes in the horizontal or vertical direction is less than a preset deviation threshold, then it is determined that the two adjacent and non-intersecting sibling child nodes have horizontal row affinity or vertical column affinity. Obtain an affinity connectivity graph consisting of three or more sibling child nodes with horizontal or vertical affinity under the same parent node, and calculate the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph. If the variance of the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph is less than or equal to a preset variance threshold, then the sibling child nodes in the affinity connectivity graph are determined to be a sequence of sibling nodes with the same category attribute.

[0015] Optionally, if the variance of the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph is less than or equal to a preset variance threshold, obtaining a sequence of three or more sibling nodes with the same parent node and the same class attribute further includes: If the sibling child nodes in the affinity connectivity graph have morphological and semantic consistency, then the sibling child nodes in the affinity connectivity graph are determined to be sibling nodes with the same category attribute.

[0016] This invention also provides a computer device, including at least one memory and at least one processor, wherein the memory stores one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the control tree construction method as described in any of the preceding embodiments.

[0017] Accordingly, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, are used to implement the control tree construction method as described in any of the preceding claims.

[0018] Accordingly, embodiments of the present invention also provide a storage medium storing one or more computer instructions, the one or more computer instructions being used to implement the control tree construction method as described in any of the preceding claims.

[0019] Compared with the prior art, the technical solution of the embodiments of the present invention has the following advantages: The control tree construction method provided in this invention first acquires the user interface image of an IVI system, and then acquires the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system. Based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, a control tree including the hierarchical nesting relationship between multiple controls is constructed. This method can realize the construction of control trees in IVI systems that do not support or provide control tree services, thus meeting the user's control tree construction needs. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an embodiment of the control tree construction method provided by the technical solution of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the present invention for obtaining spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of an IVI system. Figure 3 This is a flowchart illustrating an embodiment of the training process of the UI shape detection model provided by the technical solution of the present invention; Figure 4 This is a schematic diagram of an in-vehicle user interface; Figure 5This is an example diagram illustrating the calculation of horizontal row affinity or vertical column affinity between any two non-overlapping sibling child nodes under the same parent node, provided by the technical solution of this invention. Figure 6 This is an example diagram illustrating the method for determining whether three or more sibling child nodes in an affinity connectivity graph are a sequence of sibling nodes with the same category attribute, as provided by the technical solution of this invention. Figure 7 This is an example diagram of topological reconstruction of the subtree structure to which the sibling node sequences with the same category attribute belong, provided by the technical solution of this invention. Figure 8 This is a schematic diagram of the structure of an embodiment of the computer device provided by the technical solution of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] With the rapid development of in-vehicle intelligence, the functional iteration speed of IVI systems continues to accelerate. Automated testing has become the core means to ensure the quality of IVI systems. However, traditional automated testing technology has always relied heavily on the auxiliary functions provided by the operating system itself, such as accessibility services or control tree services.

[0023] Whether it's an IVI system developed based on Android or other mainstream automotive operating systems, the accessibility service built into the operating system itself provides native interfaces for obtaining control attributes and triggering control operations. The control tree service can directly output the hierarchical structure of the current interface, control identifiers (IDs), text content, etc. Testing tools only need to call the native interfaces exposed by the system to complete core testing steps such as control location, operation simulation, and result verification. There is no need to develop an additional underlying recognition module. The development threshold is low and the adaptation cost is small, so it has become a common choice for automated testing frameworks of early traditional IVI systems.

[0024] However, many Linux kernel-based IVI systems do not have a built-in control tree service, or have closed the relevant interface due to security factors, making it impossible to obtain the control tree of the IVI system and thus impossible to achieve automated testing of the IVI system based on the control tree.

[0025] In view of this, the control tree construction method provided in this embodiment of the invention first obtains the user interface image of the IVI system, and then obtains the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system. Based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, a control tree including the hierarchical nesting relationship between multiple controls is constructed. This method can realize the construction of control trees in IVI systems that do not support or provide control tree services, thus meeting the user's control tree construction needs.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This diagram illustrates a flowchart of an embodiment of the control tree construction method provided by the present invention. (Refer to the reference.) Figure 1 A method for constructing a control tree may specifically include the following steps: Step S110: Obtain the user interface image of the IVI system; Step S120: Obtain the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system; Step S130: Based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls, construct a control tree that includes the hierarchical nesting relationship between multiple controls.

[0028] Please continue to refer to the reference. Figure 1 Execute step S110 to obtain the user interface image of the IVI system.

[0029] Obtaining the user interface (UI) image of the IVI system provides a foundation for subsequently extracting the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the IVI system's user interface image.

[0030] The user interface of the IVI system is the medium for interaction and information exchange between the IVI system and the user. Its core function is to support various IVI applications (such as navigation, music, vehicle settings, telephone, etc.), allowing users to intuitively select functions, clearly display vehicle status, multimedia information, navigation guidance, and other content. It integrates multiple interaction methods such as touch, physical buttons, and voice to assist the driving process, taking into account both operational efficiency and driving safety.

[0031] Typically, the user interface of an IVI system includes controls, which allow users to manipulate and edit data within the IVI system, as well as execute IVI system commands. These controls can include buttons, sliders, switches, icons, list items, and so on.

[0032] In the optional implementation, in the step of obtaining the user interface image of the IVI system, the full-screen image of the IVI system is obtained in real time through the system screenshot interface of the IVI system (such as High Definition Multimedia Interface (HDMI)) or video capture card, and used as the user interface image of the IVI system.

[0033] It should be noted that in the step of obtaining the user interface image of the IVI system, a full-screen image of all user interfaces in the IVI system is obtained. This provides the foundation for subsequently identifying all controls from the user interface image of the IVI system and constructing a control tree.

[0034] Please continue to refer to the reference. Figure 1 Step S120 is executed to obtain the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system.

[0035] The spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls are obtained from the user interface image of the IVI system. This provides a foundation for constructing a control tree that includes the hierarchical nesting relationship between multiple controls based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls.

[0036] For the optional implementation methods, please refer to the reference. Figure 2 The steps of obtaining spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of an IVI system may include: In step S210, the UI shape detection model obtained in the pre-trained process is used to perform UI shape detection and category attribute recognition on the user interface image of the IVI system, so as to obtain the spatial position features and category attribute features of the controls in the user interface image of the IVI system.

[0037] In an optional implementation, embodiments of this application can train a UI shape detection model capable of locating and classifying controls in user interface images. For example, the trained UI shape detection model can predict user interface images based on the spatial position and category attributes of controls. Therefore, for user interface images of an IVI system, embodiments of this application can call the UI shape detection model to determine the spatial position and category attribute features of potential controls in each user interface image. For instance, embodiments of this application can input user interface images of an IVI system into a pre-trained UI shape detection model and obtain the spatial position and category attribute features of potential controls in multiple user interfaces output by the UI shape detection model.

[0038] In an optional implementation, the training approach for the UI shape detection model can be as follows: using the prediction results of the spatial position features and category attribute features of the controls in the training samples as training targets, the UI shape detection model is trained using the training samples; wherein, the training samples include user interface samples, the control spatial position labels are the actual spatial positions of the controls in the user interface, and the control category attribute labels are the actual category attributes of the controls in the user interface.

[0039] In the optional implementation, Figure 3 An exemplary schematic diagram illustrates the process of training a UI morphology detection model provided in an embodiment of this application, such as... Figure 3 As shown, the process of training a UI shape detection model may include the following steps.

[0040] In step S310, a dataset is obtained, which includes multiple training samples, including user interface image samples.

[0041] In an optional implementation, embodiments of this application may collect user interface images of IVI systems, such as user interface images of various types of operating systems, and use them as training samples to form a dataset.

[0042] In step S320, the training samples in the dataset are labeled with control spatial location labels and control category attribute labels respectively. The control spatial location labels are the actual spatial location coordinates of the controls in the training samples, and the control category attribute labels are the actual category attributes of the controls in the training samples.

[0043] In an optional implementation, this application embodiment can annotate the label information of each training sample. Since the UI shape detection model is an object detection and classification task model, the label information for the training samples can be control spatial location labels and control category attribute labels to indicate the true position coordinates and true category attribute information of the controls in the training samples. For example, the control spatial location label of the training sample is the true bounding box (true physical outer bounding box) of the control, and the control category attribute label is the true category attribute of the control in the training sample.

[0044] In a further optional implementation, during the step of labeling the spatial location of controls in the training samples, different methods are used to label the spatial location of controls based on the shape differences of the control outlines in the training samples. For example, for controls with rectangular outlines, the minimum bounding rectangle bounding box (the ground truth box) of the control is used as the spatial location label of the training sample; for controls with non-rectangular outlines (such as circles), both the minimum bounding rectangle bounding box and polygon mask labeling are performed, and then converted into compact physical bounding boxes in the subsequent preprocessing stage to improve the fit of the regression box.

[0045] In optional implementations, embodiments of this application may use the Pascal Visual Object Classes (PASCAL VOC) format to annotate the spatial location labels and category attribute labels of the training samples in the dataset. Of course, in the step of annotating the spatial location labels and category attribute labels of the training samples, other suitable annotation formats may also be used, such as the Common Objects in Context (COCO) format, etc., and are not limited here.

[0046] In step S330, the predicted results of the control spatial position and control category attributes of the training samples are used to train the machine learning model with initial weights, with the control spatial position label and control category attribute label of the training samples as the training targets, to obtain the UI shape detection model.

[0047] By predefining control spatial location labels and control category attribute labels for training samples, representing the true spatial location coordinates and true category attributes of controls in each training sample, embodiments of this application can use training samples to train a machine learning model. The training objective is to make the prediction results of the machine learning model on the control spatial location and control category attributes of the training samples as close as possible to the control spatial location coordinate labels and control category attribute labels of the training samples, respectively. Thus, through continuous iterative training, a UI shape detection model can be obtained.

[0048] In an optional implementation, this embodiment of the application can first divide the dataset into a training set, a test set, and a validation set. Then, it executes the step of training a machine learning model with initial weights, using the prediction results of the control spatial position and control category attributes of the training samples as training targets, respectively, to obtain the UI shape detection model. The training set typically accounts for 70% to 80% and is the core data for the model to learn features and fit parameters, used to complete the main training process. The validation set typically accounts for 10% to 15%, does not participate in gradient updates, and is used to evaluate the model's current accuracy in real time during training, assist in parameter tuning, select the optimal checkpoint, and prevent overfitting. The test set typically accounts for 10% to 15%, is completely independent of the training and parameter tuning process, and is only used after all model training is completed, to objectively evaluate the model's true generalization performance on new, unseen data.

[0049] For example, the steps of training a machine learning model with initial weights, using the predicted results of the control spatial position and control category attributes of the training samples as training targets, to obtain a UI shape detection model include: performing one iterative training on the machine learning model with initial weights each time using a preset number (e.g., batch size) of training samples from the training set; correspondingly, performing multiple iterative training sessions, continuously updating the weights of the machine learning model iteratively, until the loss value of the machine learning model converges on a preset validation set, thus obtaining the UI shape detection model. Here, the convergence of the loss value of the machine learning model on the preset validation set means that the loss value of the machine learning model on the preset validation set reaches a minimum value.

[0050] Specifically, each iteration of training includes: using the machine learning model obtained from the previous iteration to predict a preset number of training samples and obtain a preset number of prediction results; calculating the loss value using a preset loss function based on the difference between the preset number of prediction results and the true label; performing backpropagation of the network based on the calculated loss value to obtain the gradient value; and adjusting the weights of the machine learning model based on the gradient value obtained from backpropagation.

[0051] In the optional implementation, the step of calculating the loss value using a preset loss function based on the difference between a preset number of predicted results and the true labels includes a classification loss term, a confidence loss term, and a bounding box regression loss term. Specifically, the classification loss term measures the difference between the predicted result of the control category attribute of the training sample and the control category attribute label; the confidence loss term measures the accuracy of the prediction of whether the predicted box of the training sample contains the control; and the bounding box regression loss term measures the difference in position, size, and shape between the predicted result of the control spatial location of the training sample (the predicted box) and the control spatial location label (the ground truth box).

[0052] Specifically, the loss function can be expressed as: (1) in, Represents the loss function. The weighting coefficients of the classification loss term. Represents the classification loss term. This represents the weighting coefficient of the confidence loss term. This represents the confidence level loss term. This represents the weighting coefficients of the bounding box regression loss term. This represents the bounding box regression loss term.

[0053] It is understandable that the classification loss term, confidence loss term, and bounding box regression loss term can be calculated by those skilled in the art by selecting the appropriate loss function according to actual needs. For example, the classification loss term and confidence loss term can respectively adopt the Cross Entropy Loss (CE) function, and the bounding box regression loss term can adopt the Complete Intersection over Union (CIoU) Loss function. The CIoU Loss adds an aspect ratio penalty term to the Distance Intersection over Union (DIoU) Loss, comprehensively considering overlap, center distance, and shape, enabling the UI shape detection model to converge quickly when predicting long, narrow controls (such as volume controls and progress bars), avoiding prediction boxes that are too wide or too narrow.

[0054] In the optional implementation, the training process of the UI shape detection model adopts the cosine annealing learning rate scheduling algorithm. The cosine annealing learning rate scheduling algorithm makes the learning rate smoothly decay from the initial value to the minimum value in each training epoch or a preset number of iterations. This helps the model escape local optima by periodically restarting at the end of training, giving the machine learning model the ability to escape the current local optimum and explore new solution spaces, which is beneficial to improving the performance of the machine learning model.

[0055] In an optional implementation, during the training of the UI shape detection model, the Adaptive Moment Estimation (Adam) optimizer is used to update the weight parameters of the machine learning model. The Adaptive Moment Estimation optimizer cleverly combines the inertia concept of momentum with the adaptive learning rate mechanism of Root Mean Square Propagation (RMSprop), which can calculate an adaptive learning rate for each parameter in the network, perfectly balancing convergence speed and training stability through the adaptive mechanism.

[0056] For more detailed information on the generation process of the UI shape detection model, please refer to the training steps of machine learning models in existing technologies, which will not be repeated here.

[0057] In an optional implementation, embodiments of this application may perform preprocessing operations on the training samples, such as data cleaning and data augmentation, before training the machine learning model using the training samples, so that the predicted training samples can meet the training requirements of the machine learning model.

[0058] Data cleaning operations include one or more of the following: deduplication, filtering of invalid / irrelevant data, noise cleanup and normalization, missing value handling, and removal of low-quality samples. Deduplication removes duplicate samples to prevent overfitting and ensure the reasonable distribution of training data. Filtering of invalid / irrelevant data removes content irrelevant to the training objective, including redundant noise and meaningless garbled text, to reduce interference from invalid information. Noise cleanup and normalization removes HTML tags, extra spaces, special characters, format errors, and other noise from the text, unifying the encoding format and completing basic text standardization. Missing value handling addresses missing values ​​in the data, allowing for options such as filling with default values, using the mean / median, or directly deleting missing samples to prevent incomplete data from affecting training performance. Removal of low-quality samples filters out excessively long / short samples and semantically invalid low-quality content, ensuring the overall effectiveness of the input data. This includes, for example, removing training samples with poor content quality or those with incorrect labels.

[0059] In optional implementations, the data augmentation operation includes at least one of mosaic data augmentation and mix-up augmentation. Mosaic data augmentation combines four different training samples (user interface images) into a new synthetic image through random scaling, cropping, and stitching. This creates a more complex and varied background environment for each control in the newly generated training sample, forcing the model to learn more essential features of the controls (such as their inherent texture) rather than memorizing their absolute position on the screen or a fixed background. This significantly improves the model's generalization ability and robustness. Furthermore, during random scaling, some large targets (controls) are shrunk. Mosaic data augmentation increases the proportion of small targets in the training data, effectively improving the model's detection performance for small targets. Mix-Up data augmentation generates new "virtual" training data by mixing two different training samples and their labels. This forces the model to make reasonable predictions on the "connections" between samples, preventing overfitting and learning a more linear and smoother classification boundary. Moreover, by creating a large number of previously unseen intermediate samples through linear combination, it effectively expands the distribution range of the training data. Additionally, as a powerful regularization method, Mix... Up data augmentation can significantly reduce a model's reliance on specific noisy or difficult samples, thus improving its performance on the test set.

[0060] In an optional implementation, when training the machine learning model using training samples, this embodiment can extract key features from the training samples. This allows the machine learning model to use these key features to predict the spatial location and category attributes of controls within the training samples. The predicted spatial location and category attributes of the training samples are then used as training targets, with the spatial location label and category attribute label of the training samples respectively as the training objectives. The weight parameters of the machine learning model are continuously adjusted to obtain the trained UI shape detection model. This allows the UI shape detection model to predict input user interface images based on the control position and category attributes. The extracted key features from the training samples include, but are not limited to, at least one of the following: visual features of the controls (such as the texture and shape of the controls), text information (such as text information extracted from the content of the training samples), etc.

[0061] In step S220, a pre-trained text semantic recognition model is used to perform text semantic recognition processing on multiple user interface images to obtain the text semantic features present in the user interface images of the IVI system.

[0062] In optional implementations, embodiments of this application can train a model capable of locating and semantically recognizing text content in user interface images. For example, the trained text semantic recognition model can locate and interpret text content within the control prediction box output by the UI shape detection model, or even within the entire range of the user interface image. Ultimately, it endows controls, which originally only possessed spatial location and category attributes, with readable semantic information that conforms to human cognitive logic. Therefore, for user interface images of an IVI system, embodiments of this application can call the text semantic recognition model separately to determine the text information present in each user interface image. For example, embodiments of this application can input the user interface images of the IVI system into the pre-trained text semantic recognition model and obtain the text semantic features of the IVI system's user interface images output by the text semantic recognition model.

[0063] In an optional implementation, embodiments of this application can construct a dedicated image-text pair dataset for optical character recognition (OCR) in vehicle scenarios, and then use the constructed image-text pair training set to perform machine learning training to obtain a text semantic recognition model.

[0064] For example, the steps of constructing the image-text pair dataset include: constructing an in-vehicle-specific lexicon for in-vehicle interaction scenarios, which includes multiple high-frequency words in various in-vehicle interaction scenarios (such as navigation, multimedia, vehicle settings, Bluetooth communication, etc.); performing font rendering processing on the words in the in-vehicle-specific lexicon using preset fonts (such as sans-serif fonts commonly used in in-vehicle interfaces, and illuminated fonts commonly used in in-vehicle dashboard and central control screen backlight scenarios, etc.), and matching them with typical text layout rules of in-vehicle interfaces to generate corresponding text layers; superimposing the generated text layers on randomly generated in-vehicle background images to form multiple initial image-text samples; performing interference transformation processing (such as Gaussian blur, motion blur, perspective transformation, and partial occlusion, etc.) on the multiple initial image-text samples to generate multiple corresponding image samples; the multiple image samples and the corresponding text samples constitute multiple image-text pair training samples, and the multiple image-text pair training samples form an image-text pair dataset.

[0065] The image-text pair training set constructed using the above method can avoid the shortcomings of poor adaptability between general public image-text pair datasets and in-vehicle scenarios. The image-text pair training samples in the training set can accurately match the display form and interference features of in-vehicle displays. Therefore, the semantic text recognition model trained using the generated image-text pair dataset can significantly improve the accuracy of text recognition in dynamic in-vehicle driving scenarios, and can effectively solve the problem of insufficient accuracy of traditional OCR models in recognizing illuminated fonts and dynamically blurred text in in-vehicle scenarios.

[0066] For further details on training the text semantic recognition model, please refer to the aforementioned content on training the UI shape detection model; these will not be repeated here.

[0067] In step S230, the pre-trained micro-state recognition model is used to perform state recognition processing on the user interface image of the IVI system to obtain the micro-feedback state features of the controls present in the user interface image of the IVI system.

[0068] The micro-feedback states of the controls in this application embodiment are different from the conventional display states when the controls are stationary, including button click highlighting, scale changes during slider dragging, transition frames of pop-up windows, and dynamic changes in text color / transparency after touch operation.

[0069] In an optional implementation, embodiments of this application can train a microstate recognition model capable of identifying the micro-feedback states of controls in a user interface image. For example, the trained microstate recognition model can predict user interface images based on the dimension of the micro-feedback states of controls. Therefore, for user interface images of an IVI system, embodiments of this application can call the microstate recognition model to determine the micro-feedback state features of potential controls present in each user interface image. For instance, embodiments of this application can input user interface images of an IVI system into a pre-trained microstate recognition model and obtain the micro-feedback state features of potential controls present in multiple user interfaces output by the microstate recognition model.

[0070] In an optional implementation, embodiments of this application can use a pre-generated state pair dataset for machine learning training to obtain a micro-state recognition model. The state pair dataset includes multiple state pair training samples, each including an image of the same control before and after clicking. Therefore, using the state pair dataset for machine learning training enables the trained micro-state recognition model to accurately determine the micro-feedback state of the control.

[0071] In an optional implementation, before using the state pair dataset for machine learning training, the embodiments of this application may perform multi-threshold color jittering data augmentation on the state pair training samples in the dataset during the preprocessing steps of the state pair dataset. For example, the brightness, contrast, and saturation of the state pair training samples may be randomly jittered to a large amplitude. This allows the machine learning model to recognize the relative characteristics of the control's state changes before and during a click, rather than memorizing absolute color values. This is beneficial for improving the accuracy of the trained micro-state recognition model in determining the micro-feedback state of the control.

[0072] For information on using states to train a dataset for machine learning to obtain a micro-state recognition model, please refer to the aforementioned content on training a UI shape detection model; it will not be repeated here.

[0073] In step S240, the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the acquired controls are subjected to multimodal fusion processing to obtain spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls.

[0074] It should be noted that the spatial location features (such as spatial prediction boxes) and category attribute features of controls in user interface images detected by UI shape detection models, the text semantic features in user interface images identified by text semantic recognition models, and the micro feedback state features of controls obtained by micro state recognition models are heterogeneous spatial features. They exist independently in their respective feature spaces, have no correspondence or correlation with each other, and may have contradictions or conflicts.

[0075] Based on this, multimodal fusion processing is performed on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the acquired controls. That is, spatial feature alignment and contradiction resolution processing are performed on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the controls output by the UI shape detection model, the text semantic features, and the micro-feedback state features, in order to establish the association or correspondence between the spatial location features and category attribute features of the controls in the user interface image detected by the UI shape detection model, the text semantic features, and the micro-feedback state features, in the user interface image, and the micro-feedback state features, in the micro-state recognition model. This provides a foundation for the subsequent construction of a control tree with advanced logical relationships.

[0076] In the optional implementation, the spatial position features of the controls output by the UI shape detection model are represented by the coordinates of the control prediction box, and the text semantic features output by the text semantic recognition model are represented by the coordinates of the text box containing the text content.

[0077] Accordingly, for the same user interface image, the step of spatial alignment processing of the acquired spatial position features, category attribute features, text semantic features and micro feedback state features of the control includes: if the center point of the text box output by the text semantic recognition model is located within the control prediction box output by the UI shape detection model, then the text content in the text box output by the text semantic recognition model is used as the text content (text_content) attribute of the control in the control prediction box output by the UI shape detection model.

[0078] For the same user interface image, if the center point of the text box output by the text semantic recognition model is located within the control prediction box output by the UI shape detection model, or if the intersection-union ratio (IUU) between the text box corresponding to the text semantic features and the control prediction box is greater than a preset IUU threshold, it indicates that the text content in the text box output by the text semantic recognition model is directly attached to the control prediction box output by the UI shape detection model, or that the two have a high degree of spatial overlap. This indicates that the control prediction box detected by the UI shape detection model and the text box recognized by the text semantic recognition model are the same semantic object. In this case, the text content in the text box recognized by the text semantic recognition model can be used as the text content (text_content) attribute of the control.

[0079] In an optional implementation, the step of spatially aligning the spatial position features, category attribute features, text semantic features, and micro-feedback state features of the acquired controls for the same user interface image further includes: if the text box output by the text semantic recognition model is within a predetermined extension tolerance of the control prediction box output by the UI shape detection model, and the main projection axis of the text box output by the text semantic recognition model is aligned with the control prediction box output by the UI shape detection model, then the text content in the text box output by the text semantic recognition model is used as the label attribute or descriptor attribute of the control in the control prediction box output by the UI shape detection model.

[0080] For example, in typical designs like "icons with text below" or "switches with instructions on the left," the correlation between the predicted control boxes detected by the UI shape detection model and the text boxes detected by the text semantic recognition model cannot be determined based on the intersection-union ratio (IUGR). However, if the text box detected by the text semantic recognition model is within the predetermined tolerance of the predicted control box detected by the UI shape detection model, and the main projection axis of the text box is aligned with the predicted control box, it indicates that the two are strictly adjacent and aligned in layout, belonging to a strongly semantically bound composite control. In this case, the weighted Euclidean distance and projection overlap algorithm can be used to bind them based on their relative layout positions, merging the detected control and its corresponding text into a complete interactive semantic entity. That is, the text content in the text box detected by the text semantic recognition model becomes the label or descriptor attribute of the control in the predicted control box detected by the UI shape detection model. The label or descriptor attribute of the control is a user-visible control attribute used to display the control's explanatory text, providing direct information for the user to understand the control's function.

[0081] In an optional implementation, the step of conflict resolution for the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the acquired control for the same user interface image includes: if the UI shape detection model does not detect a control at a spatial location, and the text semantic recognition model detects a text box with the interactive word "determined" at the control location at the spatial location with a first detection confidence level higher than a preset detection confidence threshold, then a virtual text button is added to the spatial location as a virtual control at the spatial location.

[0082] If the UI shape detection model fails to detect a predicted control box at a spatial location, while the text semantic recognition model detects a text box with "definite" semantic content at that location, indicating a conflict between the UI shape detection model's and the text semantic recognition model's results, then because the text semantic recognition model's detection confidence level for the text box with "definite" semantic content at that location is greater than a preset first confidence threshold, it suggests that the text semantic recognition model has sufficient confidence in detecting the presence of a text box with "definite" semantic content at that location. This indicates a high probability that a control exists at that location, and consequently, it suggests that the UI shape detection model has missed a detection at that location. Therefore, a virtual text button can be added to this location as a virtual control to recall and complete the missed controls detected by the UI shape detection model.

[0083] In an optional implementation, the step of conflict resolution for the acquired spatial location features, category attribute features, text semantic features, and micro-feedback state features of the control further includes: if the category attribute features of the control output by the UI shape detection model are both lower than the preset second confidence threshold for the detection confidence of switch and button, and the micro-state recognition model identifies that the control has a binary micro-feedback state feature of "selected / unselected", or the semantic text recognition model identifies that one side of the control prediction box has the words "on / off", then the control is determined to be a switch control.

[0084] If the UI shape detection model detects a control whose category attribute is a switch or a button, and the detection confidence score is lower than the preset first confidence threshold, it indicates that the UI shape detection model lacks sufficient confidence in judging the category attribute of the control and cannot clearly classify it into any preset control type. Therefore, the embodiments of this application can combine the recognition results of the micro-state recognition model to achieve accurate determination of the category attribute of the control.

[0085] Specifically, the category attribute of the control corresponding to the micro-feedback state of the control identified by the micro-state recognition model is used as the category attribute of the control. For example, due to the blurriness of control icons in the user interface image, the category attributes of the icons in the control prediction box output by the UI shape detection model are "Switch" and "Button," with detection confidence scores of 0.51 and 0.49, respectively. However, the micro-state recognition model identifies the control as having a binary micro-feedback state feature of "selected / unselected," or the semantic text recognition model identifies the word "on / off" on one side of the control prediction box. In other words, the micro-feedback state of the control identified by the micro-state recognition model indicates that the control is more likely to be a switch. In this case, the category attribute of the icon in the control prediction box can be determined as a switch.

[0086] Please continue to refer to the reference. Figure 1 Step S130 is executed to construct a control tree that includes the hierarchical nesting relationship between multiple controls, based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls.

[0087] It's important to note that after performing multimodal feature fusion processing on the spatial location and category attributes of controls detected by the UI shape detection model, the text semantic features identified by the text semantic recognition model, and the micro-feedback state features identified by the micro-state recognition model, a flat, discrete set of controls ("visual object set") will be obtained, denoted as C = {C_1, C_2, ..., C_N}. In other words, the controls in the control set C are in a discrete state and lack logical relationships.

[0088] Therefore, embodiments of this application can construct a control tree that includes hierarchical nesting relationships between multiple discrete controls based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls. In other words, multiple discrete controls are reconstructed into a control tree structure with standard logic and a defined depth, thereby providing a prerequisite for the automated testing of IVI systems.

[0089] In an optional implementation, the step of constructing a control tree that includes hierarchical nesting relationships between multiple discrete controls, based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls, includes: creating a root node of the control tree, the root node being located at a predetermined position in the full-screen view of the user interface; obtaining a set of multiple discrete controls, the set including information on the spatial locations of the multiple discrete controls; obtaining the spatial envelope relationship between the multiple discrete controls based on the spatial location information (coordinates of the control prediction boxes); and constructing a control tree with the root node as the unique top-level node based on the spatial envelope relationship between the multiple discrete controls.

[0090] The root node serves as the top-level anchor point of the entire control tree, holding references to all child nodes. It clearly defines the hierarchical relationships between parent and child nodes, allowing for recursive traversal of all nodes in the tree, avoiding confusion in node relationships, and ensuring the integrity and traceability of the tree structure.

[0091] In the optional implementation, any position within the visible area of ​​the IVI system screen (spatial coordinates are [0, 0, Screen_With, Screen_Height]) is used as the root node of the control tree.

[0092] In an optional implementation, for a control set C = {C_1, C_2, ..., C_N} containing N discrete controls, the embodiments of this application can calculate the spatial envelope ratio between any two controls C_i and C_j in the control set C, thereby determining the spatial envelope relationship between any two controls C_i and C_j in the control set C based on the spatial envelope ratio between any two controls C_i and C_j in the control set C.

[0093] Specifically, the step of obtaining the spatial envelope relationship between multiple discrete controls based on the spatial location information of multiple discrete controls includes: for any two controls among the multiple discrete controls, calculating the area of ​​the two control prediction boxes and the area of ​​the overlapping region of the two control prediction boxes according to the coordinates of the two control prediction boxes; designating the control with the smaller area of ​​its prediction box as the first control and the control with the larger area of ​​its prediction box as the second control; and calculating the ratio between the area of ​​the first control prediction box and the area of ​​the overlapping region between the first and second control prediction boxes as the spatial envelope ratio (IR).

[0094] For example, for a control set C = {C_1, C_2, ..., C_N} (where N is an integer greater than or equal to 2) containing N discrete controls, each control C_i (where i is an integer greater than or equal to 1 and less than or equal to N) has a control prediction box, denoted as B_i = [x_{min}, y_{min}, x_{max}, y_{max}]. When calculating the spatial envelope ratio between any two controls C_i and C_j in the control set C, taking control C_j as the first control with the smaller predicted box area and control C_i as the second control with the larger predicted box area as an example, the spatial envelope ratio between the first control C_j and the second control C_i can be calculated using the following formula: IR(i,j) = Area(B_i cap B_j) / {Area(B_j)}(1) Where IR(i,j) represents the spatial envelope ratio between the first control C_i and the second control C_j, Area(B_i capB_j) represents the area of ​​the overlapping region between the prediction boxes of the first control C_i and the second control C_j, and Area(B_j) represents the area of ​​the first control C_j.

[0095] As can be seen from the above formula, the spatial envelope ratio between any two controls C_i and C_j in the control set C is used to represent the degree of overlap between the predicted bounding boxes of any two controls C_i and C_j in the control set C.

[0096] Accordingly, in the control set C, if the spatial envelope ratio between controls C_i and C_j is greater than the preset envelope ratio threshold (T_{enclosure}), it indicates that the predicted bounding boxes of controls C_i and C_j highly overlap. Therefore, it can be determined that the second control C_i, with a larger predicted bounding box area, physically encloses the first control C_j, with a smaller predicted bounding box area. In this case, the second control C_j can be used as the parent node enclosing the first control C_i. If the spatial envelope ratio between the two controls C_i and C_j is less than or equal to the envelope ratio threshold, it indicates that the overlap between the predicted bounding boxes of controls C_i and C_j is small. Therefore, it can be determined that the second control C_i, with a larger predicted bounding box area, does not physically enclose the first control C_j, with a smaller predicted bounding box area.

[0097] It should be noted that for more complex in-vehicle user interfaces in IVI systems, small controls are often enclosed by multiple large containers. Therefore, in complex in-vehicle user interfaces, determining the parent-child relationship between controls solely through spatial envelopment can lead to ambiguity in parent node determination.

[0098] For ease of understanding, Figure 4 An example diagram of an in-vehicle user interface is shown below. Please refer to the references. Figure 4 The root view of the in-vehicle user interface 400 includes a music playback card 410, which contains a playback control area 420, and the playback control area 420 contains a pause icon 430. Therefore, the pause icon 430 is enveloped by both the playback control area 420 and the music playback card 410. If only the above spatial envelopment relationship is considered, the pause icon 430 will have two parent nodes simultaneously: the playback control area 420 and the music playback card 410.

[0099] To address this situation, this application embodiment utilizes a minimum area envelope constraint algorithm to determine a unique envelope parent node from multiple envelope parent nodes, which serves as the unique direct parent node of the control. Specifically, for any control C_i in the control set C, it may have multiple envelope parent nodes in the control set C, forming an envelope parent node set, denoted as P_i = {C_{p1}, C_{p2}, ..., C_{pk}}. In other words, all elements in the envelope parent node set P_i satisfy the aforementioned envelope and enclosed relationship with control C_i. Furthermore, for the envelope parent node set P_i, the predicted control box area of ​​each element is obtained, and the envelope parent node with the smallest predicted control box area is determined as the unique direct parent node of control C_i. In other words, the unique direct parent node of control C_i satisfies: (2) in, This represents the parent node with the smallest predicted box area among the parent nodes of control C_i, which is also the unique direct parent node of control C_i. This operation returns the value of the independent variable that yields the minimum value. This represents the area of ​​the prediction box of the parent node C_{pk} in the set of parent nodes P_i.

[0100] By traversing each control C_i in the control set C and executing the minimum area envelope constraint algorithm described above, this embodiment of the application can construct an initial spatial envelope tree structure with a strict tree topology. At this point, text and icons will be accurately attached to cards, panels, or buttons, completing the initial evolution from a discrete control set to a nested tree.

[0101] It should be noted that for any two controls C_i and C_j in the control set C, in addition to the aforementioned envelope and enclosed relationship, there may also be a spatial partially overlapping but not completely enveloped relationship, that is, the spatial envelope ratio between controls C_i and C_j may be less than or equal to the envelope ratio threshold.

[0102] To address this issue, this embodiment of the application uses a preset category priority dictionary to determine the hierarchical positions of controls C_i and C_j in the initial spatial envelope tree structure. The category priority dictionary includes information on the Z-axis depth values ​​of various preset control types. The Z-axis depth value is the physical pixel offset of the control along the Z-axis, determining the control's hover height and shadow effect. Controls with larger Z-axis depth values ​​are assigned larger Z-order values, appearing visually more prominent and overshadowing controls with smaller Z-axis depth values. Accordingly, the Z-axis depth values ​​of controls in the category priority dictionary gradually increase according to their category priority from high to low.

[0103] Based on this, this embodiment of the application determines the Z-axis depth values ​​of controls C_i and C_j by calling a category priority dictionary, thereby determining the stacking order of controls C_i and C_j on the screen according to their Z-axis depth values. Specifically, if the Z-axis depth value of control C_i is greater than that of control C_j, then control C_i is determined to be on top of control C_j; if the Z-axis depth value of control C_i is less than that of control C_j, then control C_j is determined to be on top of control C_i.

[0104] As a result, controls with higher category priority are elevated in the hierarchy of the control tree, making them accessed earlier and easier to locate during traversal. This ensures that controls on the surface are clicked first, avoiding logical errors caused by clicks that penetrate the surface.

[0105] It's important to note that while the initial spatial envelope tree is physically correct after construction using the methods described above, its business logic still requires improvement. For example, the bottom of a car's air conditioning interface has five side-by-side airflow adjustment buttons, which visually lack a defined envelope. In the initial spatial envelope tree, these five buttons are directly attached to the full-screen root node. However, the developers of automated test scripts expect to iterate through these five buttons with the same category attributes as a list or group.

[0106] Based on this, in a further optional implementation, after completing the construction of the initial spatial envelope tree, this embodiment of the application uses the alignment affinity rule to reverse deduce the logical container (i.e., the special parent node) hidden in the UI design layout, and automatically inserts the corresponding virtual node in the control tree to complete the grouping of sibling node sequences that are mounted under the same parent node and do not intersect with each other, thereby realizing the topological reconstruction of the corresponding subtree structure.

[0107] In an optional implementation, embodiments of this application determine sibling child nodes with affinity in the horizontal and / or vertical directions by traversing each level of the initial spatial envelope tree and enabling one-dimensional affinity scanning and two-dimensional affinity scanning operations within the current level being traversed.

[0108] For ease of understanding, Figure 5 An example diagram is shown illustrating the calculation of horizontal row affinity or vertical column affinity between any two disjoint sibling child nodes under the same parent node. Please refer to the reference. Figure 5 Taking any two non-overlapping sibling child nodes under the same parent node, namely control C_m and control C_n, as an example, this embodiment first calculates the alignment deviation |cy_m - cy_n| between the geometric center points of control C_m and control C_n in the horizontal or vertical direction. Then, based on the alignment deviation |cy_m - cy_n| between the geometric center points (cx_m, cy_m) of control C_m and (cx_n, cy_n) of control C_n in the horizontal or vertical direction, it is determined whether control C_m and control C_n have horizontal row affinity or vertical column affinity.

[0109] Specifically, if the deviation of the geometric center points of controls C_m and C_n along the horizontal or vertical direction is less than a preset deviation threshold (such as T_{align}, and T_{align} can be 10% of the average height of the controls), then it is determined that controls C_m and C_n have horizontal row affinity or vertical column affinity.

[0110] Therefore, by traversing each level of the initial spatial envelope tree, this embodiment of the application can determine all sibling child nodes in the initial spatial envelope tree that have horizontal row affinity or vertical column affinity. Then, by using the horizontal row affinity or vertical column affinity between sibling child nodes, an affinity connectivity graph composed of three or more controls with horizontal row affinity or vertical affinity can be obtained. Based on whether the three or more sibling child nodes in the affinity connectivity graph satisfy the characteristic of equidistant distribution, it can be determined whether the three or more sibling child nodes in the affinity connectivity graph are a sequence of sibling nodes with the same origin category attribute.

[0111] For ease of understanding, Figure 6 An example diagram is shown to determine whether a sequence of more than three sibling child nodes in an affinity connectivity graph is a sequence of sibling nodes with the same class attribute. Please refer to the reference. Figure 6Taking an affinity-connected graph consisting of k (k is an integer greater than or equal to 3) sibling child nodes c_1, c_2, ..., c_k with horizontal row affinity as an example, we can first calculate the spacing vector g_m (m is an integer greater than or equal to 1 and less than or equal to k) between two adjacent sibling child nodes in the affinity-connected graph, forming the corresponding spacing vector G = {g_1, g_2, ..., g_{k-1}}, and then calculate the variance sigma of the spacing vector G. 2 (G); if the variance of the spacing vector G is sigma 2 If (G) is less than or equal to a preset variance threshold (0 is close to 0), it indicates that the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph conform to the characteristic of equal interval distribution. Therefore, it can be determined that the sibling child nodes c_1, c_2, ..., c_k are a sequence of sibling nodes with the same category attribute. If the variance of the spacing vector G is sigma 2 If (G) is greater than the variance threshold, it indicates that the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph do not conform to the equidistant distribution characteristic. Therefore, it can be determined that the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph are not a sequence of sibling nodes with the same category attribute.

[0112] In a further alternative implementation, to avoid misjudging spatially aligned but logically unrelated controls (such as a map button and a distant text label) as a sequence of sibling nodes with the same category attribute, this application embodiment introduces constraints on morphological and semantic consistency.

[0113] Specifically, please refer to the following: Figure 6 The variance sigma of the calculated spacing vector G is determined. 2 (G) After the variance threshold is less than or equal to the preset value, the category attribute features and control prediction box size features of the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph are obtained respectively. If more than a preset proportion (such as 80%) of the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph have the same category attribute and similar prediction box size features, it indicates that the k sibling child nodes c_1, c_2, ..., c_k in the affinity connectivity graph are consistent in both semantic and morphological dimensions. Therefore, it can be determined that the sibling node sequence in the affinity connectivity graph is a sibling node sequence with the same category attribute.

[0114] Furthermore, after determining that the sequence of sibling nodes in the affinity connectivity graph is a sequence of sibling nodes with the same category attribute, the embodiments of this application can perform topological transformation on the corresponding subtree structure in the initial spatial envelope tree. Specifically, please refer to the reference... Figure 7 Taking a sibling node set S_{group} = {C_{g1}, C_{g2}, ..., C_{gk}} consisting of a sequence of sibling nodes with the same category attribute as an example, calculate the minimum bounding box containing all controls in the sibling node set S_{group} as the corresponding virtual envelope box. The coordinates of this virtual envelope box are defined as V_{box} = [min(x_{min}), min(y_{min}), max(x_{max}), max(y_{max})]. Then, generate a virtual control object that is not natively visible to the system and assign it the corresponding category attribute (such as Virtual LinearLayout_Horizontal or Virtual RecyclerView). After that, unify the parent node pointers of each control in the original set S_{group} and re-point them to the newly created virtual control object, and attach the virtual control object to the original direct parent node of the control in the set S_{group}.

[0115] That is, after determining that the sibling node sequence in the affinity connectivity graph is a sibling node sequence with the same category attribute, the process of constructing the control tree may include: creating the smallest outer bounding box including the sibling node sequence with the same category attribute as a virtual envelope box; generating a virtual control object corresponding to the virtual envelope box and assigning the virtual control object the corresponding category attribute; using the virtual control object as the parent node of the sibling node sequence with the same category attribute in the virtual envelope box, and as the child node of the parent node of the sibling node sequence with the same category attribute in the virtual envelope box.

[0116] By using the aforementioned topology reconstruction method, this application embodiment can achieve the collation and categorization of messy tiled controls, and perfectly restore the visually unrecognizable hidden UI component tree (Layout Hierarchy), thereby finally obtaining a control tree structure that is geometrically accurate and structurally semantically fully conforms to the traversal logic expectations of human-computer interaction testing, thus providing a foundation for subsequent automated testing of the IVI system by traversing the control tree structure.

[0117] Based on the same inventive concept, embodiments of the present invention also provide a computer device that can implement the control tree construction method provided in embodiments of the present invention by loading a program.

[0118] refer to Figure 8 A schematic diagram of an optional hardware structure of a computer device according to an embodiment of the present invention is shown. The computer device in this embodiment includes: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0119] In this embodiment, the number of processor 01, communication interface 02, memory 03 and communication bus 04 is at least one, and processor 01, communication interface 02 and memory 03 communicate with each other through communication bus 04.

[0120] Communication interface 02 can be an interface for a communication module used for network communication, such as an interface for a Global System for Mobile Communications (GSM) module.

[0121] Processor 01 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the control tree construction method provided in this embodiment.

[0122] Memory 03 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Memory 03 stores one or more computer instructions, which are executed by processor 01 to implement the control tree construction method provided in the foregoing embodiments.

[0123] It should be noted that the aforementioned computer device may also include other devices (not shown) that may not be essential to understanding the content disclosed in the embodiments of the present invention; given that these other devices may not be essential for understanding the content disclosed in the embodiments of the present invention, the embodiments of the present invention will not describe them one by one.

[0124] Based on the same inventive concept, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, are used to implement the control tree construction method described in the embodiments of the present invention. For a detailed description of the control tree construction method provided in the embodiments of the present invention, please refer to the foregoing section.

[0125] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing one or more computer instructions for implementing the control tree construction method provided in the foregoing embodiments. For a detailed description of the control tree construction method provided in the foregoing sections, please refer to the relevant sections.

[0126] The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise stated, elements or features may be considered optional. Individual elements or features may be practiced without combination with other elements or features. Furthermore, embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some constructions of any embodiment may be included in another embodiment and may be replaced by corresponding constructions of another embodiment. It will be apparent to those skilled in the art that claims in the appended claims that are not explicitly referenced in each other may be combined to form embodiments of the present invention, or may be included as new claims in amendments filed after the submission of this application.

[0127] Embodiments of the present invention can be implemented by various means, such as hardware, firmware, software, or combinations thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0128] In firmware or software configuration, embodiments of the present invention can be implemented in the form of modules, processes, functions, etc. Software code can be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and can send data to and receive data from the processor via various known means.

[0129] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is accorded the widest scope consistent with the principles and novel features disclosed herein.

[0130] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for constructing a control tree, characterized in that, include: Obtain the user interface image of the IVI system; The spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls are obtained from the user interface image of the IVI system. Based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, a control tree is constructed that includes the hierarchical nesting relationship between the multiple controls.

2. The control tree construction method as described in claim 1, characterized in that, The acquisition of spatial location features, category attribute features, text semantic features, and micro-feedback state features of multiple discrete controls from the user interface image of the IVI system includes: A pre-trained UI shape detection model is used to perform UI shape detection and category attribute recognition on the user interface image of the IVI system, and information on the control prediction boxes and category attributes present in the user interface image of the IVI system is obtained. The pre-trained text semantic recognition model is used to perform text semantic recognition processing on the user interface image of the IVI system to obtain information about text boxes containing text content in the user interface image of the IVI system. The micro-state recognition model obtained in the pre-trained process is used to perform micro-feedback state recognition processing on the user interface image of the IVI system to obtain information on the micro-feedback state of the controls present in the user interface image of the IVI system. Multimodal fusion processing is performed on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the acquired controls to obtain the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls.

3. The control tree construction method as described in claim 2, characterized in that, For the same user interface image, the multimodal fusion processing of the acquired spatial location features, category attribute features, text semantic features, and micro-feedback state features of the controls includes at least one of the following: If the center point of the text box output by the text semantic recognition model is located within the control prediction box output by the UI shape detection model, or if the preset intersection-union ratio of the center point of the text box output by the text semantic recognition model and the control prediction box output by the UI shape detection model is greater than the preset intersection-union ratio threshold, then the text content in the text box output by the text semantic recognition model is used as the text content attribute of the control in the control prediction box output by the UI shape detection model. If the text box output by the text semantic recognition model is within the predetermined extension tolerance of the control prediction box output by the UI shape detection model, and the main projection axis of the text box output by the text semantic recognition model is aligned with the control prediction box output by the UI shape detection model, then the text content in the text box output by the text semantic recognition model is used as the label attribute or descriptor attribute of the control in the control prediction box output by the UI shape detection model. If the UI shape detection model does not detect a control at a spatial location, and the text semantic recognition model detects a text box with the interactive word "OK" semantics at the control location at the spatial location with a first detection confidence score higher than the preset detection confidence score threshold, then a virtual text button is added to the spatial location as a virtual control at the spatial location. If the category attribute features of the control output by the UI form detection model are that the detection confidence of both the switch and the button is lower than the preset second confidence threshold, and the micro state recognition model identifies that the control has a binary micro feedback state feature of "selected / unselected", or the semantic text recognition model identifies that the control prediction box has the word "on / off" on one side, then the control is determined to be a switch control.

4. The control tree construction method as described in claim 1, characterized in that, The control tree, which constructs a hierarchical nesting relationship between multiple controls based on the spatial location features, category attribute features, text semantic features, and micro-feedback state features of the multiple discrete controls, includes: Create a root node, which is located at a predetermined position in the full-screen view of the user interface; Obtain a control set of the plurality of discrete controls, the control set including information on the coordinates of the control prediction boxes of the plurality of discrete controls; Based on the coordinate information of the multiple discrete control prediction boxes in the control set, the spatial envelope relationship between the multiple discrete controls is obtained; Based on the spatial envelope relationship between the multiple discrete controls, a control tree is constructed with the root node as the unique top-level node.

5. The control tree construction method as described in claim 4, characterized in that, The step of obtaining the spatial envelope relationship between the multiple discrete controls based on the coordinate information of the multiple discrete control prediction boxes in the control set includes: For any two controls in the control set, calculate the area of ​​the two control prediction boxes and the area of ​​the overlapping area of ​​the two control prediction boxes based on the coordinates of the two control prediction boxes respectively. The control with the smaller predicted box area of ​​the two controls is designated as the first control, and the control with the larger predicted box area of ​​the two controls is designated as the second control. Calculate the area ratio between the predicted area of ​​the first control and the overlapping area between the predicted areas of the first control and the second control, and use this ratio as the spatial envelope ratio of the first control and the second control.

6. The control tree construction method as described in claim 5, characterized in that, The construction of a control tree with the root node as the unique top-level node, based on the spatial envelope relationship between the multiple discrete controls, includes: If the spatial envelope ratio is greater than a preset envelope ratio threshold, then the second control is used as the envelope parent node of the first control; the envelope parent node of each control in the control set is obtained, and corresponding envelope parent node sets are formed respectively; If there is only one envelope parent node in the set of envelope parent nodes, then the unique envelope parent node in the set of envelope parent nodes shall be used as the parent node of the corresponding control. If there are multiple envelope parent nodes in the set of envelope parent nodes, the envelope parent node with the smallest area of ​​the control prediction box is selected from the set of envelope parent nodes and used as the parent node of the corresponding control.

7. The control tree construction method as described in claim 6, characterized in that, If the two controls partially overlap and their spatial envelope ratio is less than or equal to the envelope ratio threshold, the step of constructing a control tree with the root node as the unique top-level node based on the spatial envelope relationship between the multiple discrete controls includes: The stacking relationship between the two controls is determined based on their category priority in a preset category priority order.

8. The control tree construction method as described in claim 6, characterized in that, After determining the parent node of each control in the control set, constructing the control tree with the root node as the unique top-level node further includes: Get a sequence of three or more sibling nodes with the same parent node that share the same category attribute; Create a minimum bounding box that includes a sequence of three or more sibling nodes with the same parent node and the same class attribute, as a virtual envelope box; Generate a virtual control object that corresponds to the virtual envelope and is not natively visible in the IVI system, and assign the corresponding category attribute to the virtual control object; The virtual control object is used as the parent node of the sibling node sequence with the same category attribute in the virtual envelope, and the virtual control object is used as the child node of the parent node of the sibling node sequence with the same category attribute in the virtual envelope.

9. The control tree construction method as described in claim 7, characterized in that, The step of obtaining a sequence of three or more sibling nodes with the same parent node and the same category attribute includes: For any two adjacent and non-intersecting sibling child nodes under the same parent node, if the deviation between the center points of the control prediction boxes of the two adjacent and non-intersecting sibling child nodes in the horizontal or vertical direction is less than a preset deviation threshold, then it is determined that the two adjacent and non-intersecting sibling child nodes have horizontal row affinity or vertical column affinity. Obtain an affinity connectivity graph consisting of three or more sibling child nodes with horizontal or vertical affinity under the same parent node, and calculate the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph. If the variance of the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph is less than or equal to a preset variance threshold, then the sibling child nodes in the affinity connectivity graph are determined to be a sequence of sibling nodes with the same category attribute.

10. The control tree construction method as described in claim 7, characterized in that, If the variance of the spacing vector between any two adjacent sibling child nodes in the affinity connectivity graph is less than or equal to a preset variance threshold, the step of obtaining a sequence of three or more sibling nodes with the same parent node and the same class attribute further includes: If the sibling child nodes in the affinity connectivity graph have morphological and semantic consistency, then the sibling child nodes in the affinity connectivity graph are determined to be sibling nodes with the same category attribute.

11. A computer device, characterized in that, It includes at least one memory and at least one processor, the memory storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the control tree construction method as described in any one of claims 1 to 10.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they are used to implement the control tree construction method as described in any one of claims 1 to 10.

13. A storage medium, characterized in that, The storage medium stores one or more computer instructions, which are used to implement the control tree construction method as described in any one of claims 1 to 10.

Images