Training data generation method and device, electronic equipment and storage medium

Abstract

The training data generation method and device, the electronic equipment and the storage medium provided by the present disclosure relate to the technical field of data processing, can comprehensively capture static and dynamic information of a terminal interface by collecting multi-modal data containing interface images, control structure information and context information, realize automatic semantic association and fusion of icons and text controls by screening candidate icons and text controls and calculating their spatial and semantic association scores to generate aggregated controls, generate chain training data with context association by performing interactive operations based on the aggregated controls and constructing an interactive behavior chain, and realize automatic data construction of the whole process by integrating multi-modal data, aggregated controls and interactive behavior chains to improve data generation efficiency and quality. Therefore, the present disclosure can efficiently generate high-quality, multi-modal and time-sequenced training data that meets the training needs of multi-modal large models.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a training data generation method and apparatus, electronic device and storage medium. Background Technology

[0002] When training and developing AI models with control terminal functions, automated test training data is required. Automated test training data is used to enable AI models to learn and understand corresponding operation instructions. For example, a driver can use voice interaction to control the vehicle's air conditioning temperature and airflow.

[0003] There are several typical methods for constructing such training data: script-driven interface traversal and screenshot acquisition, image semantic box selection tools based on manual annotation, and structured data export based on control attribute parsing. However, existing methods for constructing training data have low automation in the data construction process, resulting in low efficiency. The generated data modalities are singular and lack deep fusion and unified expression. Furthermore, they cannot construct data with temporal correlation, which limits the model's ability to understand user operation intentions and task context, and cannot meet the training needs of multimodal large models.

[0004] Therefore, how to efficiently generate training data that meets the training requirements of multimodal large models is an urgent problem to be solved. Summary of the Invention

[0005] This disclosure provides a training data generation method, apparatus, electronic device, and storage medium. Its main objective is to address the problem of how to efficiently generate training data that meets the training requirements of large multimodal models. According to a first aspect of this disclosure, a method for generating training data is provided, comprising: Collect multimodal data from the terminal interface, including interface images, control structure information, and context information; The control structure information is filtered to obtain multiple candidate icon controls and multiple candidate text controls. The association score is calculated based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control. Multiple aggregate controls are generated based on the association score. The interaction operation is performed based on the target aggregation control, the interface state change information during the interaction operation is obtained, and the interaction behavior chain is constructed based on the interface state change information. The target aggregation control is any control among multiple aggregation controls. Multimodal data, multiple aggregate controls, and interactive behavior chains are integrated and processed to generate structured training data for model training.

[0006] According to a second aspect of this disclosure, a training data generation apparatus is provided, comprising: The acquisition unit is used to acquire multimodal data of the terminal interface, including interface images, control structure information, and context information. The calculation unit is used to filter the control structure information to obtain multiple candidate icon controls and multiple candidate text controls, and calculate the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control. The first generation unit is used to generate multiple aggregate controls based on the association score; The acquisition unit is used to perform interactive operations based on the target aggregation control and acquire information on changes in the interface state during the interactive operation. The building unit is used to construct an interactive behavior chain based on the interface state change information, wherein the target aggregate control is any control among multiple aggregate controls; The second generation unit is used to integrate and process multimodal data, multiple aggregate controls, and interactive behavior chains to generate structured training data for model training.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method described in the first aspect above.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method of the first aspect described above.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in the first aspect above.

[0010] The training data generation method, apparatus, electronic device, and storage medium disclosed herein, by collecting multimodal data including interface images, control structure information, and contextual information, can comprehensively capture the static and dynamic information of the terminal interface, providing a foundation for data fusion. By filtering candidate icons and text controls and calculating their spatial and semantic association scores to generate aggregated controls, automatic semantic association and fusion of icons and text controls are achieved, overcoming the problems of single data modality and lack of deep fusion, and generating training samples with complete semantic expression. By executing interactive operations based on aggregated controls and constructing interactive behavior chains, the temporal changes in interface state caused by operations are automatically captured, generating chained training data with contextual association, solving the problems of inability to construct temporally associated data and limiting the model's understanding of operational intentions. By integrating multimodal data, aggregated controls, and interactive behavior chains to generate structured training data, the entire process of automated data construction is achieved, unifying data standards and formats, and improving data generation efficiency and quality. Therefore, this disclosure can efficiently generate high-quality, multimodal, and temporally associated training data that meets the training needs of large multimodal models, solving the technical problems of low efficiency, single modality, and lack of temporal association in existing dataset construction.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a training data generation method provided in an embodiment of the present disclosure. Figure 2 A flowchart illustrating another training data generation method provided in this embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a training data generation device provided in an embodiment of the present disclosure; Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] This disclosure relates to a training data generation method, and more particularly to an automated test training data generation method that integrates multimodal understanding and human-computer interaction perception. This method, through a systematic data acquisition, processing, and integration process, automatically constructs high-quality, structured training datasets, which are at least suitable for training large multimodal models with terminal interface understanding and operation capabilities.

[0015] The following description, with reference to the accompanying drawings, outlines a training data generation method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0016] Figure 1 This is a flowchart illustrating a training data generation method provided in an embodiment of the present disclosure.

[0017] like Figure 1 As shown, the method includes the following steps: Step 101: Collect multimodal data of the terminal interface, including interface images, control structure information and context information.

[0018] In the embodiments of this disclosure, multimodal data refers to a collection of multiple types of information simultaneously acquired from the terminal interface. These multiple types of information complement each other modally, jointly constituting a comprehensive description of the interface. Specifically, multimodal data includes interface images, control structure information, and contextual information.

[0019] Interface images refer to the visual representation of a terminal interface obtained through screenshots or other means. They are two-dimensional pixel matrices that contain the layout, color, and texture information of all visual elements on the interface.

[0020] Control structure information refers to the interface hierarchy description file obtained through interface automation testing frameworks (such as Android's User Interface (UI) automation testing framework (Uiautomator) or iOS's XCUITest). It is usually in the format of Extensible Markup Language (XML) or JavaScript Object Notation (JSON). It records the type, unique identifier, position coordinates on the screen, and text content of each control element (such as buttons, text boxes, icons, etc.) on the interface.

[0021] Contextual information refers to environmental parameters related to the data acquisition process, such as the timestamp at the time of acquisition and the unique page identifier of the current interface. This information helps in the time-series management and unique identification of data. By acquiring multimodal data, a rich and interconnected foundation of raw information is provided for subsequent processing.

[0022] Specifically, the content and format of multimodal data can also be represented in, but are not limited to, the following ways: remotely connect to the test device via a terminal control interface (such as Android Debug Bridge (ADB) + Uiautomator2), and collect data from multiple modalities of a certain interface to form a triplet. : The data structure is defined as follows: .

[0023] in: : Screenshot image of the interface, where H is the height of the screenshot in pixels and W is the width of the screenshot in pixels; The collection of all control structures in the screenshot interface; Contextual information, including timestamps Page Identifier (ID): The interface ID can be identified using Message-Digest Algorithm 5 (MD5) or similar algorithms.

[0024] Each control element Defined as: .

[0025] Control types include ImageView (image view), TestView (text view), ImageButton (image button), etc. : control The bounding box; : control ID; : control The text content.

[0026] Step 102: Filter the control structure information to obtain multiple candidate icon controls and multiple candidate text controls. Calculate the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control, and generate multiple aggregate controls based on the association score.

[0027] In the embodiments of this disclosure, an aggregate control refers to a higher-level UI element unit formed by logically binding multiple semantically closely related basic control elements (usually an icon control and a text control).

[0028] Candidate icon controls typically refer to ImageView or similar visual elements, which may lack explicit text descriptions. Candidate text controls refer to TextView or similar elements containing non-empty text content. Next, the spatial similarity and semantic relevance between each candidate icon control and each candidate text control are calculated. Spatial similarity is a metric used to quantify the physical proximity of two controls on the screen. It is typically calculated based on the Euclidean distance between the center point coordinates or bounding boxes of the two controls; the closer the distance, the higher the spatial similarity, indicating a close relationship in appearance. Semantic relevance is a metric used to quantify the functional or semantic connection between two controls. It can be calculated using a pre-trained multimodal large model, extracting and encoding features from the image slices of the icon control and the text content of the text control, respectively. Then, measures such as cosine similarity between the two feature vectors are calculated; higher similarity indicates stronger semantic relevance.

[0029] Subsequently, a comprehensive association score is calculated by combining spatial similarity and semantic relevance. This score reflects the likelihood that the icon and text control belong to the same functional entity. Finally, based on whether the association score exceeds a preset threshold, icon and text controls with strong association are combined to form aggregated controls. Each aggregated control typically includes a merged bounding box (covering the areas of the original icon and text) and a joint semantic description.

[0030] Specifically, multiple candidate icon controls and multiple candidate text controls can be represented in, but are not limited to, the following ways: A collection of candidate icon controls: ; Candidate text control collection: .

[0031] Step 103: Perform interactive operations based on the target aggregation control, obtain interface state change information during the interactive operation, and construct an interactive behavior chain based on the interface state change information. The target aggregation control is any control among multiple aggregation controls.

[0032] In the embodiments of this disclosure, the interaction behavior chain refers to a time-series data sequence that records a series of continuous user operations and the resulting changes in the interface state. It describes the complete process from the initial interface state, through the execution of specific operations, triggering interface jumps or content updates, until the final interface state is reached.

[0033] The process of constructing an interactive behavior chain first requires selecting a target aggregate control from multiple generated aggregate controls. The target aggregate control is the object to be interacted with. Then, an automated script or tool (such as ADB commands simulating clicks) is used to perform the interactive operation on the target aggregate control; the most common interactive operation is a click. During the execution of the interactive operation, information on changes in the interface state is continuously monitored and acquired.

[0034] Interface state change information reflects the changes that occur in the interface before and after an operation. This can include differences in the visual features of the interface image, changes in the control tree structure, and alterations to key text content. By analyzing interface state change information, a valid state transition can be identified, and this interaction and its resulting state change can be added as a step to the interaction behavior chain. By repeating this process, a complete interaction behavior chain containing multiple steps and state transitions can be constructed, thus providing the model with crucial information about the learning task flow and operational context.

[0035] Step 104: Integrate and process the multimodal data, multiple aggregate controls, and interactive behavior chains to generate structured training data for model training.

[0036] In the embodiments of this disclosure, data integration processing refers to a series of operations involving cleaning, correlating, standardizing, and format conversion of various types of data. This process organically integrates the collected raw multimodal data (including interface images, control structure information, and context information), the generated multiple aggregated controls, and the constructed interactive behavior chain information.

[0037] Specifically, this involves establishing relationships between images, controls, and interactive operations. For example, each interface image is bound to its corresponding aggregated control set, and each step in the interactive behavior chain is associated with the corresponding pre-operation and post-operation interface states. In terms of data format, it is typically converted to a standardized, machine-readable structured format, such as a structured training data format extended from a common dataset format (e.g., the COCO dataset format (Common Objects in Context, COCO)). The resulting structured training dataset contains visual, textual, structural, and behavioral sequence information, possessing high standardization and consistency. It can be directly read and used for training by large multimodal models without tedious manual cleaning or format conversion, effectively improving data utilization and model training efficiency.

[0038] Furthermore, to facilitate understanding of the implementation process of the embodiments of this disclosure, this disclosure also provides a flowchart of another training data generation method, such as... Figure 2As shown, based on multimodal information such as images, structures, text, and user operation behavior of the interface, the semantic combination relationship between icons and text is automatically identified, and structured training samples are constructed to meet the training requirements of multimodal large models.

[0039] This disclosure utilizes multimodal data, including interface images, control structure information, and contextual information, to filter candidate icons and text controls and calculate their spatial and semantic association scores to generate aggregated controls. Based on these aggregated controls, interactive operations are performed, and an interactive behavior chain is constructed. By integrating multimodal data, aggregated controls, and interactive behavior chains to generate structured training data, the entire process of training data generation is automated, significantly reducing manual annotation costs. The generated data integrates visual, textual, and interactive behavior information, enabling models trained on this data to more deeply understand the semantics and operational logic of the terminal interface, improving the model's perception and decision-making capabilities in scenarios such as automated testing and intelligent assistants.

[0040] In one possible implementation of this disclosure, before calculating the association score, it is necessary to calculate spatial similarity and semantic relevance. Specifically, but not limited to the following methods, can also be used: calculate the first spatial similarity between the target icon control and the target text control, until the similarity between each candidate icon control and each candidate text control is calculated to obtain the spatial similarity, wherein the target icon control is any one of the multiple candidate icon controls, and the target text control is any one of the multiple candidate text controls; perform joint encoding on the target icon control and the target text control based on a preset multimodal model to obtain the first semantic relevance, until the relevance between each candidate icon control and each candidate text control is calculated to obtain the semantic relevance.

[0041] In the embodiments of this disclosure, the spatial similarity calculation process adopts an iterative approach, selecting one target icon control from multiple candidate icon controls and one target text control from multiple candidate text controls. The target icon control refers to a single icon control instance to be processed in the current iteration cycle; it is typically an ImageView or similar visual element. The target text control refers to a single text control instance to be processed in the current iteration cycle; it typically contains specific textual descriptions. For each pair of target icon controls and target text controls, a first spatial similarity is calculated between them. The first spatial similarity specifically refers to the spatial relationship metric between this pair of controls, and its calculation is typically based on the bounding box coordinates of the two controls. For example, by calculating the Euclidean distance between the center points of the two bounding boxes and using a normalization function to map this distance to a standardized similarity score range. By traversing all combinations of candidate icon controls and all candidate text controls and repeating the above calculation, a complete spatial similarity matrix or set, i.e., spatial similarity, is finally obtained. This result covers the positional relationship information of all possible icon and text control pairs.

[0042] Simultaneously, a pre-set multimodal model is needed to calculate the semantic relevance between all candidate control pairs. The pre-set multimodal model is a pre-trained artificial intelligence model with cross-modal understanding capabilities, capable of processing both image and text information simultaneously and outputting a fused feature representation. For each pair of target icon controls and target text controls, this pre-set multimodal model is invoked for joint encoding. Joint encoding involves inputting the visual information of the target icon control and the text information of the target text control into the pre-set multimodal model, where feature extraction and fusion are performed through the model's deep neural network, ultimately outputting a unified, high-dimensional feature vector. Based on the feature vector generated by this joint encoding, the first semantic relevance between a pair of controls can be calculated. The first semantic relevance specifically refers to the strength of the functional or semantic association between the current pair of controls. Its value is usually obtained by calculating the cosine similarity or other similarity measures between the feature vectors of the two controls; a higher score indicates a stronger semantic match between the two controls. By iterating through all candidate icon controls and all candidate text controls, and repeating the above joint encoding and similarity calculation process, a complete semantic relevance matrix or set is finally obtained, i.e. semantic relevance. This result comprehensively reflects the degree of semantic association of all possible icon and text control pairs.

[0043] This disclosure ensures the comprehensiveness and accuracy of spatial similarity and semantic relevance data through a systematic calculation process. Spatial similarity calculation captures the layout association characteristics between controls, while semantic relevance calculation delves into the functional semantic connections between controls. Together, they provide reliable multi-dimensional input for association score calculation. This significantly improves the accuracy and reliability of aggregated control generation, enabling the final generated training data to more accurately reflect the true semantic structure of the terminal interface.

[0044] In one possible implementation of this disclosure, when calculating the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control, the following method can be used, but is not limited to: according to the first spatial similarity and the first semantic relevance, a weighted calculation is performed by a preset weight to obtain the first association score between the target icon control and the target text control; until the score between each candidate icon control and each candidate text control is calculated, the association score is obtained.

[0045] In the embodiments of this disclosure, the process of calculating the association score first requires calculating an individual score for each pair of candidate controls. Specifically, for the currently processed pair of controls, namely the target icon control and the target text control, a weighted calculation is performed based on the calculated first spatial similarity and first semantic relevance using preset weights. The preset weights refer to a set of coefficients pre-set during the weighted calculation process, used to adjust the relative importance of spatial similarity and semantic relevance in the final score. The weights can be optimized based on specific application scenarios or experience to ensure that the score calculation balances the influence of spatial layout and semantic information. The weighted calculation process is a mathematical operation that multiplies the first spatial similarity and the first semantic relevance by their respective preset weights, and then sums the results to obtain a comprehensive score. Through weighted calculation, information from both spatial and semantic dimensions can be integrated to generate a first association score between the target icon control and the target text control. The first association score is a quantitative value used to represent the comprehensive probability that the current pair of controls belongs to the same functional entity; a higher score indicates a stronger likelihood that the icon and text are combined into a single aggregate control.

[0046] The calculation process is executed iteratively, covering all possible candidate control combinations. It iterates through each pair of candidate icon controls and candidate text controls, repeatedly performing weighted calculations for each pair, namely calculating the first spatial similarity and the first semantic relevance, and then applying preset weights to obtain the corresponding first association score. Through comprehensive iteration and calculation, a complete set of association scores is finally obtained, i.e., the association score. This set of association scores contains the comprehensive score information of all candidate icon control and candidate text control pairings, providing a data foundation for subsequent steps to determine which control pairs should be combined into aggregate controls.

[0047] Specifically, the calculation of association scores can also be implemented in, but is not limited to, the following ways: for any target icon control and target text control. Define its combined association score as :

[0048] Preset weights; First spatial similarity First semantic relevance (when spatial judgment is only) ).

[0049] This disclosure employs weighted calculation processing, which flexibly integrates spatial and semantic information. This allows the association score to consider not only the physical layout relationships between controls but also deeper semantic associations, thereby improving the accuracy of control combination judgments. It ensures that the generated aggregated controls better align with the actual design intent of the user interface, providing higher-quality, more semantically consistent training samples for multimodal large-scale models, thus improving the model's performance in interface understanding and interaction tasks.

[0050] In one possible implementation of this disclosure, when calculating the first spatial similarity between the target icon control and the target text control, it can be implemented in the following ways, but not limited to: converting multiple candidate icon controls and multiple candidate text controls to a preset coordinate system, and obtaining the coordinates of the first control center point of the target icon control and the coordinates of the second control center point of the target text control; calculating the Euclidean distance based on the coordinates of the first control center point and the second control center point, and using the Euclidean distance as the first spatial similarity.

[0051] In the embodiments of this disclosure, when calculating the first spatial similarity between the target icon control and the target text control, it is necessary to transform all candidate controls into a unified reference frame. Specifically, multiple candidate icon controls and multiple candidate text controls are transformed into a preset coordinate system. The preset coordinate system refers to a predefined standard two-dimensional coordinate system, for example: with the upper left corner of the screen as the origin, the positive X-axis pointing horizontally to the right, and the positive Y-axis pointing vertically downwards, with a unit length of pixels. This coordinate transformation is necessary because different terminal devices or interfaces may have different resolutions or scaling ratios. By uniformly transforming to the preset coordinate system, the influence of device differences can be eliminated, ensuring that the spatial position information of all controls is comparable.

[0052] Under the preset coordinate system, the system obtains the coordinates of the first control's center point and the second control's center point for the target icon control. The first control's center point coordinates refer to the position of the center point of the target icon control within its bounding box, and its coordinate value is obtained by calculating the average of the coordinates of the top-left and bottom-right corners of the bounding box. Similarly, the second control's center point coordinates refer to the position of the center point of the target text control's bounding box, calculated in a similar way. These center point coordinates provide a precise reference for the control's position on the screen and are the basis for subsequent distance calculations.

[0053] After obtaining the center point coordinates, the Euclidean distance is calculated based on the center point coordinates of the first control and the center point coordinates of the second control. Euclidean distance is a mathematical method for calculating the straight-line distance between two points in a two-dimensional plane. Its formula is the sum of squares of the differences of squares, calculated based on the difference in coordinates of the two points. The smaller the Euclidean distance value, the closer the two controls are in spatial position, and vice versa. The calculated Euclidean distance is directly used as the measure of first spatial similarity. First spatial similarity specifically refers to the quantified value of the spatial relationship between a pair of target icon controls and target text controls. It is expressed in the form of Euclidean distance; the smaller the distance value, the higher the spatial similarity, indicating a closer layout association between the two controls. The calculation method based on Euclidean distance is simple and effective, intuitively reflecting the physical proximity between controls and providing crucial spatial information input for subsequent association score calculations.

[0054] Specifically, the calculation of the first spatial similarity can also be achieved in the following ways, but not limited to: the coordinates of the center point of the first control are: Similarly, the coordinates of the center point of the second control are... , Then the first spatial similarity is: .

[0055] This disclosure achieves standardization and precision in spatial similarity calculation through the aforementioned calculation of the first spatial similarity. By using a preset coordinate system, the consistency and comparability of all control position data are ensured, while the Euclidean distance calculation based on the center point coordinates provides a reliable method for measuring spatial relationships. This not only improves the accuracy of control association analysis but also enhances the accuracy of the entire training data generation process, enabling the generated aggregated controls to more realistically reflect the actual layout structure of the terminal interface, thereby providing higher-quality spatial semantic information for multimodal large-scale models.

[0056] In one possible implementation of this disclosure, when jointly encoding the target icon control and the target text control based on a preset multimodal model to obtain the first semantic relevance, the following methods can be used, but are not limited to: extracting image slices and text content of the target icon control and the target text control; performing feature fusion on the image slices and text content according to the preset multimodal model to obtain joint feature vectors corresponding to the image slices and text content respectively; calculating the cosine similarity of the joint feature vectors, and normalizing the cosine similarity through a preset activation function to obtain the first semantic relevance.

[0057] In the embodiments of this disclosure, key multimodal elements first need to be extracted from the original interface data. Specifically, image slices and text content of target icon controls and target text controls are extracted. An image slice refers to a local visual region extracted from the overall interface image based on the bounding box coordinates of the control. This region contains complete pixel information of the control and is used for subsequent visual feature extraction. Text content refers to the string data contained in the control, such as the text description displayed on the control. This text directly reflects the function or meaning of the control. By extracting image slices and text content, structured input data is provided for the multimodal model, ensuring that visual and textual information can be processed collaboratively.

[0058] After data extraction, feature fusion is performed on image slices and text content based on a pre-set multimodal model. The pre-set multimodal model refers to a pre-trained artificial intelligence model with cross-modal understanding capabilities, capable of mapping image and text information to the same semantic space. Feature fusion involves integrating the visual features of different image slices or the linguistic features of different text content through the neural network layers of the multimodal model to generate a unified, high-dimensional vector representation. This process yields joint feature vectors for both image slices and text content. The joint feature vector is a numerical representation that integrates visual and textual information; it captures the inherent semantic relationships between icons and text, allowing information from different modalities to be compared and computed in the same vector space. These vectors not only retain the features of the original data but also enhance semantic expressiveness through the encoding capabilities of the multimodal model.

[0059] Next, the cosine similarity of the joint feature vectors is calculated. Cosine similarity is a mathematical measure of the directional consistency between two vectors, with values ​​ranging from -1 to 1. A value closer to 1 indicates that the two vectors are more consistent in direction in the semantic space, meaning the icon and text are more semantically related. After calculating the cosine similarity, it is normalized using a predefined activation function. A predefined activation function is a predefined mathematical function, such as the sigmoid function or the softmax function, which maps the cosine similarity values ​​to a standardized range (e.g., 0 to 1), making the results easier to interpret and compare.

[0060] Normalization ensures that the values ​​of the first semantic relevance have a consistent scale, avoiding deviations in similarity values ​​between different control pairs due to differences in units. Ultimately, through this series of processes, the first semantic relevance is obtained, which is a quantitative value reflecting the strength of the functional or semantic association between the target icon control and the target text control. The higher the value, the more semantically matched they are.

[0061] Specifically, the calculation of the first semantic relevance can also be achieved in the following ways, but not limited to: pre-setting a multimodal large model. Joint encoding of the controls yields a joint feature vector: Joint feature vectors corresponding to image slices: ; Joint feature vector corresponding to the text content: ; The image slices corresponding to the target icon control and the target text control respectively; The text content corresponding to the target icon control and the target text control respectively; A pre-defined multimodal large-scale joint encoder that fuses image and text features to output vectors. ; control elements The first semantic relevance between them is:

[0062] in, The normalization function maps the cosine similarity to [0,1]. This disclosure achieves refinement and standardization of semantic relevance calculation through the calculation of the first semantic relevance. Extracting image slices and text content ensures the quality and representativeness of the input data. The feature fusion process utilizes a multimodal model to capture deep semantic relationships across modalities, while cosine similarity calculation and normalization provide reliable and consistent metric outputs. This significantly improves the accuracy of control semantic association analysis, enabling the generated training data to more realistically reflect the semantic structure of the terminal interface. This provides richer and more accurate semantic learning samples for large multimodal models, enhancing the model's ability to understand the functional intent of interface elements.

[0063] In one possible implementation of this disclosure, when generating multiple aggregate controls based on association scores, the following methods can be used, but are not limited to: in response to an association score not being less than a preset score threshold, the candidate icon control and candidate text control corresponding to the association score are determined as a graphic-text combination control; a fused bounding box is generated based on each graphic-text combination control to obtain multiple aggregate controls.

[0064] In the embodiments of this disclosure, during the process of generating multiple aggregated controls based on association scores, it is first necessary to determine the control combination based on the scores. Specifically, the association score reflects the probability that candidate icon controls and candidate text controls belong to the same functional entity. A preset score threshold is a pre-set critical value used to determine whether the association score is strong enough to support control combination; this threshold can be adjusted according to the needs of the actual application scenario to balance the accuracy and recall of the combination. When the association score is not lower than the threshold, it indicates that the icon and text controls have significant association in terms of spatial layout and semantic function, and therefore they are identified as a logical whole, namely, a graphic-text combined control. A graphic-text combined control refers to a composite control unit formed by logically binding at least one icon control and one text control, representing a complete interactive element in the user interface, such as a button with an icon or an icon item with text description.

[0065] After determining the combined graphic and text controls, a unified visual representation needs to be generated for each control. The merged bounding box is a rectangular box that completely covers the screen area where the original icon and text controls are located. Its coordinates are obtained by calculating the union of the original icon and text control bounding boxes; for example, the minimum X-coordinate, minimum Y-coordinate, maximum X-coordinate, and maximum Y-coordinate of the two bounding boxes are used to define the new rectangular area. The merged bounding box ensures that the combined controls are visually treated as a whole, covering all related elements. Ultimately, each combined graphic and text control and its corresponding merged bounding box together constitute an aggregated control. The aggregated control is a structured data object that not only contains the basic attributes of the original icon and text controls but also the merged spatial information and joint semantic description, thus providing richer and more accurate training samples for the multimodal model.

[0066] Specifically, the generation of aggregation controls can also be achieved in the following ways, but are not limited to: if the calculated relevance reaches a threshold. Then the text and images will be combined to form a merged border.

[0067] like: Then it is believed Combine text and images, and generate a merged bounding box:

[0068] Final generated tags:

[0069] The collection of aggregated controls is as follows: .

[0070] This disclosure uses a preset scoring threshold to ensure the reliability of control combinations and avoids erroneous associations of unrelated controls; while generating fused bounding boxes ensures the integrity of the spatial representation of the combined controls. This improves the quality of training data, enabling the generated aggregated controls to more accurately reflect the actual design intent of the user interface, providing more semantically consistent learning materials for multimodal large models, thereby enhancing the model's understanding of the functions and relationships of interface elements and supporting more precise automated testing and interaction tasks.

[0071] In one possible implementation of this disclosure, when performing an interactive operation based on a target aggregate control and obtaining interface state change information during the interactive operation, the following methods can be used, but are not limited to: selecting a target aggregate control from multiple aggregate controls on the terminal interface; automatically executing the control trigger operation according to a preset execution strategy and capturing the interface state during the control trigger process, wherein the preset execution strategy includes at least one of sequential traversal, weight priority, and random exploration; detecting and processing the interface state based on a preset interface change detection function to obtain interface state change information, wherein the interface change detection function includes at least one of visual feature distance of the interface image, edit distance of the control tree structure, and difference degree of the control text set.

[0072] In the embodiments of this disclosure, the target aggregate control refers to a single control instance selected from a plurality of generated aggregate controls, which serves as the specific object of the current interactive operation; the selection process can be based on random sampling, priority sorting, or specific business rules to ensure coverage of all interactive key elements in the interface. The terminal interface refers to a graphical user interface running on a mobile device, in-vehicle system, or other smart terminal, which contains various visual elements that the user can operate.

[0073] Predefined execution strategies refer to predefined rules for generating interactive operation sequences, including at least one of sequential traversal, weighted priority, and random exploration. Sequential traversal executes operations sequentially according to the spatial position or logical order of aggregated controls on the interface, ensuring that each control is covered. Weighted priority assigns priority based on the type, size, historical interaction frequency, or other attributes of controls, prioritizing the triggering of high-weight controls. Random exploration selects controls randomly for operation, increasing data diversity and the possibility of exploring unknown interface paths. Control-triggered operations are automated actions that simulate user interaction behaviors, most commonly click operations, but may also include long presses, swipes, and other gesture operations. These operations are implemented through terminal control interfaces (such as ADB commands) or UI automation frameworks (such as Uiautomator2). While executing control-triggered operations, the interface state is captured in real time. The interface state refers to a snapshot of the interface at a specific point in time during the interaction, including instantaneous records of multimodal data such as interface images, control hierarchy structure, and text content. The state data reflects the specific performance of the interface before and after the operation.

[0074] To quantify interface changes triggered by interactive operations, a preset interface change detection function is used to detect and process the interface state, obtaining interface state change information. The preset interface change detection function refers to a set of algorithms or metrics used to calculate differences in interface states, including at least one of the following: visual feature distance of the interface image, edit distance of the control tree structure, and difference in the control text set. The visual feature distance of the interface image is achieved by extracting image features of the interface state (such as color histograms, texture features, or feature vectors output by deep learning models) and calculating the Euclidean or cosine distance between the feature vectors before and after the operation. This distance value reflects the degree of visual change in the interface. The edit distance of the control tree structure measures structural change by comparing the minimum number of edit operations (such as inserting, deleting, or replacing nodes) between the control hierarchy tree (usually represented in XML or JSON format) before and after the operation. A larger edit distance indicates a more significant change in the interface layout or component structure. The difference in the control text set is assessed by comparing the content sets of all text controls in the interface before and after the operation, calculating the Jaccard similarity coefficient of the sets or the similarity based on text embeddings to evaluate the addition, deletion, or modification of text information. By comprehensively applying detection functions, comprehensive information on interface state changes can be generated. This information is a quantified value or vector that captures the multimodal changes in the interface caused by interactive operations, providing key input for constructing the interaction behavior chain.

[0075] Specifically, the acquisition of interface state change information can also be achieved in the following ways, but not limited to: by automatically triggering controls to generate a chain structure of page jump - image and text fusion - sample generation, forming a temporal interaction sample. On the basis of the original static interface collection, the dynamic interaction dimension is further expanded, automatically capturing the user's multi-step operation sequence in the application, generating interactive chain multimodal training samples with time and state changes, so that the multimodal large model can not only understand a single interface, but also infer the operation intention and interface state change.

[0076] Let the interface sample at step t after clicking the button (i.e., the interface state before the control is triggered) be: .

[0077] in, For the screenshot of the interface, This is a collection of aggregated controls.

[0078] Aggregator control collection: .

[0079] Operation selection and state transition: Select an interactive control in the current interface (i.e., select the target aggregate control from multiple aggregate controls) to perform an interactive operation.

[0080] Trigger controls according to a preset strategy (sequential traversal / weighted priority / random exploration), perform interactive operations, and capture the next screen (i.e., the screen state after the control is triggered):

[0081] Form an interaction triple (which contains information about interface state changes):

[0082] This disclosure automates and refines the detection of interactive operations and state changes. Intelligent pre-defined execution strategies ensure comprehensive and efficient interaction coverage, while multi-dimensional interface change detection functions provide accurate measurements of state transitions. This not only enhances the reliability and richness of interactive behavior chain data but also enables multimodal large models to learn more deeply the correlation between user operation intentions and dynamic interface responses, thereby improving the model's understanding and reasoning capabilities in complex interactive scenarios.

[0083] In one possible implementation of this disclosure, when constructing an interaction behavior chain based on interface state change information, the following methods can be used, but are not limited to: responding to the interface state change information indicating that the change between the first interface state and the second interface state is not less than a preset change threshold, extracting the first interface state, the executed action, and the second interface state from the interface state change information, wherein the first interface state is the interface state before the control is triggered, the executed action is the interactive operation executed after the control is triggered, and the second interface state is the interface state after the control is triggered; constructing a data chain of the first interface state, the executed action, and the second interface state according to the time sequence to obtain the interaction behavior chain.

[0084] In the embodiments of this disclosure, during the construction of the interaction behavior chain based on interface state change information, the validity of the state transition is first determined based on a change amount threshold. The change amount refers to the overall metric of the difference between the first and second interface states, which can be obtained by weighted fusion of multiple change detection indicators to comprehensively evaluate the significance of the interface state transition. The preset change amount threshold is a pre-set critical value used to determine whether the state change is significant enough to be considered a valid interaction transition; this threshold can be adjusted according to specific application scenarios to filter out irrelevant minor changes or noise interference, ensuring that the constructed interaction behavior chain contains only meaningful operation sequences. When the change amount is not lower than this threshold, it indicates that the interaction operation has triggered a substantial interface state transition, thus triggering the information extraction process.

[0085] After meeting the threshold condition for change, the system extracts key triplet elements from the interface state change information: the first interface state, the executed action, and the second interface state. The first interface state is a snapshot of the interface before the control triggers an operation. It contains a complete record of multimodal data such as the interface image, control hierarchy, and text content before the trigger, reflecting the interface state at the start of the interaction. The executed action refers to the specific interactive operation triggered for the target aggregate control, such as automated instructions simulating user behavior like clicking, long-pressing, or swiping. This action is generated and executed through a preset execution strategy and is the direct cause of the interface state change. The second interface state is a snapshot of the interface after the control triggers an operation. It captures the interface performance after the operation is completed, including the updated image, control tree structure, and text content, reflecting the interface state at the end of the interaction. By extracting these elements, the system can accurately record the start state, triggering action, and result state of each effective interaction, laying the foundation for constructing a coherent interaction sequence.

[0086] Subsequently, the first interface state, the executed action, and the second interface state are linked together in chronological order to form an interaction behavior chain. Chronological order refers to the time sequence of events. The triplet elements are arranged according to the timestamp or step order of the operation execution, ensuring that the earlier occurring state and action are at the beginning of the chain, with later states and actions appended sequentially. Data chain construction refers to the process of associating multiple triplet elements through logical pointers or indices to form a linear, ordered data structure; for example, the second interface state in each triplet can serve as the first interface state of the next triplet, thus chaining multiple interaction steps. The final generated interaction behavior chain is a structured data sequence that completely records the entire path from the initial interface state, through a series of executed actions triggering continuous changes in the interface state. The interaction behavior chain not only includes static interface snapshots but also incorporates dynamic operation instructions and state transition relationships, enabling the multimodal large model to learn the causal relationship between operation intentions and interface feedback, improving the model's reasoning and prediction capabilities in complex tasks.

[0087] Specifically, the construction of the interaction behavior chain can also be implemented in, but is not limited to, the following ways: An interaction chain construction determination function, if the interface state change satisfies:

[0088] Then Add to the interaction chain:

[0089] in, The interface change detection function can be based on: visual difference (image feature distance); structural difference (control tree edit distance); and text difference (control text set difference).

[0090] This disclosure automates and intelligently constructs interactive behavior chains. A threshold-based judgment mechanism ensures that the chain contains only meaningful interaction steps, avoiding interference from invalid or redundant operations. Meanwhile, the construction of time-series data chains guarantees the logical coherence and integrity of the interaction sequence. This method significantly enhances the dynamic semantic representation capability of training data, providing rich contextual learning materials for multimodal large models. It supports models in gaining a deeper understanding of user operation processes and interface state evolution patterns, thereby achieving more accurate decision-making and responses in scenarios such as automated testing and intelligent assistants.

[0091] In one possible implementation of this disclosure, when integrating multimodal data, multiple aggregate controls, and interactive behavior chains to generate structured training data for model training, the following methods can be used, but are not limited to: performing data cleaning and standardization on the multimodal data, multiple aggregate controls, and interactive behavior chains to obtain processed multimodal data, processed multiple aggregate controls, and processed interactive behavior chains; establishing a three-layer association between the processed interface image, processed multiple aggregate controls, and processed interactive behavior chains in the processed multimodal data to obtain multimodal fusion data, wherein the three-layer association includes at least associating the corresponding aggregate controls and interactive behavior chains with the interface identifier of the processed interface image as the center; and generating structured training data conforming to a preset format based on the multimodal fusion data.

[0092] In the embodiments of this disclosure, data cleaning and standardization refer to a series of operations aimed at improving data quality, unifying data formats, and standardizing data values. For multimodal data, cleaning and standardization include size normalization of interface images, i.e., scaling all screenshots to a uniform preset resolution to ensure consistency of visual input; simultaneously, image format conversion is performed, for example, uniformly converting to Portable Network Graphics (PNG) format to maintain lossless quality, and performing quality checks to remove blurry, damaged, or invalid image files. For control structure information, field standardization is performed to ensure that attributes such as control type, bounding box coordinates, control identifier, and text content are represented using uniform key names and data types, for example, uniformly representing the bounding box as... The pixel coordinate format is specified. For the interaction behavior chain, cleaning and standardization focus on timestamp alignment and sequence sorting, that is, strictly arranging the interaction steps in chronological order according to the timestamp of the operation execution to ensure the correctness of the temporal logic of actions and state changes. Through the above processing, the system obtains processed multimodal data, processed multiple aggregate controls, and processed interaction behavior chains. These data have a consistent format, standardized structure, and reliable quality, laying the foundation for subsequent association and fusion.

[0093] After data cleaning and standardization, the system establishes a three-layer association between the processed interface images, multiple aggregated controls, and the processed interactive behavior chain in the processed multimodal data. This three-layer association is a hierarchical data linking mechanism centered on the processed interface images, logically binding related aggregated control sets and interactive behavior chain steps through their unique interface identifiers. Specifically, the first layer of association centers on the interface identifier of the processed interface image, matching each image with its corresponding aggregated control set. The interface identifier can be a unique hash value generated based on the page content (e.g., MD5 code) or a unique ID assigned by the system, which accurately identifies all aggregated controls within the interface image. The second layer of association links the aggregated controls with specific operation steps in the interactive behavior chain. For example, the operation of clicking a specific aggregated control is recorded in the interactive behavior chain, and this operation and its resulting state change are associated with that control. The third layer of association integrates the first two layers, forming a complete data graph of image-control-interaction, ensuring that each interface state not only includes its visual presentation and control composition but also the possible operations performed in that state and their resulting subsequent state changes. Through this three-layer association, multimodal fusion data is obtained, which achieves the organic unity of static interface information and dynamic interactive behavior, providing rich contextual information for the model.

[0094] Finally, structured training data conforming to a preset format is generated based on the multimodal fusion data. The preset format refers to a standardized data architecture designed for training large multimodal models, typically referencing and extending common dataset formats (such as the COCO dataset format), but adding support for elements specific to this solution, such as interactive behaviors and aggregation controls. During the generation process, all elements in the multimodal fusion data, including image paths, image sizes, fusion bounding box coordinates of aggregation controls, control types, associated text descriptions, action sequences in the interaction chain, and state transition relationships, are organized and serialized according to the field definitions of the preset format. The generated structured training data is a machine-readable file (such as JSON or JSONL format), with complete fields, a clear structure, and explicit semantics, which can be directly read and used for training by mainstream multimodal models without additional data cleaning or format conversion steps. This highly structured data output not only improves the convenience of data utilization but also ensures that training data generated from different batches and sources has consistent specifications and quality, greatly promoting large-scale, high-efficiency model training processes.

[0095] Specifically, the process of generating structured training data can be achieved in the following ways, but not limited to: standardizing, cleaning, associating, fusing, and converting unstructured / semi-structured data (multimodal raw samples, aggregated control sets, and interaction chain time-series information) to generate a structured dataset that meets the training requirements of multimodal large models.

[0096] Unstructured / semi-structured data specifically includes: The raw sample set output by the multimodal interface data acquisition module includes screenshot images. (Screenshots at different time steps t), control structure collection (Including basic information such as control type, bounding box, ID, and text), context information (timestamp) Page IDp, etc.

[0097] The collection of aggregated controls output by the image and text element recognition and combination module: Each of them Image and text fusion tags (including fusion bounding boxes) (Joint semantic description, original control association relationships, etc.)

[0098] Interaction chain output by the interaction behavior chain extension module: Each of them (Including the temporal relationship between the current state, action, and next state).

[0099] Data cleaning and standardization: Perform uniform preprocessing on the raw image data, including size normalization (e.g., scaling to a fixed resolution). ), format conversion (e.g., unify to PNG format), quality verification (remove blurry or damaged screenshots).

[0100] Normalize the fields of the control structure data: This involves classifying the control type... Bounding box , ,text Information such as fields should be converted to key-value pairs to ensure that field names and data types are consistent (e.g., bounding boxes should consistently use key-value pairs). (pixel coordinate format).

[0101] Timestamp alignment of interaction chain timing information: based on context information timestamps The actions will be sorted in chronological order to ensure the consistency of the timing between actions and state changes.

[0102] Establish a three-layer relationship between image, control, and interaction: Screenshot of the interface The ID is used as the core key to associate the collection of aggregate controls corresponding to the image. (Matching by page IDp); With interaction chain In This serves as a link, connecting the actions corresponding to each state. With the next state This forms a closed-loop relationship of "state-action-feedback".

[0103] Integrating semantic information from combined text and images: This involves combining aggregated controls. fusion bounding box Joint semantic description (based on spatial-semantic combination score generation) and original controls (icon), The basic information (text) is bound together to form a "visual-text" joint feature.

[0104] Refer to the preset format, such as the COCO dataset general format, and expand the multimodal and interaction dimension fields to generate the final training dataset.

[0105] The final training dataset must have at least the following characteristics: It covers multimodal information including images, structures, text, and interactions, meeting the needs of joint modeling of large multimodal models; Field standardization ensures compatibility with mainstream multimodal model input formats, allowing for direct use in training without additional cleaning. It contains temporal interaction chain information, supporting the model to learn the state transition rules of the interface and the user's operation intentions.

[0106] Through the data integration module, the fragmented data generated by the preceding module is transformed into high-quality, structured, and multi-dimensional training samples, effectively solving the problems of scattered formats, single modality, and lack of interactive semantics in existing technologies, and providing core training data support for automated testing of large models.

[0107] This disclosure transforms previously scattered and heterogeneous multi-source data into unified, standardized, and semantically rich structured training data through data integration and processing steps. This significantly improves the quality and usability of the training data, enabling multimodal large models to simultaneously utilize visual, textual, structural, and behavioral sequence information for joint learning. This allows for a deeper understanding of the complex semantics and interaction logic of the terminal interface, resulting in stronger perception and decision-making capabilities in scenarios such as automated testing and intelligent task execution.

[0108] In one possible implementation of this disclosure, the structured training data at least conforms to a common image annotation dataset format; Structured training data includes at least image fields, annotation fields, and interaction chain fields. The annotation fields include fused bounding boxes and joint semantic descriptions, and the interaction chain fields include state sequences, action sequences, and feedback sequences.

[0109] In the embodiments of this disclosure, the structured training data includes at least image fields, annotation fields, and interaction chain fields. These fields together constitute a multi-level, multi-modal data structure that fully covers static interface information and dynamic interactive behaviors. The image field refers to a database segment used to store basic information about the interface image, typically including metadata such as the image file's unique identifier, storage path, image width, and height. This information provides the model with a raw visual input reference. The annotation field refers to a database segment used to store annotation information for controls in the interface. It extends the traditional image annotation format, including not only basic object bounding boxes and category labels, but also, in particular, fused bounding boxes and joint semantic descriptions. The fused bounding box is a rectangular coordinate frame generated through a spatial-semantic fusion mechanism that covers the combined area of ​​icon and text controls. Its coordinate values ​​are expressed in pixels, defining the precise position and range of the aggregated control in the image. The joint semantic description is a textual description of the function or meaning of the control corresponding to the fused bounding box. It integrates the visual features of the icon and the text content to form a natural language statement, such as: "Click the settings button" or "Open the album icon," thereby providing the model with rich semantic context.

[0110] Furthermore, the structured training data includes an interaction chain field, a significant extension to the general image annotation dataset format, used to capture and store dynamic interaction sequence information. The interaction chain field is a database segment used to record the temporal relationship between user actions and interface state changes, including state sequences, action sequences, and feedback sequences. A state sequence is a collection of interface state snapshots arranged chronologically; each snapshot corresponds to an interface image, control structure, and contextual information at a specific point in time, reflecting the interface evolution trajectory during the interaction process. An action sequence is a collection of user action records corresponding to the state sequences, such as clicks and swipes. Each action is associated with a specific state in the state sequence, representing the operation performed in that state. A feedback sequence is a collection of interface change information triggered by the action, such as a new interface state after a jump or content update. It describes the direct consequences of the action, forming a closed-loop logic of state-action-feedback. By integrating state sequences, action sequences, and feedback sequences, the interaction chain field provides the model with a complete interaction context, supporting the model in learning the causal relationship between operational intentions and dynamic interface responses.

[0111] This disclosure, through its format definition, generates structured training data that not only retains the standardization and compatibility of general image annotation datasets but also enhances the semantic richness and dynamic interaction capabilities of the data by introducing fused bounding boxes, joint semantic descriptions, and interaction chain fields. This enables the data to simultaneously meet the joint modeling needs of multimodal models for visual, textual, structural, and behavioral sequences, improving the model's accuracy in understanding complex interfaces and executing automated tasks.

[0112] In one possible implementation of this disclosure, the collection of multimodal data from the terminal interface can be achieved in the following ways, but is not limited to: collecting multimodal data from the terminal interface through a terminal control interface, wherein the terminal control interface is a remote control interface adapted to the terminal's operating system, used to realize navigation control and data collection of the terminal interface.

[0113] In the embodiments of this disclosure, the terminal control interface refers to a software or hardware-level communication bridge used to establish a connection with the terminal device and transmit control commands and data. Its core function is to enable remote operation and status monitoring of the terminal device. This terminal control interface is a remote control interface adapted to the terminal's operating system. This means that the interface's design and implementation are specifically optimized and configured for compatibility with the operating system (e.g., Android, iOS, HarmonyOS) running on the terminal device, ensuring stable operation and consistent functional support in different system environments. For example, for Android systems, this interface may be implemented based on the ADB (Android Debug Bridge) protocol; for iOS systems, it may be built using the XCUITest framework or similar technologies. The remote control interface allows external systems (such as computers or servers) to communicate with the terminal device via wired or wireless means, performing complex operations without physical contact with the device, thereby supporting large-scale, distributed data acquisition tasks.

[0114] Navigation control refers to the process of simulating user interaction through programmed instructions to drive state transitions on the terminal interface, such as automatically launching applications, navigating to different functional pages, scrolling screen content, or triggering specific menus. These operations are executed based on preset scripts or algorithm sequences, ensuring coverage of the critical paths and core functional areas of the terminal interface, laying the foundation for comprehensive data collection. Data collection refers to the synchronous acquisition of multimodal data from the terminal interface during navigation control, including interface images, control structure information, and contextual information. Interface images are captured by calling the system's screenshot function and saved in pixel matrix format; control structure information is obtained by parsing the interface's hierarchical files (such as UI description trees in XML or JSON format), including the type, position, identifier, and text attributes of controls; contextual information records environmental parameters related to the collection process, such as timestamps, page hash identifiers, and device status. All this data is collected and temporarily stored in real time, forming a structured multimodal data set.

[0115] This disclosure automates the entire multimodal data acquisition process through the use of a terminal control interface. It not only eliminates the uncertainty and inefficiency of manual operation but also ensures data consistency in format, quality, and timing, adapting to the needs of large-scale, multi-terminal testing scenarios. Furthermore, due to the remote nature of the interface, the data acquisition process does not interfere with the normal use of terminal devices, improving the practicality and deployment flexibility of the solution. The acquired multimodal data provides solid support for subsequent aggregation control generation, interactive behavior chain construction, and data integration, ultimately contributing to the generation of high-quality training data.

[0116] Corresponding to the training data generation method described above, this invention also proposes a training data generation apparatus. Since the apparatus embodiments of this invention correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0117] Figure 3 This is a schematic diagram of the structure of a training data generation device provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, it includes: The acquisition unit 31 is used to acquire multimodal data of the terminal interface, wherein the multimodal data includes interface images, control structure information and context information; The calculation unit 32 is used to filter the control structure information to obtain multiple candidate icon controls and multiple candidate text controls, and calculate the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control. The first generation unit 33 is used to generate multiple aggregate controls based on the association score; The acquisition unit 34 is used to perform interactive operations based on the target aggregation control and acquire interface state change information during the interactive operation process; Construction unit 35 is used to construct an interactive behavior chain based on interface state change information, wherein the target aggregation control is any control among multiple aggregation controls; The second generation unit 36 ​​is used to integrate and process multimodal data, multiple aggregation controls and interactive behavior chains to generate structured training data for model training.

[0118] Furthermore, in one possible implementation of this embodiment, the computing unit 32 is further configured to: Calculate the first spatial similarity between the target icon control and the target text control, and continue to calculate the similarity between each candidate icon control and each candidate text control to obtain the spatial similarity. Here, the target icon control is any one of the multiple candidate icon controls, and the target text control is any one of the multiple candidate text controls. Based on a preset multimodal model, the target icon control and the target text control are jointly encoded to obtain the first semantic relevance. This process continues until the relevance between each candidate icon control and each candidate text control is calculated to obtain the semantic relevance.

[0119] Furthermore, in one possible implementation of this disclosure embodiment, the computing unit 32 is specifically used for: Based on the first spatial similarity and the first semantic relevance, a weighted calculation is performed using preset weights to obtain the first association score between the target icon control and the target text control. The correlation score is obtained after calculating the score between each candidate icon control and each candidate text control.

[0120] Furthermore, in one possible implementation of this disclosure embodiment, the computing unit 32 is specifically used for: Convert multiple candidate icon controls and multiple candidate text controls to a preset coordinate system, and obtain the coordinates of the first control center point of the target icon control and the coordinates of the second control center point of the target text control; The Euclidean distance is calculated based on the coordinates of the center point of the first control and the center coordinates of the second control, and the Euclidean distance is used as the first spatial similarity.

[0121] Furthermore, in one possible implementation of this disclosure embodiment, the computing unit 32 is specifically used for: Extract image slices and text content from the target icon control and the target text control; Based on a pre-defined multimodal model, feature fusion is performed on image slices and text content to obtain joint feature vectors corresponding to each image slice and text content. The cosine similarity of the joint feature vectors is calculated, and the cosine similarity is normalized by a preset activation function to obtain the first semantic relevance.

[0122] Furthermore, in one possible implementation of this disclosure embodiment, the first generation unit 33 is specifically used for: In response to a correlation score that is not less than a preset score threshold, the candidate icon control and candidate text control corresponding to the correlation score are determined as a graphic-text combination control. A merged bounding box is generated based on each graphic and text combination control, resulting in multiple aggregated controls.

[0123] Furthermore, in one possible implementation of this disclosure embodiment, the acquisition unit 34 is specifically used for: Select the target aggregation control from among the multiple aggregation controls on the terminal interface; Automatically execute control triggering operations according to a preset execution strategy and capture the interface state during the control triggering process. The preset execution strategy includes at least one of sequential traversal, weight priority, and random exploration. The interface state is detected and processed based on a preset interface change detection function to obtain interface state change information. The interface change detection function includes at least one of the following: visual feature distance of interface image, edit distance of control tree structure, and difference degree of control text set.

[0124] Furthermore, in one possible implementation of this disclosure embodiment, the construction unit 35 is specifically used for: In response to the interface state change information indicating that the change between the first interface state and the second interface state is not less than a preset change threshold, the first interface state, the action to be executed, and the second interface state in the interface state change information are extracted. The first interface state is the interface state before the control is triggered, the action to be executed is the interactive operation to be executed after the control is triggered, and the second interface state is the interface state after the control is triggered. The interaction behavior chain is obtained by constructing a data chain based on the first interface state, the executed action, and the second interface state in sequence.

[0125] Furthermore, in one possible implementation of this disclosure embodiment, the second generation unit 36 ​​is specifically used for: Data cleaning and standardization are performed on multimodal data, multiple aggregate controls, and interaction behavior chains to obtain processed multimodal data, processed multiple aggregate controls, and processed interaction behavior chains. Establish a three-layer association between the processed interface image, multiple processed aggregate controls, and the processed interactive behavior chain in the processed multimodal data to obtain multimodal fusion data. The three-layer association includes at least the association of the corresponding aggregate controls and interactive behavior chain with the interface identifier of the processed interface image as the center. Structured training data conforming to a preset format is generated based on multimodal fusion data.

[0126] Furthermore, in one possible implementation of this disclosure embodiment, the structured training data at least conforms to a general image annotation dataset format; Structured training data includes at least image fields, annotation fields, and interaction chain fields. The annotation fields include fused bounding boxes and joint semantic descriptions, and the interaction chain fields include state sequences, action sequences, and feedback sequences.

[0127] Furthermore, in one possible implementation of this embodiment, the acquisition unit 31 is specifically used for: Multimodal data from the terminal interface is collected through the terminal control interface, which is a remote control interface adapted to the terminal's operating system and used to realize navigation control and data collection of the terminal interface.

[0128] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0129] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 402 or a computer program loaded from storage unit 408 into RAM (Random Access Memory) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O (Input / Output) interface 405 is also connected to bus 404.

[0132] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as training data generation methods. For example, in some embodiments, the training data generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned training data generation method by any other suitable means (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0139] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0140] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0141] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating training data, characterized in that, include: Collect multimodal data from the terminal interface, wherein the multimodal data includes interface images, control structure information, and context information; The control structure information is filtered to obtain multiple candidate icon controls and multiple candidate text controls. The association score is calculated based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control. Multiple aggregate controls are generated based on the association score. The interaction operation is performed based on the target aggregation control, the interface state change information during the interaction operation is obtained, and the interaction behavior chain is constructed based on the interface state change information, wherein the target aggregation control is any control among the plurality of aggregation controls; The multimodal data, the multiple aggregation controls, and the interaction behavior chain are integrated and processed to generate structured training data for model training. The step of integrating and processing the multimodal data, the multiple aggregation controls, and the interaction behavior chain to generate structured training data for model training includes: The multimodal data, the multiple aggregate controls, and the interaction behavior chain are cleaned and standardized to obtain processed multimodal data, processed multiple aggregate controls, and processed interaction behavior chain. A three-layer association is established between the processed interface image, the processed multiple aggregate controls, and the processed interactive behavior chain in the processed multimodal data to obtain multimodal fusion data. The three-layer association includes at least associating the corresponding aggregate controls and interactive behavior chain with the interface identifier of the processed interface image as the center. The structured training data conforming to a preset format is generated based on the multimodal fusion data.

2. The training data generation method according to claim 1, characterized in that, Before calculating the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control, the method further includes: Calculate the first spatial similarity between the target icon control and the target text control, until the similarity between each candidate icon control and each candidate text control is calculated, and obtain the spatial similarity, wherein the target icon control is any candidate icon control among a plurality of candidate icon controls, and the target text control is any candidate text control among a plurality of candidate text controls; The target icon control and the target text control are jointly encoded based on a preset multimodal model to obtain the first semantic relevance. This process continues until the relevance between each candidate icon control and each candidate text control is calculated to obtain the semantic relevance.

3. The training data generation method according to claim 2, characterized in that, The calculation of the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control includes: Based on the first spatial similarity and the first semantic relevance, a weighted calculation is performed using preset weights to obtain the first association score between the target icon control and the target text control. The correlation score is obtained after calculating the score between each candidate icon control and each candidate text control.

4. The training data generation method according to claim 2, characterized in that, The calculation of the first spatial similarity between the target icon control and the target text control includes: The candidate icon controls and candidate text controls are converted to a preset coordinate system, and the coordinates of the first control center point of the target icon control and the coordinates of the second control center point of the target text control are obtained. The Euclidean distance is calculated based on the coordinates of the center point of the first control and the center point of the second control, and the Euclidean distance is used as the first spatial similarity.

5. The training data generation method according to claim 2, characterized in that, The first semantic relevance obtained by jointly encoding the target icon control and the target text control based on a preset multimodal model includes: Extract image slices and text content from the target icon control and the target text control; The image slice and the text content are fused according to the preset multimodal model to obtain the joint feature vectors corresponding to the image slice and the text content respectively; The cosine similarity of the joint feature vectors is calculated, and the cosine similarity is normalized by a preset activation function to obtain the first semantic relevance.

6. The training data generation method according to claim 1, characterized in that, The generation of multiple aggregate controls based on the association score includes: In response to the association score being not less than a preset score threshold, the candidate icon control and candidate text control corresponding to the association score are determined as a graphic-text combination control; A merged bounding box is generated based on each of the graphic and text combination controls to obtain the plurality of aggregate controls.

7. The training data generation method according to claim 1, characterized in that, The step of performing interactive operations based on the target aggregation control and obtaining interface state change information during the interactive operation includes: Select the target aggregation control from the plurality of aggregation controls on the terminal interface; Automatically execute control triggering operations according to a preset execution strategy, and capture the interface state during the control triggering process. The preset execution strategy includes at least one of sequential traversal, weight priority, and random exploration. The interface state is detected and processed based on a preset interface change detection function to obtain interface state change information. The interface change detection function includes at least one of the following: visual feature distance of interface image, edit distance of control tree structure, and difference degree of control text set.

8. The training data generation method according to claim 7, characterized in that, The step of constructing the interaction behavior chain based on the interface state change information includes: In response to the interface state change information indicating that the change between the first interface state and the second interface state is not less than a preset change threshold, the first interface state, the action to be performed, and the second interface state in the interface state change information are extracted, wherein the first interface state is the interface state before the control is triggered, the action to be performed is the interactive operation to be performed after the control is triggered, and the second interface state is the interface state after the control is triggered. The first interface state, the executed action, and the second interface state are constructed into a data chain according to the time sequence to obtain the interaction behavior chain.

9. The training data generation method according to claim 1, characterized in that, The structured training data must at least conform to a common image annotation dataset format; The structured training data includes at least image fields, annotation fields, and interaction chain fields. The annotation fields include fused bounding boxes and joint semantic descriptions, and the interaction chain fields include state sequences, action sequences, and feedback sequences.

10. The training data generation method according to claim 1, characterized in that, The multimodal data collected by the terminal interface includes: The multimodal data of the terminal interface is collected through a terminal control interface, wherein the terminal control interface is a remote control interface adapted to the terminal's operating system, used to realize navigation control and data collection of the terminal interface.

11. A training data generation device, characterized in that, include: The acquisition unit is used to acquire multimodal data of the terminal interface, wherein the multimodal data includes interface images, control structure information and context information; The calculation unit is used to filter the control structure information to obtain multiple candidate icon controls and multiple candidate text controls, and calculate the association score based on the spatial similarity and semantic relevance between each candidate icon control and each candidate text control; The first generation unit is used to generate multiple aggregate controls based on the association score; The acquisition unit is used to perform interactive operations based on the target aggregation control and acquire information on changes in the interface state during the interactive operation. A construction unit is used to construct an interactive behavior chain based on the interface state change information, wherein the target aggregation control is any one of the plurality of aggregation controls; The second generation unit is used to integrate and process the multimodal data, the multiple aggregation controls, and the interactive behavior chain to generate structured training data for model training. The second generating unit is specifically used for: The multimodal data, the multiple aggregate controls, and the interaction behavior chain are cleaned and standardized to obtain processed multimodal data, processed multiple aggregate controls, and processed interaction behavior chain. A three-layer association is established between the processed interface image, the processed multiple aggregate controls, and the processed interactive behavior chain in the processed multimodal data to obtain multimodal fusion data. The three-layer association includes at least associating the corresponding aggregate controls and interactive behavior chain with the interface identifier of the processed interface image as the center. The structured training data conforming to a preset format is generated based on the multimodal fusion data.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images