Training and inference method based on speech-text-image multi-modal contrast learning

Abstract

The application is specifically a training and inference method based on voice-text-image multi-modal contrast learning, and relates to the technical field of artificial intelligence, comprising: constructing a triple of speech-text- GUI screenshot, processing element mask; generating positive and negative alignment pairs of instruction clauses and GUI elements; multi-encoder processing corresponding modal, Rewriter converting voice transcription text into structured sequence; optimizing alignment through global / local dual fusion, dual contrast learning and coordinate loss. In the application, the global and local dual-level fusion and contrast learning mechanism realizes fine-grained alignment of voice, text and GUI elements; with the help of SAM to generate element mask and post-processing optimization, combined with RoIAlign to extract accurate element-level visual features, laying a foundation for alignment; through linear projection to unify the dimensions of each modal, and then through the global fusion layer to realize coarse-grained semantic correlation, and through the local fusion layer to capture the fine correspondence of sub-steps and elements with cross-modal attention and gating mechanism, and to integrate the prosodic information of voice to enhance the correlation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training and inference method based on speech-text-image multimodal contrastive learning. Background Technology

[0002] Early automation of graphical user interfaces (GUIs) primarily relied on rule-based scripting and screen recording / playback mechanisms. These methods require the pre-definition of numerous operational rules or macros, and while they achieve basic automation, they struggle to adapt to the complexity and dynamic changes of modern software interfaces. For example, when the position or style of interface elements changes, script-based or recording-based solutions often fail to execute, exhibiting poor stability and adaptability.

[0003] With the development of computer vision and pattern recognition technologies, some research attempts to assist GUI automation through image recognition methods. These methods typically utilize object detection or pattern matching to identify interface elements, thereby achieving a certain degree of flexibility. However, since interface elements are often small, dense, and stylistically similar, recognition methods relying solely on visual features still have significant limitations when facing dynamic interfaces and complex contexts.

[0004] In recent years, multimodal methods have been gradually introduced into GUIAgent research, attempting to improve the system's interactivity and understanding capabilities by combining visual information with language commands. Existing methods mostly use text as the primary input, employing large-scale pre-trained image-text models to achieve interface element recognition and command execution. However, these methods have the following shortcomings: Text-based training bias: Most existing GUI agents use text commands as the primary training input and fine-tune based on a large-scale image-text pre-trained multimodal backbone. This training paradigm leads to a significant performance drop when processing speech input due to insufficient alignment between the speech and image modalities. Experimental results with UITron-Speech show that models relying solely on speech training perform poorly, while introducing a mixed modality strategy of 70% speech + 30% text during training can alleviate this bias problem to some extent.

[0005] Insufficient fine-grained positioning: In real GUI interfaces, controls are often small, densely distributed, and clustered with highly similar styles (such as toolbar icons). Traditional visual encoders are prone to subtle deviations when locating these "small elements," and even an error of only a few pixels can lead to click operation failure. To address this issue, UI-AGILE divides the interface into multiple small regions, predicts coordinates at the region level, and then selects the region with the highest matching degree with the instruction as the grounding result. UITron-Speech proposes a two-step scaling and cropping method, which crops the image from the original image with height and width equal to the original... Figure 1 The local region / k is magnified by a factor of k and re-input into the model to improve the localization accuracy of small targets. Although these methods alleviate the localization problem to some extent, a unified cross-modal fine alignment mechanism is still lacking.

[0006] In real-world user scenarios, voice input is gradually becoming a more human-friendly interaction mode due to its convenience and naturalness. Although voice interaction has seen initial applications in smart homes and in-vehicle systems, it still faces significant challenges when manipulating complex GUI interfaces, including the following: Cross-modal semantic alignment is challenging, and there is a lack of fine-grained association between speech and GUI visual elements. Existing methods generally face difficulties in achieving fine-grained semantic alignment when mapping speech commands to operable GUI elements. A common approach is to globally encode the entire screen and then match or retrieve the command (usually in text form) against this global representation. However, this coarse-grained global fusion fails to provide element-level positioning information, making it prone to errors when multiple similar controls or small targets exist. Recent GUI-specific research (such as ShowUI and work surrounding GUI visualization grounding) has further improved the selection of visual tokens and interaction history modeling, enhancing overall efficiency and multi-step interaction capabilities. However, these works mainly focus on "how to encode the screen more efficiently" and "how to better manage the visual-history," without fundamentally solving the problem of fine-grained correspondence between speech modalities and GUI elements simultaneously. Audio-visual joint modeling (such as Meerkat, and frameworks based on multimodal contrastive learning) has shown the potential to align sound and vision at a fine-grained level in "general" scenarios. However, most of these works focus on natural scenes (spatial-temporal alignment of objects with ambient sounds and events) rather than structured GUI screens. Their methods and datasets cannot be directly transferred to GUI tasks that require processing a large number of small, regularized UI elements. Therefore, there is still a significant gap when using speech directly for GUI grounding.

[0007] Using speech directly in end-to-end processing presents challenges: As a natural form of interaction, speech carries not only task-related semantic information but also non-linguistic factors such as tone and emotion. When processing complex instructions, this additional information doesn't effectively aid task execution but instead increases the difficulty of model parsing. In contrast, text possesses stronger structured expressive capabilities, clearly presenting task steps and conditional relationships through punctuation, clause breaks, and other methods. UITron-Speech demonstrates that in short-instruction scenarios, speech and text input perform comparably, but in long-instruction scenarios, speech's executability and robustness significantly decrease, exhibiting problems such as insufficient information granularity, difficulty in disambiguation, and incomplete structured expression.

[0008] Therefore, a training and inference method based on speech-text-image multimodal contrastive learning is proposed to address the aforementioned problems. Summary of the Invention

[0009] The purpose of this invention is to propose a training and inference method based on multimodal contrastive learning of speech, text, and image in order to solve the above problems.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: Training and inference methods based on speech-text-image multimodal contrastive learning include: Construct a text-GUI screenshot triplet, process element masks; generate positive and negative alignment pairs between instruction clauses and GUI elements; Multiple encoders process corresponding modalities; the Rewriter converts speech to text into structured sequences; alignment is optimized through global / local dual fusion, dual contrastive learning, and coordinate loss; the decoder locates elements and predicts action sequences. In the initial stage of training, text samples were used as the main focus, and the proportion of speech samples was gradually increased; training was conducted in layers according to the length of the instructions.

[0011] Preferably, the construction of the phrase-text-GUI screenshot triple, processing element masks, and generating positive and negative alignment pairs between instruction clauses and GUI elements specifically includes: Acquire screenshots and related metadata; run SAM on each screenshot to obtain a set of masks, calculate and retain the attributes of each mask; For each ground truth bounding box in the original dataset Find it Largest mask Generate multiple speaker, multiple accent, multiple speech rate, and multiple noise versions for each text instruction, and perform post-processing; Design a real-world voice capture task. After obtaining the screenshot mask and OCR text, further align the instructions and elements; automatically generate high-confidence region pairs for each instruction.

[0012] Preferably, the multi-encoder processes the corresponding modality, and the Rewriter converts speech-to-text into a structured sequence; alignment is optimized through global / local dual fusion, dual contrastive learning, and coordinate loss; the decoder locates elements and predicts action sequences, specifically including: The encoders include: speech encoder, rewriter, text encoder, image encoder, and region pooling; A speech encoder that performs both ASR tasks and speech representation learning simultaneously; Rewriter converts raw text into structured text. ; A text encoder converts rewritten text into a fixed-size semantic vector and sub-steps; The image encoder and region pooling divide the screenshot into patches of a preset size, and then encode them to obtain a patch embedding sequence.

[0013] Preferably, the method further includes multimodal fusion: To ensure dimensional consistency across different modal features, a linear projection is first performed on the outputs of each modality to unify their projection onto a single dimension. ; Speech frame-level sequence Through linear layer Projection as Preserve rhythmic information; Global speech embedding a is also projected accordingly to obtain ; Text substep embedding sequence ,in The number of sub-steps, through a linear layer Projection as Global text embedding Also, perform the corresponding projection to obtain ; Image patch sequence ,pass get ; Similarly, element-level features Similarly, the projection is .

[0014] Preferably, the method further includes cross-modal attention fusion: A Transformer-based multimodal cross-modal fusion module, including a global fusion layer and a local fusion layer, is adopted to capture complementary information between modes; Global Fusion Layer: Embeds global speech Global text embedding With image patch sequence spliced ​​into a joint sequence ; Input a single-layer Transformer encoder and output the fused global representation. The output yields the final global multimodal embedding. , used for coarse-grained semantic alignment; Local Blending Layer: For fine-grained element positioning, it merges the text sub-step sequence. With element-level visual features Paired input cross-modal attention module; For each sub-step m, construct query-key-value pairs: As a query As keys and values, calculate attention weights: ;in, The projection matrix; Fusion characteristics are ,in ; To incorporate rhythmic information, global speech is embedded. and ; Fusion through gating mechanism ,in This is element-wise multiplication; Activated for Sigmoid; The final local fusion sequence is obtained. Used for element-level alignment and positioning.

[0015] Preferably, the method further includes two-level contrastive learning: Design a two-level comparative learning task that combines global and local perspectives: Global comparative learning: in batch Within this context, calculate the similarity matrix between the global multimodal embedding and the corresponding speech-text-image triple; Local contrastive learning: For each sub-step and element, use local fusion features to construct an element and similarity matrix, and calculate the loss and total local loss; In addition, to improve positioning accuracy, coordinate regression auxiliary loss is introduced: the L1 loss between the predicted element center coordinates and the true bounding box is added to the total loss.

[0016] Preferably, the method further includes decoding the Action decision: Element decoding and localization: Using a locally fused sequence, a lightweight Transformer decoder is input, and a cross-attention mechanism is used to further refine the matching of sub-steps and elements; the output is the element probability distribution for each sub-step; the element with the highest probability is selected, and the coordinates are combined with the coordinate regression head to predict and refine the coordinates to correct fine-grained biases; Action type and sequence decision: For each location element, an action classification head is attached.

[0017] Preferably, the initial training phase primarily uses text samples, gradually increasing the proportion of speech samples; the training is stratified according to instruction length, specifically including: Mixed modality sampling: In the initial training phase, text samples are the main focus to ensure the stability of the model in the semantic space; as the training progresses, the proportion of speech samples is gradually increased to improve the alignment ability of speech modalities and to alleviate the bias caused by image-text pre-training.

[0018] Preferably, the method further includes: Length-sensitive learning: Training samples are stratified according to instruction length. Short instructions are trained first to learn basic actions and short sentences, then medium and long instructions are introduced, and finally mixed training is performed.

[0019] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention achieves fine-grained alignment of speech, text, and GUI elements through a global and local dual-level fusion and contrastive learning mechanism; it uses SAM to generate element masks and optimizes them through post-processing, and combines RoIAlign to extract accurate element-level visual features, laying the foundation for alignment; it unifies various modal dimensions through linear projection, and then achieves coarse-grained semantic association through a global fusion layer, while the local fusion layer captures the fine correspondence between sub-steps and elements with cross-modal attention and gating mechanisms, and incorporates speech prosodic information to enhance the relevance.

[0020] 2. This invention significantly improves the reliability of voice-driven GUI interaction through multi-dimensional optimization. On the one hand, it adopts a dataset construction strategy that combines synthetic speech and real speech to cover diverse speech variants and alleviate the problems of data bias and generalization. On the other hand, it introduces a Rewriter module to rewrite the raw_text transcribed from speech into a structured atomic operation sequence, correcting colloquial expressions, decomposing complex instructions, and solving the defect of speech lacking structured labeling. At the same time, the hybrid modality sampling strategy and length-sensitive course learning enable the model to steadily master the association rules of speech-text-image, effectively capturing reasonable information of non-linguistic factors in speech and filtering interference. Attached Figure Description

[0021] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0023] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0024] Example 1 Its specific implementation method is combined with the appendix Figure 1 A detailed explanation will be provided.

[0025] Appendix Figure 1 The flowchart of the training and inference method based on speech-text-image multimodal contrastive learning provided in the embodiments of the present invention shows the complete steps from constructing speech-text-GUI screenshot triples to hierarchical training according to instruction length.

[0026] In this embodiment, it includes: Construct a text-GUI screenshot triplet, process element masks; generate positive and negative alignment pairs between instruction clauses and GUI elements; Specifically, it includes: Use the existing GUIimage-text dataset to obtain screenshots and related metadata, including screenshots, text commands, action / bbox / DOM tags; run SAM on each screenshot to obtain a set of masks. Each mask is a binary mask or a polygonal outline; the mask set is post-processed to merge masks with high overlap (IoU>0.8) or containment relationships to avoid redundant subdivision; Filter out tiny masks and save them separately as tiny-maskpools, which can be used for noise processing; calculate and preserve the properties of each mask; For each ground truth bounding box in the original dataset Find it Largest mask If multiple bounding boxes are mapped to the same mask, the mapping relationship is saved. Once the mapping is successful, the mask's OCR_Text will be filled. The mapping result, along with the IoU value, is written to the metadata. The strategy of synthesized speech and real speech collection generates multiple speakers, accents, speech rates and noise levels for each text instruction, covering diverse variations to alleviate data scarcity and bias. To make the synthesized speech more realistic, it is post-processed to approximate the real scene by adding background music for multi-person conversations, street noise, and environmental sounds (such as wind and rain).

[0027] Real-world speech acquisition is crucial; if only synthesized speech is used, the model cannot generalize to real-world scenarios. The real-world speech acquisition task is designed to be divided into instruction reading and natural speech. Instruction reading involves the speaker directly reading the instructions; natural speech involves the speaker expressing the desired operation naturally, including necessary pauses, interjections, and colloquialisms.

[0028] After obtaining the screenshot mask and OCR text, the instructions are further aligned with the elements; A hierarchical priority strategy is adopted to automatically generate high-confidence region pairs for each instruction (or its clauses); The specific process is as follows: First, find substrings in the original instruction that are identical to the regionOCR text (ignoring case and spaces). If a match is found, a mapping is directly established. Then, vectorize the instruction and the ocr_text, compare their similarity, and if the similarity exceeds a certain threshold, a mapping is established. Positive example pair generation: If the target word or phrase of the instruction can match the corresponding element mask, a set of clause-element positive example pairs is obtained. ,in, For instruction clauses, For the corresponding element region; To improve the model's discriminative ability, negative samples need to be constructed. Hard negative sample generation involves selecting elements visually or textually similar to the target element from the same or adjacent screens, such as buttons with the same icon style, text with similar labels, or elements with adjacent coordinates. In-batch negative samples involve treating other samples within a mini-batch as negative examples for comparative learning.

[0029] Multiple encoders process corresponding modalities; the Rewriter converts speech to text into structured sequences; alignment is optimized through global / local dual fusion, dual contrastive learning, and coordinate loss; the decoder locates elements and predicts action sequences. Specifically, it includes: The encoders include: speech encoder, rewriter, text encoder, image encoder, and region pooling; The speech encoder needs to perform both ASR (Automatic Speech Representation) tasks and speech representation learning simultaneously. Using the Belle-whisper-large-v3-zh model, the input is raw audio, and the output is a rich, context-aware sequence of hidden states. ,in It is the number of time steps. It is the feature dimension. In ASR tasks, it is... For key-value pairs used for cross-attention, tokens for transcribed text are generated, ultimately yielding the raw text of the transcribed text. In speech extraction tasks, Each vector in the sequence contains rich information about its corresponding time point and its context. This is used as a frame-level sequence feature. For frame-level sequences Average pooling is used to obtain the global speech embedding 'a'; Rewriter, using the flan-t5-base model, converts the raw text above into structured text. First, standardize the format and correct colloquial / incorrect words, rewriting vague instructions into precise, structured action descriptions. Second, decompose complex instructions into sequences of atomic operations. ; The text encoder, using the Sentence-T5 model, converts the rewritten text into fixed-size semantic vectors. Kyoko Steps ; Image encoder and region pooling divide the screenshot into patches of a preset size (16x16), which are then encoded using CLIP-ViT-B / 16 to obtain a patch embedding sequence. This represents the visual representation of the screenshot; a mask for each element. Executing RolAlign yields an element-level embedding set. , used to align with text substeps.

[0030] It also includes multimodal fusion: To achieve fine-grained alignment between speech, text, and GUI image elements, a multimodal fusion contrastive learning framework with global and local dual-level layers is adopted. The framework first performs feature fusion through a cross-modal attention mechanism, dimensionally aligning and interactively fusing the prosodic / temporal information of speech (frame sequence), the structured semantics of text (sub-step embedding), and the global patch sequence and element-level visual features of images; Subsequently, a two-level contrastive learning task is introduced, including global contrastive learning and local contrastive learning, to narrow the representation distance of positive sample pairs (matched modal pairs) and widen the distance of negative sample pairs, thereby improving cross-modal semantic consistency and fine-grained localization accuracy of GUI elements. The specific implementation steps are as follows: To ensure dimensional consistency across different modal features, a linear projection is first performed on the outputs of each modality to unify their projection onto a single dimension. ; The formula is as follows: ; Speech frame-level sequence Through linear layer Projection as Preserve rhythmic information; Global speech embedding a is also projected accordingly to obtain ; Text substep embedding sequence ,in The number of sub-steps, through a linear layer Projection as Global text embedding Also, perform the corresponding projection to obtain ; Image patch sequence ,pass get ; Similarly, element-level features Similarly, the projection is .

[0031] Cross-modal attention fusion: A Transformer-based multimodal cross-modal fusion module, including a global fusion layer and a local fusion layer, is adopted to capture complementary information between modes; Global Fusion Layer: Embeds global speech Global text embedding With image patch sequence spliced ​​into a joint sequence ; Input a single-layer Transformer encoder (including multi-head attention mechanism, h=8 heads), output the fused global representation. Subsequently, the final global multimodal embedding is obtained through the output of the [CLS]token (pre-placed at the beginning of the sequence). , used for coarse-grained semantic alignment; Local Blending Layer: For fine-grained element positioning, it merges the text sub-step sequence. With element-level visual features Paired input cross-modal attention module; For each sub-step m, construct query-key-value pairs: As a query As keys and values, calculate attention weights: ;in, The projection matrix; Fusion characteristics are ,in ; To incorporate rhythmic information, global speech is embedded. and ; Fusion through gating mechanism ,in This is element-wise multiplication; Activated for Sigmoid; The final local fusion sequence is obtained. Used for element-level alignment and positioning.

[0032] Two-level comparative learning: Design a two-level contrastive learning task with global and local levels, using the InfoNCE loss function; Positive and negative samples are constructed based on local alignment pairs: positive samples are matched modal pairs (instruction clause - corresponding element), and negative samples include hard negative samples (similar but mismatched elements) and intra-batch negative samples (other samples within the mini-batch). The total loss is ,in =0.5; Global comparative learning: in batch (batch size) ), compute global multimodal embedding Similarity matrix with corresponding speech-text-image triples ,in The loss function is: This loss narrows down the global representation of the matching triples, enhancing overall semantic consistency; Local comparative learning: for each sub-step and elements Using local fusion features and Construct an element and similarity matrix For positive samples The loss is: Total local loss ; In addition, to improve positioning accuracy, coordinate regression auxiliary loss is introduced: predicting the center coordinates of the elements. L1 loss compared to the true bounding box and add to the total loss ,in These are the cross-entropy losses for ASR and rewriting tasks, respectively.

[0033] It also includes decoding Action decisions: Based on the fused multimodal representation, a hierarchical decoder is used to generate GUI action decisions, including element localization and action type prediction, supporting end-to-end reasoning; Element decoding and localization: using local fusion sequences The input is a lightweight Transformer decoder (2 layers, 4 heads), which further refines the matching of sub-steps and elements using a cross-attention mechanism; the output is the element probability distribution for each sub-step. ,in The element with the highest probability of being elected. It also combines a coordinate regression head (MLP: two fully connected layers, ReLU activation) to predict refined coordinates. To correct for fine-grained deviations; Action type and sequence decision: For each localization element, an action classification head (MLP: input fusion feature) is attached. Output action categories such as "click", "swipe", "input text", etc. For multi-step instructions, the decoder generates an action sequence. The sequence probability is optimized by beamsearch (beamwidth=3) to ensure that the operation order conforms to the rewritten structured text.

[0034] Initially, text samples were used as the primary training samples, with the proportion of speech samples gradually increased; training was stratified according to the length of instructions. Specifically, it includes: Mixed modality sampling: In the initial training phase, text samples (image-text pairs) are the main focus to ensure the stability of the model in the semantic space; as the training progresses, the proportion of speech samples is gradually increased to improve the alignment ability of speech modalities and to alleviate the bias (text bias) caused by image-text pre-training.

[0035] Length-sensitive learning: Training samples are stratified according to instruction length (short / medium / long). Short instructions are trained first to learn basic actions and alignment with short sentences, then medium and long instructions are introduced, and finally mixed training is performed.

[0036] Example 2: Training and Inference of SAMCP Model Based on Android GUI Dataset In this embodiment, it includes: This embodiment uses a GUI automation task on the Android platform as an example to describe in detail the complete implementation process of the method of the present invention. The training dataset uses the publicly available Rico dataset (containing approximately 72K screenshots, corresponding text instructions, and bounding box labels), expanded to approximately 10K speech-text-image triples. The entire process is divided into three stages: data preparation, model training, and inference execution.

[0037] Phase 1: Data Preparation and Preprocessing The training dataset was collected and constructed by extracting screenshots, text instructions (such as "click the search button"), and metadata (action / bbox / DOM tags) from the Rico dataset. For each screenshot, SegmentAnythingModel (SAM) was run to generate a set of binary masks (each mask corresponding to a potential UI element, such as a button or text box). Post-processing steps included: merging overlapping masks (IoU > 0.8); filtering out tiny masks with an area < 1% of the screen and storing them in a tiny-maskpool for noise enhancement; and calculating the attributes of each mask (center coordinates, size, OCR text). For each ground truth bounding box, the mask with the largest IoU was matched, and OCR text was added to the metadata, generating approximately 5K high-confidence element masks.

[0038] Synthesis and Real Speech Acquisition: For each text instruction, multiple speech variations (10 speakers, 5 accents, 3 speech rates) were generated using a TTS tool (Microsoft Azure TTS). Noise was added in post-processing (background music, street sounds, signal-to-noise ratio). Simultaneously, real speech was acquired: 20 speakers were recruited to perform instruction reading (direct reading) and natural speaking tasks ("Um, I want to click that blue search icon"). 3-5 variations were recorded for each instruction, totaling 30K audio tracks with a total duration of approximately 50 hours. The audio sampling rate was uniformly set to 16kHz.

[0039] Local alignment pair construction: String matching (ignoring case and spaces) is performed on the instruction clause (sentence segmented using NLP tools) and OCR text; if no exact match is found, Sentence-BERT is used to calculate similarity (threshold 0.7). The clause "Click to search" matches the search button mask, forming {clause, element region} pairs. Negative examples: Hard negative samples are selected from similar elements on the same screen (adjacent icons, visual cosine similarity > 0.8); intra-batch negative samples are randomly drawn from mini-batch. Finally, a set of 10K positive and negative pairs is constructed for comparative learning.

[0040] Phase Two: Model Training Model structure initialization: The speech encoder uses Belle-whisper-large-v3-zh, taking the original audio as input and outputting a sequence of hidden states. (T≈300), the ASR task generates raw_text, and the global embedding is obtained by frame-level average pooling. The rewriter uses flan-t5-base to rewrite the raw text into a structured sequence, outputting M=3-5 sub-steps. The text encoder Sentence-T5 converts the rewritten text into sub-step embeddings. The image encoder CLIP-ViT-B / 16 divides the screenshot (width W, height H) into 16x16 patches, outputting a sequence. (N=(W / 16)×(H / 16)); Perform RoIAlign on each mask to extract element-level features. (K≈20 / screenshot).

[0041] Multimodal fusion and contrastive learning: Dimensional projection to D=768. Global fusion: stitching. Input a single-layer Transformer (8 heads), [CLS] output is Local fusion: Text sub-steps are used as queries, element features are used as keys / values, and attention weights are calculated. , fusion Then gating integration have to Comparative learning: Batch size 32, InfoNCE calculates global loss. and local loss Add coordinate L1 loss Total loss Adam optimizer (lr=1e-4), trained for 20 epochs.

[0042] Phase Three: Reasoning and Execution Input processing: Voice commands are acquired in real time, processed by ASR, Rewriter, and text encoding to obtain a sequence of sub-steps. Screenshots are processed by an image encoder to obtain patch / element features.

[0043] Fusion and Decoding: Global / Local Fusion and Element localization: Input a 2-layer Transformer decoder (4 heads), output probability. ,select MLP regression coordinates Action decision: The classification head predicts the type, and beamsearch (width=3) generates the sequence. .

[0044] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0045] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0046] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0047] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0048] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0049] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0050] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0051] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0052] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0053] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A training and inference method based on speech-text-image multimodal contrastive learning, characterized in that, include: Constructing a speech-text-GUI screenshot triple and processing element masks, specifically including: Obtain screenshots and related metadata. The original dataset is an existing GUI image-text dataset, which includes screenshots, text commands, actions / bounding boxes / DOM tags. Run SAM on each screenshot to obtain a set of masks, and calculate and retain the attributes of each mask. For each ground truth bounding box in the original dataset Find it Largest mask ; Generate positive and negative alignment pairs between instruction clauses and GUI elements, specifically including: Generate multiple speaker, multiple accent, multiple speech rate, and multiple noise versions for each text instruction, and perform post-processing; Design a real-world voice capture task. After obtaining the screenshot mask and OCR text, further align the instructions with the elements; automatically generate high-confidence region pairs for each instruction. The design incorporates two levels of contrastive learning tasks: global contrastive learning and local contrastive learning. The L1 loss between the predicted element center coordinates and the true bounding box is introduced as an auxiliary loss for coordinate regression to improve localization accuracy. Multiple encoders process corresponding modalities; the Rewriter converts speech-to-text into structured sequences; alignment is optimized through global / local dual fusion, two-level contrastive learning, and coordinate loss; action decisions are decoded, and action sequences are predicted, specifically including: The encoders include: speech encoder, rewriter, text encoder, image encoder, and region pooling; The speech encoder simultaneously performs automatic speech recognition (ASR) tasks and speech representation learning, with the ASR task outputting raw_text transcribed from the original audio. Rewriter converts raw text into structured text. ; A text encoder converts rewritten text into a fixed-size semantic vector and sub-steps; The image encoder and region pooling divide the screenshot into image patches of a preset size, and then encode them to obtain an image patch embedding sequence. It also includes multimodal fusion: To ensure dimensional consistency across different modal features, a linear projection is first performed on the outputs of each modality to unify their projection onto a single dimension. ; Speech frame-level sequence Through linear layer Projection as Preserve rhythmic information; Average pooling is performed on the speech frame-level sequence to obtain the global speech embedding 'a'. The global speech embedding 'a' is also projected accordingly to obtain... ; Text substep embedding sequence ,in The number of sub-steps, through a linear layer Projection as Global text embedding Also, perform the corresponding projection to obtain ; Image patch sequence ,pass get ; Similarly, element-level features Similarly, the projection is ; Where T is the number of time steps in the speech frame-level sequence. For speech feature dimension, For text feature dimensions, For image feature dimensions, To unify projection dimensions; It also includes cross-modal attention fusion: A Transformer-based multimodal cross-modal fusion module, including a global fusion layer and a local fusion layer, is adopted to capture complementary information between modes; Global Fusion Layer: Embeds global speech Global text embedding With image patch sequence spliced ​​into a joint sequence ; Input a single-layer Transformer encoder and output the fused global representation. The output yields the final global multimodal embedding. , used for coarse-grained semantic alignment; Local Blending Layer: For fine-tuning element positioning, it merges the text sub-step sequence. With element-level visual features Paired input cross-modal attention module; For each sub-step m, construct query-key-value pairs: As a query As keys and values, calculate attention weights: ;in, The projection matrix; Fusion characteristics are ,in ; For the number of attention heads; To incorporate rhythmic information, global speech is embedded. and Integration; Fusion through gating mechanism ,in This is element-wise multiplication; Activated for Sigmoid; The final local fusion sequence is obtained. , used for element-level alignment and positioning; It also includes decoding Action decisions: Element decoding and localization: Using a locally fused sequence, a lightweight Transformer decoder is input, and a cross-attention mechanism is used to further refine the matching of sub-steps and elements; the output is the element probability distribution for each sub-step; the element with the highest probability is selected, and the coordinates are combined with the coordinate regression head to predict and refine the coordinates to correct fine-grained biases; Action type and sequence decision: For each localized element, an action classification head is attached; Initially, training primarily uses text samples, gradually increasing the proportion of speech samples; training is stratified according to instruction length, specifically including: Mixed modality sampling: In the initial training phase, text samples are the main focus to ensure the stability of the model in the semantic space; as the training progresses, the proportion of speech samples is gradually increased to improve the alignment ability of speech modalities and to alleviate the bias caused by image-text pre-training. Also includes: Length-sensitive learning: Training samples are stratified according to instruction length. Short instructions are trained first to learn basic actions and short sentences, then medium and long instructions are introduced, and finally mixed training is performed.

2. The training and inference method based on speech-text-image multimodal contrastive learning according to claim 1, characterized in that, It also includes two-level contrastive learning: Design a two-level comparative learning task that combines global and local perspectives: Global comparative learning: in batch Within this context, calculate the similarity matrix between the global multimodal embedding and the corresponding speech-text-image triple; Local contrastive learning: For each sub-step and element, use local fusion features to construct an element and similarity matrix, and calculate the loss and total local loss.

Images