Sample processing method, apparatus, device, medium, and program product

Abstract

Embodiments of the present application disclose a sample processing method, device, equipment, medium and program product. The method comprises: acquiring a plurality of interactive interfaces; adjusting the resolution of the plurality of interactive interfaces according to the time sequence information of each interactive interface to obtain adjusted interactive interfaces; wherein the resolution of at least two adjusted interactive interfaces is different; and extracting sample features from the adjusted interactive interfaces, wherein the sample features are used for training a visual language model.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of data processing technology, and in particular to a sample processing method, apparatus, device, medium, and program product. Background Technology

[0002] In modern human-computer interaction systems, visual language models are widely used to understand the relationship between the user interface and user behavior. These models typically require training using a sequence of historical and current UI screenshots to capture task context and generate reasonable responses or decisions. In related technologies, to enable the model to effectively process this image information, all UI screenshots are usually uniformly adjusted to a fixed resolution, then segmented into fixed-size image blocks, converted into visual tokens by a visual encoder, and finally concatenated with text tokens before being input into a language model for processing.

[0003] However, using this method, which applies the same processing precision to all image information in the time series, results in a large number of visual tokens, causing the input sequence for model training to be too long, increasing computational resource consumption, and leading to low model training efficiency. Summary of the Invention

[0004] This application provides at least one sample processing method, apparatus, device, medium, and program product.

[0005] The technical solution of this application embodiment is implemented as follows: On one hand, embodiments of this application provide a sample processing method, the method comprising: Get multiple interactive interfaces; Based on the timing information of each interactive interface, the resolutions of multiple interactive interfaces are adjusted to obtain multiple adjusted interactive interfaces; wherein at least two of the adjusted interactive interfaces have different resolutions. Sample features are extracted from multiple adjusted interactive interfaces, and these sample features are used to train a visual language model.

[0006] In some embodiments, adjusting the resolution of multiple interactive interfaces based on the timing information of each interactive interface to obtain adjusted multiple interactive interfaces includes: Identify the temporal position of each of the interactive interfaces in the time series; Based on the time interval where the time position is located and the preset correspondence, the resolutions of multiple interactive interfaces are adjusted to obtain multiple adjusted interactive interfaces; wherein, the preset correspondence includes the relationship between different time intervals and their corresponding resolutions, and at least two time intervals have different resolutions.

[0007] In some embodiments, the resolution corresponding to the earlier time interval in the time interval is not greater than the resolution corresponding to the later time interval in the time interval.

[0008] In some embodiments, extracting sample features from the adjusted plurality of interactive interfaces includes: The interface data of each adjusted interactive interface is encoded to obtain sample features corresponding to each interactive interface. The number of sample features corresponding to interactive interfaces with different resolutions is different. The visual language model is trained using the sample features corresponding to each of the interactive interfaces.

[0009] In some embodiments, adjusting the resolution of the plurality of interactive interfaces based on the timing information of each interactive interface includes: Based on the timing information of each interactive interface, determine the resolution corresponding to each interactive interface; Interpolation algorithms are used to sample multiple interactive interfaces to adjust them to the corresponding resolution.

[0010] In some embodiments, encoding the interface data of each adjusted interactive interface to obtain sample features corresponding to each interactive interface includes: Each interactive interface data is input into a visual encoder to obtain multiple image blocks corresponding to each interactive interface output by the visual encoder. Each image block is converted into a corresponding sample feature, wherein the number of sample features generated by the interactive interface data corresponding to the first resolution is less than the number of sample features generated by the interactive interface data corresponding to the second resolution, and the first resolution is less than the second resolution.

[0011] Secondly, embodiments of this application provide a sample processing apparatus, the apparatus comprising: The acquisition module is used to acquire multiple interactive interfaces; A determining module is configured to adjust the resolution of multiple interactive interfaces based on timing information of the multiple interactive interfaces to obtain multiple adjusted interactive interfaces; wherein at least two of the adjusted interactive interfaces have different resolutions. An extraction module is used to extract sample features from multiple adjusted interactive interfaces, and the sample features are used to train a visual language model.

[0012] Thirdly, embodiments of this application provide a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps of the method provided in any of the first aspects of this application.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the first aspects of embodiments of this application.

[0014] Fifthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements the steps of the method described in any of the first aspects of embodiments of this application.

[0015] This application provides a sample processing method that involves acquiring multiple interactive interfaces; adjusting the resolution of the multiple interactive interfaces based on their temporal information to obtain adjusted interactive interfaces; wherein at least two of the adjusted interactive interfaces have different resolutions; and extracting sample features from the adjusted interactive interfaces, which are then used to train a visual language model. By adjusting the resolution of interactive interfaces at different temporal intervals and then extracting sample features, the number of features to be extracted can be determined based on the contribution of each interactive interface to model training. This eliminates the need to maintain high resolution for all interactive interfaces, effectively reducing the number of image patches and corresponding sample features input to the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0017] Figure 1 This is a flowchart of a sample processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a sample processing device provided in an embodiment of this application; Figure 3 This is a flowchart of a model training method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a training system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application.

[0018] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0022] In the following description, the terms interactive interface, temporal information, resolution, visual language model, sample features, etc., are core concepts of this invention. For ease of understanding, these terms are briefly explained below: 1) Interactive Interface: Refers to the graphical user interface (GUI) generated during user interaction with the system, triggered by different interactive actions (such as clicking, swiping, inputting, etc.), such as dialog boxes, operation panels, menu interfaces, etc. These interfaces usually exist in the form of images and are used to convey user intentions or system feedback.

[0023] 2) Timing information: This refers to the chronological order and relative position of multiple interactive interfaces in a time series. This information is used to determine the temporal proximity of a particular interactive interface, thereby deciding the image resolution strategy to be applied in subsequent processing.

[0024] 3) Resolution: In this application, it specifically refers to the spatial size of the image, such as 1920×1080 or 1024×576. The higher the resolution, the richer the image details, but it also leads to the generation of more visual tokens, increasing memory consumption and computational overhead.

[0025] 4) Visual Language Model (VLM): A deep learning model capable of processing both image and text data simultaneously, commonly used for tasks such as image description generation, image-text question answering, and instruction execution. VLM training typically relies on image-text pairs, converting images into visual tokens using a visual encoder, then concatenating them with text tokens before feeding them into the language model for training.

[0026] 5) Sample Features: These refer to the feature vectors extracted from the adjusted interactive interface for model training. They are typically a set of high-dimensional vector representations obtained by encoding image patches using a visual encoder (such as a Vision Transformer). The number of sample features is closely related to the image resolution; the higher the resolution, the more sample features are generated. It should be noted that in this application, sample features and visual tokens refer to the same concept, i.e., the high-dimensional vector representations obtained by encoding image patches using a visual encoder.

[0027] 6) Bicubic interpolation and Lanzos interpolation: Both are commonly used interpolation methods in image scaling. The former has higher computational efficiency and is suitable for medium-precision requirements; the latter can better preserve image details and is suitable for high-quality scaling scenarios.

[0028] Currently, in modern human-computer interaction systems, visual language models are widely used to understand the relationship between user interfaces and user behavior. These models typically require training using a sequence of multiple historical and current UI screenshots to capture task context and generate reasonable responses or decisions. In related technologies, to enable the model to effectively process this image information, all UI screenshots are usually uniformly adjusted to a fixed resolution, then segmented into fixed-size image blocks, converted into visual tokens by a visual encoder, and finally concatenated with text tokens before being input into a language model for processing.

[0029] The contribution of a UI interaction sequence, composed of multiple historical UI screenshots and the current UI screenshot, to the model's generation decision decreases over time. In other words, the current frame can carry the user's immediate action objects (such as clicked buttons, input text boxes, and pop-up states), recent frames can reflect the operation chain (such as the menu opened in the previous step), while older frames only provide basic context (such as the initial interface layout). The contribution of the UI interaction sequence to the current decision decays over time.

[0030] Therefore, if the implementation method in the relevant technology is adopted to uniformly adjust all UI screenshots to a fixed resolution, it will lead to a surge in the number of visual tokens, resulting in redundancy. Specifically, if the fixed resolution is the optimal high-definition specification for the model (e.g., 1920×1080, approximately 2.07 million pixels), even with a 10-frame sequence, the total number of pixels reaches 20.7 million; if 20 historical frames are included, the total number of pixels will soar to 41.4 million; this will lead to redundancy in the total number of pixels. Furthermore, under the condition of a fixed resolution at the optimal high-definition specification for the model, if visual tokens (features) are generated, taking the visual encoder ViT-B / 16 as an example, which divides the image into 16×16 pixel image patches, a 1920×1080 resolution image needs to be divided into 7200 patches, corresponding to 7200 visual tokens. If all 10 frames are processed in this way, there will be 72,000 visual tokens alone, far exceeding the maximum length of a single input sequence for a large language model. If the sequence is forcibly compressed, older frames or tokens need to be pruned or merged, which would result in loss of context. If it is not compressed, it needs to be adapted to an ultra-long sequence model, which would further increase the technical threshold.

[0031] On the other hand, the dramatic increase in the number of visual tokens leads to increased computational resource consumption. First, the visual tokens generated by a single ViT-B / 16 have a dimension of 768, and the visual token tensor size of a 10-frame sequence is 52,992,000 dimensions; the input tensor size will reach approximately 53 million dimensions. Stored in half-precision (FP16), a single sample requires approximately 102MB of GPU memory (53 million × 2 bytes); if batch training is performed (Batch Size=32), the input tensor alone requires 3.26GB of GPU memory. Adding model parameters, gradient information, etc., this easily exceeds the conventional GPU memory limit, leading to training interruptions or the need to reduce the batch size. Second, the core operation of large language models is the self-attention mechanism. When the sequence length increases from 1000 to 7000, the computational load increases 49 times. If the self-attention operation takes 1ms per sample, a 10-frame sequence will take 49ms. The total time for batch training will double or even increase several times, making each training cycle extremely long and resulting in low model training efficiency.

[0032] On the other hand, fixed-resolution processing can lead to an imbalance in the information weights of different UI screenshots. Specifically, key information such as button text and pop-up prompts in the current frame has the same importance as redundant information such as background images and irrelevant controls in older frames at the token level. The self-attention mechanism distributes computing power equally among all tokens, diluting the focus on core information. This causes the model to learn irrelevant connections from older frames, resulting in misjudgments in actual inference due to unseen details from older frames.

[0033] To address the aforementioned issues, this application proposes a sample processing method, apparatus, device, medium, and program product. By adjusting the resolution of interactive interfaces at different time points and then extracting sample features, the number of features to be extracted can be determined based on the contribution of each interactive interface at different time points to model training. This eliminates the need to maintain high-resolution processing for all interactive interfaces, effectively reducing the number of image patches and corresponding sample features input into the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0034] Figure 1 This is a flowchart of a sample processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method can be executed by an electronic device, which may be a computing device, a graphics processing unit (GPU), or a dedicated training system. That is, the sample processing methods of the various embodiments of this application can be executed by a computing device, by a GPU, or interactively between a computing device and a GPU. The method includes the following steps.

[0035] Step S101: Obtain multiple interactive interfaces.

[0036] The interactive interface can be a display interface generated in response to different interactive actions.

[0037] In this embodiment, the interactive interface refers to a graphical user interface (GUI) generated during user interaction with the system, triggered by different interactive actions (such as clicking, swiping, and input). For example, the GUI may include, but is not limited to, dialog boxes, operation panels, and menu interfaces. This GUI typically exists in image form and can be used to convey user intent or system feedback. The device providing the interactive interface can be any computing device with image acquisition capabilities, such as a computer, smartphone, tablet, or server. This computing device can receive user interaction operations and generate corresponding GUI images based on the user's operation results. For example, when a user runs a task automation tool on a mobile phone, each click of a button or swipe of the screen generates a new GUI image, which is recorded as input data for subsequent processing.

[0038] Interactive actions can include, but are not limited to, user actions such as clicking buttons, selecting options, entering text, and dragging controls. Each interactive action results in the generation and display of a specific graphical user interface (GUI) to the user. Therefore, multiple GUIs can arise from a series of user actions. For example, in an application, a user can use interactive actions to sequentially open the main interface, settings interface, function option interface, etc., with each GUI representing the user's interaction state with the system at a particular moment.

[0039] In this step, screen recording tools on the device, interface capture, and other methods can be used to record the interface changes during the user's operation in real time, and automatically save them as a continuous screenshot sequence according to the timestamp. The continuous screenshot sequence includes multiple frames of interactive interface in a continuous time sequence, including both the current frame (the interface corresponding to the user's immediate operation) and historical frames (several interfaces before the current frame, such as the preceding interface and the initial interface in the operation chain). In this way, a complete interactive context chain can be formed.

[0040] Step S102: Adjust the resolution of the multiple interactive interfaces according to the timing information of the multiple interactive interfaces to obtain the adjusted multiple interactive interfaces.

[0041] Among them, at least two of the adjusted interactive interfaces have different resolutions.

[0042] In this embodiment, temporal information refers to the sequential order of multiple interactive interfaces in a time series and their relative positions. In this embodiment, temporal information can be used to determine the temporal proximity of a particular interactive interface and to decide the image resolution strategy applied to that interface in subsequent processing. Specifically, the interactive interface at the current moment (i.e., the most recently generated interface) corresponds to the most recent temporal position, while interactive interfaces from historical moments correspond to temporal positions that are further back than the current moment.

[0043] Resolution can be used to characterize the spatial size of an image. For example, the resolution in this embodiment could be 1920×1080 or 1024×576, etc. Higher resolution results in richer image details, but also leads to the generation of more visual tokens, increasing memory consumption and computational overhead.

[0044] Therefore, in order to balance the relationship between information retention and resource consumption, in this embodiment of the application, the resolution corresponding to each interactive interface can first be determined based on the timing information of multiple interactive interfaces.

[0045] In one possible implementation, a fixed mapping table can be pre-set. This table represents the mapping relationship between time intervals and corresponding resolutions. This allows resolutions to be directly assigned according to the time interval in which each interactive interface is located, without the need for dynamic calculation. After reading the timestamps of each interface, the corresponding time interval is determined, and the resolution in the mapping table is directly called (e.g., T0 matches 1920×1080, T-2 matches 1344×756).

[0046] For example, the current frame (T0) can be used as a baseline to divide the time distance into three intervals: interval A: the current frame (T0), with a time distance of 0; interval B: the previous 1-3 frames (T1-T3), with a time distance of 1-3; and interval C: the previous 4 frames and earlier (T4 and earlier), with a time distance ≥ 4. Among them, interval A can correspond to 1920×1080 (highest) to retain the control and text details of the current operation; interval B can correspond to 1344×756 (medium) to retain key changes in the recent operation chain; and interval C can correspond to 1024×576 (lowest) to retain only the basic layout context.

[0047] In another possible implementation, the resolution can be linearly reduced over time, based on the highest resolution of the current frame, so that the contribution of different interactive interfaces is proportional to the resolution. This allows for precise adaptation to the contribution of interactive interfaces at different time intervals, improving flexibility.

[0048] For example, first, a baseline resolution can be determined, such as setting the current frame as the highest resolution, like 1920×1080. Then, for each additional frame, the width and height of the resolution decrease according to a decreasing factor. For instance, with a decreasing factor k of 0.8, for each additional frame, the width and height become 80% of the previous frame. When the time distance between a certain interactive interface and the current frame is d, the resolution of that interactive interface is calculated as R(d) = R0 × (k). d .

[0049] It should be noted that since visual encoders have integer pixel requirements for input resolution, the calculation results can be normalized to multiples of 16 (such as the 16×16 image blocks commonly used in ViT), such as normalizing 1229×691 to 1232×688.

[0050] In another possible implementation, in addition to timing information, the resolution can be dynamically adjusted according to the type of interaction scenario, thus adapting to scenarios with large differences in interaction complexity.

[0051] Different resolution weight coefficients can be assigned to different interaction scenario types. For example, in the case of form input, since details such as text boxes and button text need to be preserved, the weight coefficient can be set to 1; in the case of menu switching, since only the position of menu options needs to be preserved and the text can be simplified, the weight coefficient can be set to 0.8; in the case of page navigation, only basic identifiers such as page titles need to be preserved, so the weight coefficient can be set to 0.5.

[0052] In this embodiment of the application, after obtaining the resolution corresponding to each interactive interface, the original image can be downsampled using an image scaling algorithm to adjust the multiple interactive interfaces and obtain the adjusted multiple interactive interfaces.

[0053] Image scaling algorithms can include, but are not limited to, bicubic interpolation and Lanczos interpolation. Bicubic interpolation is suitable for medium-precision requirements and has high computational efficiency; while Lanczos interpolation can better preserve image details. Depending on the actual application scenario and hardware conditions, a suitable interpolation algorithm can be selected for scaling the interactive interface.

[0054] In practice, for each interactive interface, based on the resolution determined in the above steps, an interpolation algorithm can be used to scale the corresponding interactive interface pixel by pixel to ensure that the image size is consistent with the determined resolution. For example, if a historical frame (T-2) corresponds to a target resolution of 1024×576 and the original interface is 1920×1080, then it can be downsampled and scaled from 1920×1080 to 1024×576.

[0055] In this step, after obtaining the resolution corresponding to each interactive interface, the original resolution of each interactive interface can be determined first. Then, based on the determined resolution of each interactive interface and the original resolution of each interactive interface, the interactive interface to be adjusted can be determined. Finally, the interactive interface to be adjusted can be adjusted to obtain multiple adjusted interactive interfaces.

[0056] It should be noted that the resolutions of at least two of the adjusted user interfaces are different.

[0057] Step S103: Extract sample features from the adjusted multiple interactive interfaces.

[0058] Sample features are used to train the visual language model. Sample features refer to the feature vectors extracted from the adjusted interactive interface for model training; that is, a set of high-dimensional vector representations obtained by encoding image patches using a visual encoder (such as VisionTransformer). The number of sample features is closely related to the image resolution; the higher the resolution, the more sample features are generated. Therefore, by processing historical frames at low resolution, the number of sample features can be effectively reduced, thereby reducing the computational burden of model training.

[0059] In this step, after adjusting each interactive interface to the corresponding resolution, the image corresponding to the adjusted interactive interface can be converted into a visual token (sample feature) that can be processed by the visual language model through a visual encoder.

[0060] For example, the adjusted image can first be segmented into fixed-size image patches using a visual encoder (such as ViT). Each image patch is then transformed into a fixed-dimensional vector (e.g., 768-dimensional) through linear projection. Each vector is encoded, and the encoded vectors are concatenated to form the sample features (visual token sequence) of the interface. Higher resolution interfaces generate more tokens. Next, the user instruction text corresponding to the interface (e.g., "Click the login button") can be converted into text tokens using a text encoder and concatenated with the visual token sequence. Finally, the concatenated sequence can be input into a visual language model, where supervised fine-tuning learns the mapping relationship between decisions and responses based on the interface sequence and instructions.

[0061] By adopting the above technical solution, by adjusting the resolution of the interactive interface at different time sequences and then extracting sample features, it is possible to determine the number of features to be extracted based on the contribution of the interactive interface at different time sequences to the model training. It is not necessary to maintain high resolution processing for all interactive interfaces, which can effectively reduce the number of image patches and corresponding sample features input into the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0062] In some embodiments, step S102 can be implemented by steps S1021 and S1022.

[0063] Step S1021: Identify the temporal position of each interactive interface in the time series.

[0064] Here, the temporal position refers to the relative order of the interactive interface in the time sequence, used to determine whether the interactive interface belongs to the current frame or a historical frame. For example, in a UI screenshot sequence containing four frames (T... 3,T 2,T Each frame (1, T0) has a distinct temporal position. By identifying the temporal position, we can determine how far each interactive interface is from the current moment and decide how to handle the interactive interface accordingly. Specifically, the temporal position can be identified using timestamps or frame indexes.

[0065] In this step, when capturing the interactive interface, a unique high-precision timestamp can be bound to each frame of the interface. The timestamp value can correspond one-to-one with the time the interface was generated.

[0066] First, timestamps can be extracted from the metadata of all interactive interfaces to form a timestamp set. The maximum value in the timestamp set is then selected, and the corresponding interface is the current frame (T0), with its timestamp serving as the baseline timestamp. Then, the timestamps can be directly compared and sorted to determine the temporal position of each frame. For example, if the timestamps are sorted from largest to smallest as T0→T-1→T-2→T-3, the corresponding temporal positions are current frame→previous frame→previous 2 frames→previous 3 frames.

[0067] In one possible implementation, when capturing the interactive interface, each frame of the interactive interface image can be assigned a continuously increasing frame index according to the generation order of the interactive interface; wherein the index value of the frame index can strictly correspond to the generation order of each frame of the interactive interface image (i.e., the later the generation time, the larger the index). Specifically, firstly, all frame indices can be extracted from the captured interactive interface sequence to determine the maximum index value. The interface corresponding to the maximum index value is the current frame (T0). The difference between the index of the remaining frames and the maximum value is the temporal distance. Then, the temporal position is directly determined based on the temporal distance. For example, a temporal distance of 0 is the current frame, a temporal distance of 1 is the previous frame, and a temporal distance ≥ 2 is a distant historical frame.

[0068] Step S1022: Adjust the resolution of multiple interactive interfaces according to the time interval where the time position is located and the preset correspondence, so as to obtain multiple interactive interfaces after adjustment.

[0069] The pre-defined correspondence includes the relationship between different time intervals and their corresponding resolutions. A time interval can refer to several time periods into which the entire time series is divided. Each interval can correspond to the same or different resolution processing strategies, but at least two time intervals must have different resolutions. For example, the interval where the current frame is located (e.g., T0) uses the highest resolution, while the most recent historical frame (e.g., T...) uses the highest resolution. 1) Use a second-highest resolution and older frames (such as T). 2. T 3) Then, gradually downsample to a lower resolution. The preset mapping relationship is a pre-defined set of mapping rules that binds each time series interval to a specific resolution value. The preset mapping relationship can be configured according to actual task requirements, for example, adjusted based on empirical data or model performance test results.

[0070] In some embodiments, the resolution corresponding to the earlier time interval in the time interval is not greater than the resolution corresponding to the later time interval in the time interval.

[0071] In this embodiment, the resolution corresponding to an earlier time interval (i.e., a time period that occurred earlier) is no greater than the resolution corresponding to a later time interval (i.e., a time period closer to the current moment). This ensures that the image processing accuracy gradually increases as the image's time position approaches the current frame, thereby guaranteeing that the current frame is input to the model at the highest resolution, while historical frames gradually have their resolution reduced according to their time proximity, achieving differentiated processing.

[0072] For example, suppose there are four frames in an image sequence: T-3, T-2, T-1, and T0, belonging to three time intervals: [T-3, T-2], [T-2, T-1], and [T-1, T0]. We can set the resolution of the first time interval to 1024×576, the second to 1344×756, and the third to 1600×900, with T0 retaining its original resolution of 1920×1080. Images closer to the current frame have higher resolutions, thus preserving more detail in the model.

[0073] In this embodiment, by introducing a rule that the resolution corresponding to an earlier time interval is no greater than the resolution corresponding to a later time interval, dynamic adjustment of the image sequence resolution can be achieved. This rule reduces the number of visual tokens in historical frames without losing key information, thereby reducing memory usage and computational complexity during model training. The differentiated resolution strategy formed by introducing this rule helps optimize model attention allocation, allowing the model to focus more on the high-precision details of the current frame while still acquiring sufficient historical context information.

[0074] In this step, a fixed mapping table (i.e., a preset correspondence) can be pre-set. This mapping table can represent the mapping relationship between time intervals and corresponding resolutions. In this way, the resolution can be directly assigned according to the time interval in which each interactive interface is located, without the need for dynamic calculation. After reading the timestamp of each interface, the time interval to which it belongs is determined, and the resolution in the mapping table is directly called (e.g., T0 matches 1920×1080, T-2 matches 1344×756).

[0075] For example, the current frame (T0) can be used as a baseline to divide the time distance into three intervals: interval A: the current frame (T0), with a time distance of 0; interval B: the previous 1-3 frames (T1-T3), with a time distance of 1 to 3; and interval C: the previous 4 frames and earlier (T4 and earlier), with a time distance ≥ 4. Among them, interval A can correspond to 1920×1080 (highest) to retain the control and text details of the current operation; interval B can correspond to 1344×756 (medium) to retain key changes in the recent operation chain; and interval C can correspond to 1024×576 (lowest) to retain only the basic layout context.

[0076] By adopting the above technical solution, the resolution of the interactive interface can be determined according to its position in the time series, which can effectively reduce unnecessary visual token generation, thereby significantly reducing memory usage and improving the efficiency and stability of model training.

[0077] In some embodiments, step S103 can be implemented in the following ways.

[0078] Specifically, the interface data of each adjusted interactive interface can be encoded to obtain the sample features corresponding to each interactive interface.

[0079] The number of sample features varies depending on the resolution of the interactive interface.

[0080] In this embodiment, each interactive interface can first be adjusted to the corresponding resolution to obtain the interactive interface data corresponding to each interactive interface.

[0081] The interactive interface data consists of sampled and adjusted image data. This data retains key image information while removing unnecessary redundancy. For example, data from the current frame and recent frames retain complete details (such as control text and button states), while data from older frames only retains the core outline (such as page layout and key control positions). This reduces the amount of data without losing the crucial information needed for decision-making.

[0082] In this step, before adjusting each interactive interface to the corresponding resolution, the original interactive interface can be preprocessed to avoid image quality distortion or information loss after adjustment.

[0083] In some embodiments, the original interface can be converted to a uniform image format (such as PNG or JPG format with RGB channels). This ensures that the pixel data can be recognized by the interpolation algorithm and that the initial resolution of the original interface is recorded. Then, based on the timing information of each interactive interface, the resolution corresponding to each interactive interface can be determined. For each interactive interface, based on its determined resolution (such as 1920×1080, 1344×756), the interpolation algorithm can be used to sample multiple interactive interfaces to obtain the interactive interface data corresponding to each interactive interface.

[0084] In one possible implementation, for each interactive interface, when the initial resolution is higher than the determined resolution, pixels can be merged using an interpolation algorithm to remove redundant details (such as background textures and irrelevant decorative elements). Specifically, bicubic interpolation or Lanzos interpolation algorithms can be used to merge pixels and remove redundant details.

[0085] Interpolation algorithms are image processing techniques used to adjust the size or resolution of an image by calculating interpolation between pixels without altering the image content. Interpolation algorithms preserve the basic features and details of an image and achieve smooth transitions between different resolutions. Interpolation algorithms include at least one of bicubic interpolation and Lanzos interpolation.

[0086] Bicubic interpolation is an image scaling method that determines the value of a target pixel by calculating the weighted average of its 16 neighboring pixels in four directions. Bicubic interpolation can effectively preserve image details and edge information, making it suitable for moderate image scaling operations. In this application, when downsampling historical frame images, using bicubic interpolation can reduce resolution while maintaining image sharpness and structural integrity as much as possible, thus ensuring that the model can still obtain sufficiently effective contextual information.

[0087] Lanczos interpolation is also a high-quality image scaling algorithm. Its core idea is to use a window function (such as the Sinc function) to filter the frequency domain information of the original image, and then transform the frequency domain information back to the spatial domain to generate an image of the target size. Compared to bicubic interpolation, Lanczos interpolation better preserves high-frequency details when handling high-magnification scaling, but its computational complexity is higher. In this application, Lanczos interpolation can be used to process earlier historical frames at low resolution, thereby further reducing the number of visual tokens, lowering memory usage, and improving training efficiency while ensuring visual information quality.

[0088] By employing at least one of the two interpolation algorithms mentioned above, this application can achieve a flexible and efficient resolution adjustment strategy between historical frames at different time intervals.

[0089] It should be noted that before downsampling, the image can be smoothed using a low-pass filter (such as a Gaussian filter), which can avoid the jagged effect caused by the sharp reduction in pixels.

[0090] In another possible implementation, for each interactive interface, when the initial resolution is lower than the determined resolution, pixels can also be supplemented through interpolation algorithms, and smooth expansion can be performed based on the original information to ensure that the image size meets the requirements.

[0091] In this embodiment, the data of each interactive interface can then be encoded to obtain the sample features corresponding to each interactive interface.

[0092] The number of sample features varies depending on the resolution of the interactive interface.

[0093] In this embodiment, encoding refers to converting image data into a vector form that the model can recognize, typically using a visual encoder (such as a Vision Transformer, ViT). ViT divides the image into multiple image patches and maps the images to visual tokens. Since the number of sample features generated from interactive interface data at different resolutions varies, the number of visual tokens generated from interactive interface data in historical frames will also be reduced accordingly, thus reducing the overall length of the input sequence.

[0094] Sample features are vector representations output by the visual encoder, used to describe the visual content of an image. In this invention, due to the different resolutions of each frame, the number of sample features corresponding to the interactive interface data at different resolutions will also differ. The current frame has more sample features, while historical frames have fewer. This design approach, where the number of sample features varies with resolution, helps the model focus more on the information needed for the current decision.

[0095] In this step, before encoding each interactive interface data, the output interactive interface data can first be converted into the input format required by the visual encoder. For example, interactive page data in the format H×W×C can be converted into the B×C×H×W format required by ViT, where H is the height, W is the width, C is the number of channels, and B is the number of samples processed in a batch. Then, the pixel values ​​can be normalized according to the pre-training standard of the visual encoder, for example, scaling the RGB pixel values ​​of 0-255 to the range of [-1,1] or [0,1].

[0096] After converting the output interactive interface data into the input format required by the visual encoder, each interactive interface data can be encoded to obtain the sample features corresponding to each interactive interface.

[0097] First, the interactive interface data can be segmented into non-overlapping image blocks of a fixed size using a visual encoder. The higher the resolution, the more image blocks there are, and the more sample features there are. For example, when the resolution of the current frame is 1920×1080, the number of segments obtained is (1920 / 16)×(1080 / 16)=7200 blocks; when the resolution of a recent frame is 1344×756, the number of segments obtained is (1344 / 16)×(756 / 16)=3969 blocks; and when the resolution of a distant frame is 1024×576, the number of segments obtained is (1024 / 16)×(576 / 16)=2304 blocks. The number of features decreases in a stepwise manner across the three frames.

[0098] Then, a linear transformation can be performed on each image patch to convert it from a pixel matrix (e.g., 16×16×3) into a fixed-dimensional vector (e.g., 768-dimensional), forming an initial visual token. Each image patch corresponds to one initial token, and the number of tokens is the same as the number of image patches.

[0099] This allows for effective control of the input data size while ensuring model performance, thereby improving training efficiency.

[0100] In some embodiments, each interactive interface data can be input into a visual encoder to obtain multiple image blocks corresponding to each interactive interface output by the visual encoder; then each image block can be converted into corresponding sample features.

[0101] Among them, the number of sample features generated by the interactive interface data corresponding to the first resolution is less than the number of sample features generated by the interactive interface data corresponding to the second resolution, and the first resolution is less than the second resolution.

[0102] A visual encoder is a deep learning model component used to convert images into vector representations, such as Vision Transformer (ViT) and ResNet. The role of a visual encoder is to segment an image into several fixed-size image patches and map each patch to a high-dimensional vector (i.e., patch embedding), thus providing a foundation for subsequent feature extraction and semantic understanding. Visual encoders are typically composed of convolutional layers or self-attention mechanisms, effectively capturing both local and global information in an image.

[0103] An image patch is a unit formed by dividing the entire image into several small regions. Each image patch typically has a fixed size (e.g., 14×14 pixels) and is generated across the entire image using a sliding window approach. The way image patches are divided determines how the visual encoder perceives the image content and directly affects the quantity and quality of the final visual tokens.

[0104] By controlling the number of sample features at different resolutions, the length of the input sequence can be significantly reduced while preserving contextual information. Controlling the number of sample features at different resolutions not only helps save GPU memory resources but also speeds up model training.

[0105] In this way, the importance of each frame in the image sequence can be distinguished based on different temporal information; thus, the attention allocation of the model can be optimized, allowing the model to focus more on the high-precision details of the current frame; thereby significantly reducing memory usage and training time, and improving training efficiency and model generalization ability.

[0106] In this embodiment, the visual language model is a deep learning model capable of processing both image and text information simultaneously. It is commonly used in multimodal tasks such as image-text understanding and question-answering systems. The visual language model can receive text tokens from instruction text and visual tokens from images as input, and it performs supervised fine-tuning training. For example, a visual encoder may include a Vision Transformer model.

[0107] Sample features are used to train the visual language model. Specifically, sample features can be concatenated with text features and then input into the model. The model optimizes its parameters through backpropagation to better perform the specified task. Because the number of sample features generated from interactive interface data at different resolutions varies, the model can learn the importance of the current frame more efficiently without being interfered with by redundant historical frames.

[0108] In this embodiment, a multimodal input sequence can first be constructed to achieve effective fusion of visual and textual information. User command text matching the interactive interface sequence (such as "click the search box at the top of the page and enter 'weather'") can be converted into a text token sequence using a text encoder. Then, the text tokens and visual tokens (i.e., sample features) are concatenated in the logical order of "text command - temporal interface" to form a standard input sequence. Since the total number of visual tokens has been optimized in the previous differential resolution processing, the sequence length can fully adapt to the model's input requirements, preserving the complete interactive context without truncation.

[0109] Next, the model forward propagation is performed, and the concatenated multimodal sequence is input into the visual language model. Cross-modal information fusion is achieved through the cross-attention mechanism inside the model. That is, the model automatically learns the relationship between text tokens and visual tokens, and generates predicted outputs according to the training task objectives. For example, the interactive decision task outputs operation coordinates, the intent understanding task outputs semantic parsing results, and the response generation task outputs natural language responses, so as to complete the training of the visual language model using the sample features corresponding to each interactive interface.

[0110] By adjusting the interactive interfaces of different time sequences to the resolution corresponding to their time sequence information through the above steps, the number of image blocks and sample features input into the visual language model can be effectively reduced, significantly reducing memory usage and improving training efficiency. At the same time, the design of "the closer the time sequence, the higher the resolution" preserves key information and compresses redundant content, making the model more focused on the information needed for the current decision, and improving the accuracy and generalization of understanding user intent.

[0111] Figure 2 This is a schematic diagram of a sample processing device provided in an embodiment of this application, as shown below. Figure 2As shown, the sample processing device 200 may include an acquisition module 201, a determination module 202, and an extraction module 203.

[0112] Module 201 is used to acquire multiple interactive interfaces; The determining module 202 is used to adjust the resolution of multiple interactive interfaces according to the timing information of multiple interactive interfaces to obtain multiple adjusted interactive interfaces; wherein, at least two of the adjusted interactive interfaces have different resolutions. Extraction module 203 is used to extract sample features from multiple adjusted interactive interfaces, and the sample features are used to train the visual language model.

[0113] In some embodiments, the determining module 202 can be used to identify the temporal position of each interactive interface in the time series; adjust the resolution of multiple interactive interfaces according to the temporal interval in which the temporal position is located and a preset correspondence, to obtain multiple interactive interfaces after adjustment; wherein, the preset correspondence includes the relationship between different temporal intervals and their corresponding resolutions, and at least two temporal intervals have different resolutions.

[0114] In some embodiments, the resolution corresponding to the earlier time interval in the time interval is not greater than the resolution corresponding to the later time interval in the time interval.

[0115] In some embodiments, the extraction module 203 can be used to encode the interface data of each adjusted interactive interface to obtain sample features corresponding to each interactive interface, wherein the number of sample features corresponding to interactive interfaces with different resolutions is different.

[0116] In some embodiments, the determining module 202 can be used to determine the resolution corresponding to each interactive interface based on the timing information of each interactive interface; and to sample multiple interactive interfaces using an interpolation algorithm to adjust the multiple interactive interfaces to the corresponding resolution. The interpolation algorithm includes at least one of bicubic interpolation and Lanzos interpolation.

[0117] In some embodiments, the extraction module 203 can be used to input each interactive interface data into a visual encoder to obtain multiple image blocks corresponding to each interactive interface output by the visual encoder; and convert each image block into corresponding sample features, wherein the number of sample features generated by the interactive interface data corresponding to the first resolution is less than the number of sample features generated by the interactive interface data corresponding to the second resolution, and the first resolution is less than the second resolution.

[0118] By adopting the above technical solution, by adjusting the resolution of the interactive interface at different time sequences and then extracting sample features, it is possible to determine the number of features to be extracted based on the contribution of the interactive interface at different time sequences to the model training. It is not necessary to maintain high resolution processing for all interactive interfaces, which can effectively reduce the number of image patches and corresponding sample features input into the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0119] The description of the above device-side embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0120] The following describes the application of the sample processing method provided in the embodiments of this application in a real-world scenario, mainly involving a visual language model training method and system based on multi-resolution historical frames.

[0121] In the current mainstream training paradigms of GUI agents or visual language models (such as Qwen-VL, LLaVA, etc.), the model usually needs to process a sequence containing multiple historical UI screenshots and the current UI screenshot in order to understand the context of the task and make the correct decision.

[0122] In related technologies, a uniform resolution strategy is typically used when processing these image sequences. Specifically, each UI screenshot in the sequence, whether from the current frame or a historical frame, is uniformly scaled to a fixed resolution (e.g., 336×336 pixels). Subsequently, the image is segmented into fixed-size patches, and each patch is converted into one or more visual tokens by a visual encoder (such as ViT). Finally, these visual token sequences are concatenated with the instruction text token sequence and input into a large language model for processing. This approach processes all image information in the time series indiscriminately and with equal precision, failing to consider that the weight of historical frame information in the decision-making process may be lower than that of the current frame, leading to the following problems.

[0123] On the one hand, maintaining all historical frames at a high resolution results in an extremely large number of generated visual tokens. In the Transformer architecture, the computational complexity and memory usage of the attention mechanism are proportional to the square of the token sequence length. This leads to a sharp increase in the VRAM required for training when handling complex tasks with long historical sequences, placing extremely high demands on hardware and resulting in massive consumption of VRAM resources.

[0124] On the other hand, a large number of visual tokens not only increases the burden on GPU memory, but also significantly increases the computational load of the model, resulting in excessively long individual training steps, reduced overall training throughput, and extended model development and iteration cycles, leading to low model training efficiency.

[0125] On the other hand, while this scheme treats all frames equally, in reality, for many decisions made by the GUI Agent, the high-precision details of the current UI are crucial, while the role of historical interfaces is more to provide context, and the value of their high-frequency details is relatively low. Existing methods fail to effectively distinguish this information priority, introducing a large number of potentially redundant visual tokens, which may interfere with the model's focus on the most critical information of the current frame, thus leading to information redundancy and a lack of distinction between primary and secondary information.

[0126] To address the aforementioned technical problems, the embodiments of this application can be used to solve the technical problems of excessive training resource consumption and low efficiency caused by high-resolution processing of historical frames in related technologies.

[0127] This application provides a novel visual language model training method that aims to significantly reduce the number of visual tokens in the supervised fine-tuning (SFT) stage of the GUIAgent, thereby reducing memory usage, improving training efficiency, and maintaining the model's performance on complex tasks as much as possible.

[0128] This visual language model training method can apply different resolution processing strategies to the input UI screenshot sequence based on its time distance. Figure 3 This is a flowchart of a model training method provided in an embodiment of this application, such as... Figure 3 As shown, this method is mainly implemented through the following steps.

[0129] Step S301: Receive image sequence and instruction data.

[0130] The image sequence can be an interactive interface sequence composed of multiple interactive interfaces in the above embodiments.

[0131] In this step, a training sample can be input into the system. This training sample contains a sequence of UI screenshots arranged in chronological order and the corresponding user command text for each screenshot. For example, n+1 frames from timestamp Tn to T0, where T0 is the current frame.

[0132] For example, a data loader can be used to identify the temporal order of each frame when reading samples.

[0133] Step S302: Perform differential resolution processing on the image sequence.

[0134] In this step, different resolution processing can be applied to each frame based on its position in the time series.

[0135] Specifically, this can be achieved through the following steps S3021 and S3022: Step S3021: Process the current frame.

[0136] In this step, the current frame can be held or scaled to the preset highest resolution (such as 1920×1080 or the best high-definition resolution supported by the model) to ensure that the model can obtain the most accurate and richest current interface details for making immediate decisions.

[0137] For example, the resolution of T0 can be kept at 1920×1080.

[0138] Step S3022: Process recent historical frames.

[0139] In this step, historical frames are processed with decreasing resolution. For example, the previous frame (T...) is processed with decreasing resolution. 1) Downsample to a medium resolution (e.g., 1344×756), and use earlier frames (T... 2) Downsampling to a lower resolution (e.g., 1024×576). The degree of downsampling is positively correlated with the time elapsed between the current frame and the previous frame; that is, the older the frame, the lower the resolution. Specifically, downsampling can employ domain-standard image processing algorithms, such as bicubic interpolation or Lanczos interpolation.

[0140] For example, T can be... One frame was downsampled to 1600×900, which can reduce T Two frames were downsampled to 1344×756, which can be T Three frames were downsampled to 1024×576.

[0141] Step S303: Visually encode the processed image sequence to obtain a visual token.

[0142] In this step, the images of each frame after differential resolution processing can be fed into a visual encoder (such as Vision Transformer, ViT). The encoder segments each downsampled image into image patches and converts them into visual tokens. Due to the reduced resolution of historical frames, the number of visual tokens generated will be much less than that of the current frame.

[0143] For example, when using the ViT-L / 14 visual encoder, it can segment an image into 14×14 image blocks. In related technologies, all frames are 1920×1080, and the total number of tokens may be extremely high. However, in the embodiment of this application, only T0 generates a high number of tokens, and the number of tokens generated by historical frames such as T-1, T-2, and T-3 will decrease significantly in turn.

[0144] Step S304: Train the visual language model using visual tokens.

[0145] In this step, the instruction text token is concatenated with the visual token sequence generated from all frames (including the high-resolution current frame and the low-resolution historical frames) to form the final input sequence. This sequence is then input into the main body of a large language model for standard supervised fine-tuning training (calculating loss, backpropagation, and updating weights).

[0146] For example, the instruction token and multiple sets of visual tokens are concatenated and then fed into a visual language model such as Qwen2.5-VL for fine-tuning. Because the total length of the input sequence is significantly shortened, the peak GPU memory usage during training is greatly reduced, thus improving training speed.

[0147] By adopting the above technical solution, by adjusting the interactive interfaces of different time series to the resolution corresponding to their time series information, and then extracting sample features, it is not necessary to maintain high resolution processing for all interactive interfaces. This can effectively reduce the number of image blocks and corresponding sample features input into the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0148] Figure 4 This is a schematic diagram of the structure of a training system provided in an embodiment of this application, such as... Figure 4 As shown, the system 400 includes a data receiving module 401, an image preprocessing module 402, a visual encoding module 403, and a model training module 404.

[0149] The data receiving module 401 is used to receive training samples, that is, to acquire multiple interactive interfaces.

[0150] The image preprocessing module 402 is used to perform differential downsampling on the image based on the timestamp, that is, to determine the resolution corresponding to each interactive interface based on the time sequence information of multiple interactive interfaces, and to adjust each interactive interface to the corresponding resolution.

[0151] The visual encoding module 403 is used to extract sample features from the adjusted multiple interactive interfaces.

[0152] The model training module 404 is used to train the visual language model based on sample features.

[0153] By adopting the above technical solution, by adjusting the interactive interfaces of different time series to the resolution corresponding to their time series information, and then extracting sample features, it is not necessary to maintain high resolution processing for all interactive interfaces. This can effectively reduce the number of image blocks and corresponding sample features input into the visual language model, thereby significantly reducing memory usage and improving the training efficiency of the visual language model.

[0154] In one aspect, by reducing the resolution of historical frames, the total number of visual tokens input into the model is directly and effectively reduced, thereby significantly reducing the demand for GPU memory during training. This allows for larger batch sizes or the training of more complex models under the same hardware conditions. Furthermore, the shortened input sequence length reduces the computational load of core modules such as the attention mechanism, accelerating the forward and backward propagation of the model, increasing the number of training steps per unit time, and shortening the model training cycle. Finally, information prioritization is introduced at the data level. By providing the model with the most accurate current frame and generalized historical frames, the model may be guided to focus more on the core visual information needed for the current decision, reducing interference from unnecessary historical details and potentially improving model performance.

[0155] Furthermore, the embodiments of this application can also provide several extension directions, such as adding new application scenarios, introducing additional parameter settings, and optimizing system response speed.

[0156] Specifically, the resolution reduction strategy can be dynamically adjusted based on the task type. For example, when certain tasks heavily rely on historical frames, the resolution of the images used by these tasks can be appropriately increased. Attention mechanisms can also be combined, assigning different weights to visual tokens with different resolutions to further enhance the model's ability to capture key information when processing data. This method can also be extended to video data or other temporal vision tasks, thereby improving the training efficiency and resource utilization of various visual language models.

[0157] This application embodiment allows for the application of different, progressively decreasing resolution processing to images at different positions in a time series. Specifically, it includes a method that maintains the currently acquired image at the highest resolution while simultaneously processing the resolution of historical images inversely proportional to their corresponding time duration. When applied to supervised fine-tuning (SFT) scenarios for GUI agents or visual language models, this method can reduce memory consumption and improve training efficiency.

[0158] It should be noted that, in the embodiments of this application, if any of the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0159] This application also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implement the above-described method. The computer-readable storage medium can be transient or non-transient.

[0160] This application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement some or all of the steps in any of the above-described methods. The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0161] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0162] Figure 5 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application, such as... Figure 5As shown, the hardware entity of the computer device 500 includes a processor 501 and a memory 502, wherein the memory 502 stores a computer program that can run on the processor 501, and the processor 501 executes the program to implement the steps in the method of any of the above embodiments.

[0163] The memory 502 stores computer programs that can run on the processor. The memory 502 is configured to store instructions and applications that can be executed by the processor 501. It can also cache data to be processed or already processed by the processor 501 and various modules in the computer device 500 (e.g., image data, audio data, voice communication data and video communication data). It can be implemented by flash memory or random access memory (RAM).

[0164] The processor 501 executes the steps of the above-described task execution method when executing the program. The processor 501 typically controls the overall operation of the computer device 500.

[0165] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0166] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0167] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0168] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0171] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0174] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A sample processing method, characterized in that, The method includes: Get multiple interactive interfaces; Based on the timing information of each interactive interface, the resolution of multiple interactive interfaces is adjusted to obtain multiple adjusted interactive interfaces; wherein, at least two of the adjusted interactive interfaces have different resolutions. Sample features are extracted from multiple adjusted interactive interfaces, and these sample features are used to train a visual language model. The step of adjusting the resolution of multiple interactive interfaces based on the timing information of each interactive interface to obtain adjusted multiple interactive interfaces includes: Identify the temporal position of each of the interactive interfaces in the time series; Based on the time interval where the time sequence position is located and a preset correspondence, the resolutions corresponding to multiple interactive interfaces are adjusted; wherein, the preset correspondence includes the relationship between different time intervals and their corresponding resolutions, and at least two time intervals correspond to different resolutions; or... Based on adjusting the resolution according to the time interval, the resolutions of multiple interactive interfaces are also adjusted according to the interactive scene types associated with the interactive interfaces, resulting in multiple adjusted interactive interfaces; different interactive scene types are assigned different resolution weight coefficients.

2. The method according to claim 1, characterized in that, The resolution corresponding to the earlier time interval in the time interval is not greater than the resolution corresponding to the later time interval in the time interval.

3. The method according to claim 1 or 2, characterized in that, The step of extracting sample features from the adjusted multiple interactive interfaces includes: The interface data of each adjusted interactive interface is encoded to obtain sample features corresponding to each interactive interface, wherein the number of sample features corresponding to interactive interfaces with different resolutions is different.

4. The method according to claim 1 or 2, characterized in that, Adjusting the resolution of multiple interactive interfaces based on the timing information of each interactive interface includes: Based on the timing information of each interactive interface, determine the resolution corresponding to each interactive interface; Interpolation algorithms are used to sample multiple interactive interfaces to adjust them to the corresponding resolution.

5. The method according to claim 3, characterized in that, The process of encoding the interface data of each adjusted interactive interface to obtain sample features corresponding to each interactive interface includes: Each interface data is input into a visual encoder to obtain multiple image blocks corresponding to each interactive interface output by the visual encoder. Each image block is converted into a corresponding sample feature, wherein the number of sample features generated from the interface data corresponding to the first resolution is less than the number of sample features generated from the interface data corresponding to the second resolution, and the first resolution is less than the second resolution.

6. A sample processing device, characterized in that, The device includes: The acquisition module is used to acquire multiple interactive interfaces; A determining module is used to adjust the resolution of multiple interactive interfaces based on the timing information of each interactive interface to obtain multiple adjusted interactive interfaces; wherein at least two of the adjusted interactive interfaces have different resolutions. An extraction module is used to extract sample features from multiple adjusted interactive interfaces, the sample features being used to train a visual language model; The determining module is used to identify the temporal position of each interactive interface in a time series; and adjust the resolution corresponding to multiple interactive interfaces according to the temporal interval in which the temporal position is located and a preset correspondence; wherein the preset correspondence includes the relationship between different temporal intervals and their corresponding resolutions, and at least two temporal intervals correspond to different resolutions; or... Based on adjusting the resolution according to the time interval, the resolutions of multiple interactive interfaces are also adjusted according to the interactive scene types associated with the interactive interfaces, resulting in multiple adjusted interactive interfaces; different interactive scene types are assigned different resolution weight coefficients.

7. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, said computer program product comprising a non-transitory computer-readable storage medium storing a computer program, characterized in that, When the computer program is read and executed by a computer, it implements the method described in any one of claims 1 to 5.

Images