Data processing methods, devices, and systems
By pre-determining the number of target visual lexical units and adjusting the input sequence in the central processing unit, the inference latency and throughput bottlenecks of large visual models in high-concurrency scenarios are solved, achieving reduced computation and improved response speed.
Patent Information
- Application Number
- CN202611262533.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
In high-concurrency scenarios, when large visual models process sequences containing hundreds to thousands of visual tokens, inference response latency and single-card inference throughput become bottlenecks. Existing solutions lack a visual token compression inference acceleration method specifically designed for multi-process inference architectures.
By pre-determining the number of target visual terms in the central processing unit and adjusting the number of image placeholders in the input sequence, a shorter input sequence is constructed. After visual feature extraction, target visual features are selected for filling, reducing subsequent computation and key-value cache usage.
It reduces computational load, decreases request-response latency, increases inference throughput per GPU, and alleviates performance bottlenecks in high-concurrency scenarios.
Smart Images

Figure CN123119166A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of computer technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology
[0002] With the development of multimodal large model inference, some visual large models have been able to be deployed in large numbers in online inference scenarios. However, the inference cost of visual large models is significantly higher than that of pure text large language models. Visual large models receive multimodal data of images and text. After extracting features from the images, the obtained visual features are used as visual tokens and the corresponding input sequence of text (which contains text tokens and image placeholders) to fill the input sequence. The filled sequence is then input into the large language model for inference.
[0003] However, after visual feature extraction, the input sequence contains hundreds to thousands of visual tokens. In other words, large language models will process sequences containing hundreds to thousands of visual tokens. In high-concurrency scenarios, visual tokens directly become the bottleneck for inference response latency and single-card inference throughput (number of requests processed per second). Summary of the Invention
[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.
[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Obtain the initial number of visual lexical units of the target image in the data processing request; Based on the target reduction parameters and the initial number of visual words, the target number of visual words is determined, and the input sequence corresponding to the data processing request is determined based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words; Visual features are extracted from the target image to generate initial visual features corresponding to the initial number of visual words, and target visual features corresponding to the target number of visual words are determined from the initial visual features. The target visual features corresponding to the number of target visual words are filled into the image placeholders of the input sequence to obtain the target sequence, and the processing result corresponding to the data processing request is obtained based on the target sequence.
[0006] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The quantity acquisition module is configured to obtain the initial number of visual lemmas of the target image in the data processing request; The sequence determination module is configured to determine the target number of visual words based on the target reduction parameters and the initial number of visual words, and to determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words; The feature selection module is configured to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The sequence filling module is configured to fill the target visual features corresponding to the number of target visual words into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
[0007] According to a third aspect of the embodiments of this specification, a data processing system is provided, including a central processing unit and an image processing unit, wherein... The central processing unit is configured to obtain the initial number of visual words in the target image in the data processing request; determine the target number of visual words based on the target reduction parameters and the initial number of visual words; and determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words. The image processing unit is used to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The central processing unit is further configured to fill the target visual features corresponding to the number of target visual lexical units into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
[0008] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.
[0009] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.
[0010] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.
[0011] One embodiment of this specification provides a data processing method that performs reduction calculations on the initial visual word data corresponding to the target image based on the target reduction parameters to determine the number of target visual words, thereby compressing the number of image placeholders in the input sequence to the number of target visual words. Image placeholders are reserved according to the compressed number of target visual words to construct an input sequence with a shorter sequence length. When visual features are extracted from the target image and initial visual features are generated, target visual features with the number of target visual words are selected from the initial visual features corresponding to the number of initial visual words, thereby reducing the number of visual features. The target visual features are then filled with the corresponding image placeholders in the input sequence to obtain the target sequence. By reducing the feature vectors that need to be processed subsequently, the computational load, key-value cache usage, and memory access overhead are reduced during the inference stage, thereby improving the inference throughput of a single card and reducing request response latency, alleviating the performance bottleneck caused by a large number of visual words in high-concurrency scenarios. Attached Figure Description
[0012] Figure 1 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 4 This is a flowchart of the visual feature selection process in a data processing method provided in one embodiment of this specification; Figure 5 This is an implementation flowchart of a data processing method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation
[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.
[0018] sglang: A high-throughput inference framework for service-oriented large models.
[0019] Multimodal large model: A generative model capable of processing both images and text simultaneously.
[0020] ViT: Vision Transformer, a multimodal large-scale visual encoder that segments images and feeds them into the visual encoder to calculate visual features.
[0021] Visual token: A discrete visual representation unit output from an image after ViT processing and ultimately fed into a large language model. Each token corresponds to a semantic vector of an image region.
[0022] Token: The smallest semantic and computational unit processed by the model.
[0023] Token compression: Select a suitable number of representative tokens from the original visual tokens output by ViT and feed them into the large language model to shorten the sequence length and reduce the amount of inference computation.
[0024] Dominant token: an important token, which is a visual token ranked high by importance metric and carries information-intensive semantics (such as faces, text, and object edges).
[0025] Contextual token: A visual token sampled from non-Dominant tokens, used to preserve the spatial coverage of the entire image and prevent the loss of background information.
[0026] Monkey patch: A technique for dynamically replacing class method or function references at runtime, which can inject custom logic without modifying the source code of third-party libraries.
[0027] In existing solutions, it is known that the visual tokens output by ViT are redundant, that is, some tokens carry semantic key information (faces, text, object edges), and many redundant tokens are highly similar in feature space (background, solid color areas). Therefore, visual tokens can be appropriately selected to reduce the number of visual tokens processed.
[0028] However, when applying the visual token compression scheme to a specific inference architecture, changes in the number of tokens are incompatible with the existing architecture of the sglang inference engine. Specifically, the sglang inference engine expands image placeholders to a fixed number during the pre-filling stage of request processing. Therefore, downstream stages (especially the image processing stage) need to strictly ensure that the number of tokens in each stage remains consistent. If only the number of visual tokens output by ViT is reduced, the number of visual tokens output by ViT after reduction will not match the number of image placeholders in the pre-filling stage, leading to request failure.
[0029] Furthermore, sglang introduces mechanisms such as segmentation pre-padding and cache reuse to optimize long context processing. These mechanisms rely on the precise positional offset information of the image to segment features, and changes in the number of tokens can cause misalignment of multiple positional index data.
[0030] In summary, existing solutions lack a visual token compression inference acceleration method specifically designed for the sglang multi-process inference architecture.
[0031] This specification provides a data processing method, and also relates to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.
[0032] See Figure 1 , Figure 1 This diagram illustrates a scenario of a data processing method provided in one embodiment of this specification.
[0033] Specifically, the data processing method is implemented using the end device 102 and the server 104. The server 104 includes a central processing unit 1042 and an image processing unit 1044. The end device 102 is used to send a data processing request to the server 104, which carries the target text and the target image.
[0034] Upon receiving a data processing request from the end device 102, the server 104 uses the central processing unit 1042 to obtain the initial number of visual terms in the target image of the data processing request. Based on the target reduction parameters and the initial number of visual terms, the server determines the target number of visual terms and determines the input sequence corresponding to the data processing request based on the target number of visual terms. The input sequence includes image placeholders corresponding to the target number of visual terms. The image processing unit 1044 extracts visual features from the target image, generating initial visual features corresponding to the initial number of visual terms. The server then determines the target visual features corresponding to the target number of visual terms from these initial visual features. The central processing unit 1042 fills the image placeholders in the input sequence with the target visual features corresponding to the target number of visual terms to obtain the target sequence. Based on the target sequence, the server obtains the processing result corresponding to the data processing request. The processing result is then returned to the end device 102.
[0035] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, data processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.
[0036] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.
[0037] See Figure 2 , Figure 2 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.
[0038] Step 202: Obtain the initial number of visual lexical units of the target image in the data processing request.
[0039] Here, a data processing request can be understood as an inference request initiated by the caller to the server, containing multimodal data such as images and text; the target image can be understood as the image data to be processed carried in the data processing request, for which feature extraction and compression will be performed subsequently. The initial number of visual tokens can be understood as the number of visual tokens (i.e., the number of visual tokens) that can be output after the target image is processed by the visual encoder.
[0040] It should be noted that visual features are high-dimensional semantic vectors generated after an image is processed by a visual encoder. They are abstract numerical descriptions of the image. Visual tokens, on the other hand, are discrete or continuous representation units obtained by further mapping these high-dimensional feature vectors. They are used to align inputs with downstream modules such as large language models.
[0041] Upon receiving a data processing request, the number of visual lemmas in the target image of the data processing request is calculated, and the number of image placeholders in the request sequence corresponding to the data processing request is determined based on the calculated number of visual lemmas.
[0042] In one or more embodiments of this specification, the initial number of visual terms is determined by the size information of the target image and a preset aggregation parameter, and does not change with compression or filtering operations. Specific implementation methods are described below: Based on the initial number of visual lexical units of the target image in the data processing request, including: Receive a data processing request and obtain the target text and target image in the data processing request; The central processing unit obtains the text lexical sequence corresponding to the target text, and obtains the initial number of visual lexical units of the target image based on the size information of the target image and preset aggregation parameters.
[0043] The text word sequence can be understood as an index sequence obtained after the target text has been segmented. Each index in the sequence corresponds to a word block in the vocabulary. This sequence will serve as the text part of the multimodal input sequence. Size information can be understood as the feature map size information of the target image in three dimensions: time (t), height (h), and width (w). For a single image, the time dimension information is usually 1; for a video frame sequence, the time dimension information represents the number of frames in the video frame sequence. The height and width information can be understood as the number of grids in the feature map in space. For example, h=16, w=16 means that the feature map has 16 grid positions in both the height and width directions.
[0044] The preset aggregation parameter can be understood as the merging factor of spatial downsampling, that is, how many adjacent feature vectors are merged into one visual token in the spatial dimension. This preset aggregation parameter does not change with the input image. For example, with h=16 and w=16, if the preset aggregation parameter is 1, the initial number of visual tokens is (1×16×16) / 1=256, meaning no spatial merging is performed, and each feature vector at each position in the feature map corresponds to one visual token. However, with the preset aggregation parameter 2, it means that two positions are merged in both the height and width dimensions, so the initial number of visual tokens is (1×16×16) / 4=64, meaning that four adjacent feature vectors are merged into one visual token.
[0045] In this embodiment, on the server side, the central processing unit and the image processing unit will collaboratively process the received data processing requests. Specifically, the central processing unit is used for sequence construction and numerical calculations, while the image processing unit is used for visual feature extraction.
[0046] Specifically, upon receiving a data processing request, the server-side Central Processing Unit (CPU) parses the target text and target image from the request. For the target text, the CPU calls a word segmenter to convert it into a text word sequence, which will subsequently form a multimodal input sequence together with the image placeholders.
[0047] For a target image, the central processing unit (CPU) calculates the initial number of visual terms corresponding to the target image based on the image's size information (t, h, w) and preset aggregation parameters for spatial merging. This calculation is performed at the CPU level, calculating the expected number of visual terms to be output by the visual encoder based on the image size information, without triggering actual visual feature extraction calculations, resulting in low computational overhead.
[0048] For example, if the data processing request is a text-image comprehension task, the target text is "Please describe the content of this image", the target image is a 448×448 pixel image, and its size information is time dimension 1, height dimension 28, and width dimension 28. The preset aggregation parameter is spatial merging unit 2. Then, the central processing unit calls the word segmenter to obtain 81 text words for the target text, and calculates the initial number of visual words for the target image as (1×28×28) / 4=196.
[0049] The data processing method provided in the embodiments of this specification completes the construction of the text word sequence and the pre-calculation of the initial visual word quantity at the central processing unit, so that the multimodal data has obtained a unified serialized representation before entering the image processing unit. This avoids the repeated execution of text processing and quantity calculation operations on the image processing unit side, and reduces the burden on the image processing unit.
[0050] Step 204: Determine the target number of visual words based on the target reduction parameters and the initial number of visual words, and determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words.
[0051] The target reduction parameter can be understood as the compression rate used to control the proportion of visual words retained. This parameter determines the extent to which the initial number of visual words is reduced. The target number of visual words can be understood as the actual number of visual words that need to be retained, calculated after applying the target reduction parameter to the initial number of visual words. The subsequent image processing unit will select the corresponding number of target visual features from the initial visual features according to this number.
[0052] The input sequence can be understood as an index sequence obtained by arranging the text word sequence and image placeholders in the original order. This sequence contains image placeholders corresponding to the number of target visual words, and each placeholder is used to mark the embedding position of a visual word in the sequence.
[0053] Image placeholders can be understood as index marks reserved in the input sequence for filling in visual features. Their position in the sequence indicates where the corresponding visual feature should be embedded.
[0054] In this embodiment, to avoid the inconsistency between the number of visual words obtained by the subsequent image processing unit after compressing the visual features and the number of image placeholders in the input sequence, which would lead to processing abnormalities, the central processing unit pre-calculates the target number of visual words after reduction based on the target reduction parameters, and adjusts the structure of the input sequence according to the target number of visual words, so that the number of image placeholders in the input sequence is consistent with the number of visual words after compression by the subsequent image processing unit.
[0055] Specifically, given the initial number of visual words, the target number of visual words is calculated based on preset target reduction parameters. This target number is the number of visual words to be retained in the initial visual features extracted by the subsequent image processing unit. Once the target number of visual words is determined, the central processing unit adjusts the structure of the input sequence accordingly. In practice, while preserving the original text word sequence and its relative order, the number of image placeholders in the sequence is reduced from the number of placeholders corresponding to the initial number of visual words to the number of placeholders corresponding to the target number of visual words.
[0056] In the reduced input sequence, the number of text terms remains the same, the number of image placeholders decreases, and the overall sequence length is shortened accordingly.
[0057] For example, if the target reduction parameter is 0.6, and the initial number of visual words calculated above is 196, then the target number of visual words is ceil(196×0.6)=118, where ceil is rounded up. The number of image placeholders in the input sequence can be reduced from 196 to 118, while the number of text words remains unchanged at 81. The total length of the input sequence is shortened from 277 to 199.
[0058] The data processing method provided in the embodiments of this specification predetermines the number of target visual words at the central processing unit and adjusts the number of image placeholders in the input sequence accordingly. This ensures that the structure of the input sequence is aligned with the number of visual words in the subsequent image processing units before entering the downstream large language model for processing, thus structurally eliminating the risk of processing anomalies caused by quantity mismatch.
[0059] In one or more embodiments of this specification, the central processing unit can construct an initial sequence based on the text word sequence and the initial number of visual words. After calculating the target number of visual words, the number of image placeholders in the initial sequence is adjusted to adapt the input sequence to subsequent visual compression. Specific implementation methods are as follows: Before determining the input sequence corresponding to the data processing request based on the number of target visual lexical units, the method further includes: Based on the central processing unit, an initial sequence is constructed according to the text word sequence corresponding to the target text and the image placeholders corresponding to the initial number of visual words; Determining the input sequence corresponding to the data processing request based on the number of target visual lexical units includes: Based on the central processing unit, the image placeholders corresponding to the target number of visual words are determined from the image placeholders corresponding to the initial number of visual words in the initial sequence using a uniform sampling method. The initial sequence is updated based on the image placeholders corresponding to the target number of visual lexical units to obtain an input sequence containing the text lexical sequence and the image placeholders corresponding to the target number of visual lexical units.
[0060] The initial sequence can be understood as a complete index sequence assembled according to the text word sequence corresponding to the target text and the image placeholders corresponding to the initial number of visual words, arranged in the original order. The number of image placeholders in the initial sequence is equal to the number of initial visual words, indicating the position and number of visual features to be embedded without compression.
[0061] The input sequence is the updated sequence obtained by uniformly filtering and reducing the image placeholders in the initial sequence. The number of image placeholders in the updated sequence is equal to the number of target visual words, while the text word sequence and its relative order remain unchanged.
[0062] Uniform sampling can be understood as selecting reserved positions at equal intervals within the location range of image placeholders, so that the reserved placeholders are evenly distributed throughout the entire image area.
[0063] Specifically, the central processing unit (CPU) first constructs an initial sequence containing text words and image placeholders based on the text word sequence corresponding to the target text and the image placeholders corresponding to the initial number of visual words, following the order of the target text and target image in the original request. Then, based on the number of target visual words, the CPU locates the image placeholder position intervals corresponding to the target image in the initial sequence. Within these intervals, it selects placeholder positions equal to the number of target visual words using uniform sampling, and removes the unselected placeholders from the sequence. After removal, the number and order of text words remain unchanged in the resulting input sequence, but the number of image placeholders is reduced to the number of target visual words, thus shortening the total sequence length.
[0064] In fact, in a multi-image scenario, if the data processing request contains two target images, and the initial visual word counts of the two target images are 196 and 784 respectively, and the target visual word counts of the two images are 118 and 471 respectively, then the central processing unit independently performs uniform sampling in the image placeholder position ranges of the two images respectively, and each retains the corresponding number of placeholders without affecting each other.
[0065] The data processing method provided in the embodiments of this specification involves the central processing unit reducing image placeholders by uniform sampling when constructing the input sequence. This ensures that the image placeholders retained in the input sequence are evenly distributed in space, avoiding the problem of poor subsequent feature filling position due to the concentration of image placeholders in a certain area.
[0066] In one or more embodiments of this specification, when adjusting the sequence, corresponding target index data is generated based on the position of the adjusted image placeholder in the input sequence, so that the visual features output by the subsequent image processing unit are accurately filled into the corresponding placeholder positions in the input sequence. Specific implementation methods are described below: After determining the input sequence corresponding to the data processing request based on the number of target visual lexical units, the method further includes: Based on the image placeholders corresponding to the target visual word count, target index data corresponding to the input sequence is obtained. The target index data includes offset data, which is determined according to the position of the image placeholders corresponding to the target visual word count in the input sequence. The image processing unit obtains the target visual word count based on the offset data.
[0067] The target index data can be understood as an index structure describing the positional information in the input sequence, containing offset data corresponding to each target image. The offset data can be understood as a pair of data indicating the start and end positions of image placeholders in the input sequence. The length of the interval between the start and end values of this pair is equal to the number of target visual words corresponding to that image. The offset data is determined based on the actual position of the image placeholders corresponding to the number of target visual words in the input sequence.
[0068] The image processing unit can obtain the number of target visual words based on this offset data, specifically by calculating the difference between the end position and the start position plus one.
[0069] Specifically, after the central processing unit reduces the image placeholders in the initial sequence according to the number of target visual terms and obtains the updated input sequence, it further generates corresponding offset data for the target image based on the actual arrangement position of each image placeholder in the input sequence. This offset data records the start and end position indices of the image placeholders of the target image in the input sequence.
[0070] For a single target image, the offset data directly reflects the start and end positions of the placeholder interval for that image. For multiple target images, each target image corresponds to a set of offset data, and the sets of offset data are arranged sequentially according to the order of the target images in the input sequence.
[0071] For example, if an input sequence has a length of 199, where text words occupy positions 0 to 80 and the image placeholders corresponding to the target image occupy positions 81 to 198, then the offset data of the target image is the start position 81 and the end position 198. After reading the offset data, the image processing unit can obtain the number of target visual words as 118 based on the offset data of the target image.
[0072] In practical applications, the target index data also includes the position matrix corresponding to the input sequence. The position matrix records the precise coordinates of each token (text token, visual token) in its respective modality. To maintain positional consistency, the position matrix corresponding to the multimodal rotation position code of the input sequence is filtered to extract the position codes of the retained tokens.
[0073] It should be noted that the offset data generated by the central processing unit is not limited to indicating the position range of image placeholders, but is also used as a reference by the image processing unit when verifying the number of features. That is, if the image processing unit detects that the number indicated by the offset data is inconsistent with the expected number derived based on the image size (i.e., the number of target visual features calculated above), it triggers a safety fallback logic, directly returning the uncompressed original features to avoid processing failure due to data anomalies.
[0074] The data processing method provided in the embodiments of this specification generates target index data containing offset data after adjusting the input sequence, so that the image processing unit can directly obtain the number of target visual words from the offset data without performing independent quantity calculation, thereby achieving quantity alignment between the central processing unit and the image processing unit at the data structure level.
[0075] Step 206: Extract visual features from the target image to generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features.
[0076] Visual feature extraction can be understood as the process of inputting a target image into a visual encoder, performing image segmentation, feature encoding, and other computational operations, and then outputting a feature vector. Initial visual features can be understood as the visual features output by the visual encoder, corresponding to the initial number of visual words; these visual features are multi-dimensional feature vectors. Target visual features can be understood as the features selected from the initial visual features, corresponding to the target number of visual words.
[0077] Specifically, the visual encoder in the image processing unit divides the target image into multiple image patches, and then performs encoding calculations through a multi-layer neural network to obtain the initial visual features corresponding to the initial number of visual words. In practical applications, for a single target image, the image processing unit obtains the number of target visual words corresponding to that image based on the offset data generated by the central processing unit. After the visual encoder extracts features from the image, it outputs the initial visual features. Then, based on the calculated number of target visual words, it selects the feature vector of the target visual word count from the initial visual features. For multiple target images, the image processing unit independently performs feature extraction and selection operations on each target image, and each image outputs the target visual features corresponding to its target visual word count.
[0078] For example, in one application scenario, the visual encoder in the image processing unit extracts features from a 448×448 pixel target image, outputting 196 initial visual features, each with a feature dimension of 4096. The offset data transmitted by the central processing unit indicates that the number of target visual features in the image is 118. Based on this, the image processing unit selects 118 target visual features from the 196 initial visual features, resulting in target visual features of [118, 4096].
[0079] The data processing method provided in the embodiments of this specification performs visual feature extraction and quantity filtering through an image processing unit, so that the number of output target visual features is strictly consistent with the number of image placeholders in the input sequence adjusted by the central processing unit.
[0080] In one or more embodiments of this specification, when determining target visual features from initial visual features, different selection strategies are used for different visual features. When a first number of first visual features and a second number of second visual features are selected respectively, the two are merged to generate the target visual feature. Specific implementation methods are described below: The step of extracting visual features from the target image to generate initial visual features corresponding to the initial number of visual words, and determining the target visual features corresponding to the target number of visual words from the initial visual features, includes: Based on the image processing unit, visual features are extracted from the target image to generate initial visual features corresponding to the initial number of visual words, and a first number of first visual features and a second number of second visual features are selected from the initial visual features. Based on the first number of first visual features and the second number of second visual features, target visual features corresponding to the target number of visual words are generated, wherein the sum of the first number and the second number is the target number of visual words.
[0081] The first visual feature can be understood as a visual feature with high importance selected from the initial visual features according to its importance score, carrying the semantic features of information-dense regions in the target image. The second visual feature can be understood as a visual feature selected in a spatially uniform distribution after excluding the first visual feature from the initial visual features, used to supplement the first visual feature's insufficient spatial coverage.
[0082] In practice, after feature extraction from the target image, the initial visual features include features with high information density (carrying key semantic information) and features that cover the global space and encompass the image background but have lower information density. When selecting features, relying on a single strategy is insufficient to simultaneously address the needs of both information density and spatial coverage. Selecting only highly important visual features based on importance may result in selections concentrated in a localized area of the image, losing background information; conversely, uniform sampling may miss important regions carrying key semantic information. Therefore, in this embodiment, two complementary selection strategies are employed to collaboratively determine the target visual features.
[0083] Specifically, the visual encoder in the image processing unit extracts features from the target image and outputs initial visual features for the initial number of visual words. The image processing unit obtains the number of target visual words corresponding to the target image based on the offset data transmitted by the central processing unit, and determines the allocation between the first number and the second number.
[0084] The image processing unit calculates the importance score of each initial visual feature, and selects a first number of initial visual features as the first visual features according to their importance scores from high to low. From the remaining unselected visual features, a second number of initial visual features are selected as the second visual features using an equidistant sampling method. The specific implementation method is as follows: The step of selecting a first number of first visual features and a second number of second visual features from the initial visual features includes: Calculate the importance score corresponding to each initial visual feature along the feature dimension, and determine the first number of first visual features from the initial visual features corresponding to the number of initial visual words based on the importance score; The first number of first visual features are excluded from the initial visual features corresponding to the initial number of initial visual lexical units to obtain supplementary visual features; A second number of second visual features are determined from the supplementary visual features using a uniform sampling method.
[0085] The importance score quantifies the semantic importance of each initial visual feature. Supplementary visual features can be understood as the remaining visual features after excluding a first number of initial visual features, serving as a candidate set for second visual features. The first visual features characterize information-dense regions in the target image; the second visual features characterize spatial coverage supplementation of the target image.
[0086] Specifically, the image processing unit calculates the importance score of each initial visual feature along the feature dimension. In this embodiment, the L2 norm or attention score of each initial visual feature can be calculated, and the value of the L2 norm or attention score is used as the importance score of the corresponding initial visual feature. The initial visual features are sorted from high to low according to their importance scores, and the first number of visual features in the ranking are selected as the first visual features.
[0087] Given a first visual feature, visual features that have been selected as the first visual feature are excluded from the initial visual features, and the remaining visual features are used as supplementary visual features. Based on the supplementary visual features, a second number of visual features are selected as the second visual features in an equidistant sampling manner. Through uniform sampling, it can be ensured that the selected second visual features cover as much of the target image as possible in terms of spatial location.
[0088] The determined first and second visual features are then sorted according to their original positions in the initial visual features and merged to generate the target visual features. This achieves the merging of a first number of first visual features and a second number of second visual features to generate the target visual features corresponding to the target number of visual words.
[0089] It should be noted that in this embodiment, the L2 norm of each initial visual feature is calculated along the feature dimension because for a visual feature tensor with shape [number of lexical units, feature dimension], the L2 norm can be calculated along different dimensions. If calculated along the feature dimension, the output shape is [number of lexical units], with each value corresponding to the overall information content of a lexical unit, suitable for ranking and filtering lexical units by importance; if calculated along the lexical unit dimension, the output shape is [feature dimension], reflecting the global activation intensity of each feature dimension, which cannot be used to distinguish the importance of different lexical units. Therefore, this embodiment explicitly calculates along the feature dimension to obtain an independent measure of the importance of each visual feature.
[0090] For example, in one application scenario, the initial number of visual lexical units is 196. The L2 norm of each initial visual feature is calculated as an importance score, resulting in 196 score values corresponding to each initial visual feature. The 108 initial visual features with the highest scores are selected as the first visual features. These first visual features usually cover key semantic regions such as faces and text in the image.
[0091] Of the remaining 88 unselected visual features, 10 are selected as second visual features with equal step sizes to cover the edges and background areas of the image; the 108 first visual features are merged with the 10 second visual features to obtain 118 target visual features.
[0092] In this embodiment, if the importance score is calculated based on the L2 norm, the calculation can be performed without relying on attention weights, which is applicable to visual encoder architectures that do not have CLS flags and whose attention weights are not available, and has good model compatibility.
[0093] The data processing method provided in the embodiments of this specification, through different feature selection methods, enables the final retained target visual features to both carry the semantics of information-dense areas in the image and maintain coverage of the entire image spatial structure, avoiding insufficient perception of the overall layout of the image by the model due to excessive concentration of features in a certain local area.
[0094] In one or more embodiments of this specification, the features output by the visual encoder at different network depths have different semantic granularities. For example, shallow features typically contain rich spatial texture information, while deep features typically carry high-order semantic information. Based on this, this embodiment selects the last network layer in the visual encoder with the richest semantic information as the target layer, and performs subsequent feature selection based on the features output by this target layer. Specific implementation methods are described below: The step of extracting visual features from the target image to generate initial visual features corresponding to the initial number of visual words, and determining the target visual features corresponding to the target number of visual words from the initial visual features, includes: Visual features are extracted from the target image based on the visual encoder in the image processing unit to obtain the initial visual features corresponding to the target level and the number of initial visual words; By selecting features from the initial visual features of the initial number of visual words, the retained index data of the target number of visual words is determined, wherein the retained index data is the word position corresponding to each retained visual feature; Based on the retained index data, feature vectors for corresponding word positions are extracted from the initial visual features of the initial number of initial visual words to obtain the target visual features corresponding to the target number of visual words.
[0095] The visual encoder comprises multiple network layers, each outputting features corresponding to its own layer. In this embodiment, the last network layer is selected as the target layer because its merged hidden state output integrates shallow spatial texture and deep semantic information, containing the richest semantic representation. Therefore, feature selection based on this layer can more accurately identify information-dense regions.
[0096] The initial visual features can be understood as the merged hidden state output by the visual encoder, and the number of its lexical units is equal to the number of initial visual lexical units. The retained index data can be understood as the set of indices obtained by selecting features from the initial visual features according to the above feature selection method, which indicates which visual feature lexical positions should be retained, and the number of indices in the retained index data is equal to the number of target visual lexical units.
[0097] The target visual feature can be understood as the visual feature obtained after extracting the corresponding position feature vector from the initial visual feature based on the retained index data, and its number of lexical units is equal to the number of target visual lexical units.
[0098] Specifically, the visual encoder in the image processing unit performs visual feature extraction on the target image, selects the last network layer as the target layer, and obtains the initial visual features output by the target layer. The number of words in the initial visual features is the initial number of visual words.
[0099] Feature selection is performed on the initial visual features. For example, the L2 norm of each initial visual feature can be calculated along the feature dimension as an importance score. Based on the importance score, a first number of first visual features to be retained are determined, and a second number of second visual features are obtained from the remaining visual features by uniform sampling. Based on the word positions corresponding to the first and second visual features, retained index data is generated.
[0100] Based on the retained index data, feature vectors for corresponding word positions are extracted from the initial visual features to obtain target visual features, where the number of words in the target visual features is equal to the number of target visual words.
[0101] For example, the last network layer of a visual encoder typically outputs the merged hidden state. The L2 norm is calculated for the initial visual features output by this layer to achieve feature selection and determine the number of target visual words in the retained index data. For example, if the number of target visual words is 118, the retained index data will contain 118 index positions.
[0102] After completing the L2-norm-based feature selection, the retained target visual features will be used as shared high-quality visual representations in subsequent main task branches and / or deep stacked structures. The main branch is responsible for performing core tasks (such as target classification or semantic discrimination), while the deep stack branch serves as an auxiliary supervision or multi-layer prediction fusion function. By uniformly applying the target visual features to both the main and deep stack branches, the input feature distributions of the main and deep stack branches are consistent, facilitating collaborative optimization within the same normalized feature space.
[0103] The data processing method provided in the embodiments of this specification generates retained index data based on the target level, thereby accurately extracting target visual features from the initial visual features based on the retained index data, and the target visual features can be applied to different subsequent task branches.
[0104] Step 208: Fill the target visual features corresponding to the number of target visual words into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
[0105] The target sequence can be understood as a continuous vector sequence composed of text word sequences and target visual features, obtained by filling in the target visual features according to the positional order of the image placeholders in the input sequence. This sequence can be directly fed into the downstream model for subsequent processing. The processing result can be understood as the target sequence being input into the downstream target model, which generates a response output to the data processing request based on the input text and image content.
[0106] In the above steps, the central processing unit completes the structural adjustment of the input sequence, and the image processing unit completes the extraction and filtering of visual features. The image placeholders in the input sequence are index placeholders and do not contain actual vector representations. By filling the corresponding image placeholder positions with the target visual feature vectors output by the image processing unit, a continuous vector sequence that can be processed by the target model is generated. In this embodiment, the target model can be any machine learning model capable of processing feature sequences, and is not limited here.
[0107] Specifically, the central processing unit (CPU) arranges the target visual features of the target image according to their original spatial order in the initial visual features and fills them into the position intervals of the corresponding image placeholders in the input sequence. After the filling operation is completed, each image placeholder in the input sequence is replaced with its corresponding visual feature, thus forming a complete target sequence based on the previous text word sequence and the filled target visual features. Taking a multimodal large model as an example, the CPU inputs the target sequence into the multimodal large model, which then performs inference based on the target sequence to generate the processing result for the data processing request.
[0108] For example, the input sequence is 199 in length, with image placeholders occupying positions 81 to 198. The target visual features output by the image processing unit are 118 feature vectors, each with a dimension of 4096. The central processing unit fills the 118 feature vectors one by one into positions 81 to 198 according to the position range of the image placeholders indicated by the offset data, replacing the original image placeholders. The target text has been converted into an embedding vector and filled into positions 0 to 80 before being processed by the image processing unit, thus forming a complete target sequence with a shape of [199, 4096].
[0109] When the data processing request is an image description request, the target sequence is sent to the multimodal large model, which then calculates and outputs descriptions of the target text and the target image.
[0110] By precisely filling in the target visual features obtained by the image processing unit according to the position order of the image placeholders in the input sequence, the orderly fusion of visual features and text features in the sequence is achieved.
[0111] In one or more embodiments of this specification, when filling the image placeholders of the input sequence with target visual features, the position is located based on the target index data. Specific implementation methods are described below: The step of filling the target visual features corresponding to the number of target visual words into the image placeholders of the input sequence to obtain the target sequence includes: Based on the target index data corresponding to the input sequence, the target visual features corresponding to the number of target visual words are filled into the image placeholders of the input sequence to obtain the target sequence.
[0112] In fact, after the input sequence is adjusted, the actual position of the image placeholder in the sequence has changed compared to the initial sequence. Therefore, accurate position index information is needed to guide the filling of visual features, ensuring that each target visual feature is filled into the correct position of its corresponding image placeholder, and avoiding mismatch problems caused by incorrect filling position.
[0113] Specifically, based on the target index data corresponding to the input sequence, the offset data of each target image contained therein is obtained to determine the position of the image placeholder for each target visual feature in the input sequence. The offset data records the start and end positions of the image placeholders in the input sequence. Using this start position as the filling starting point, each target visual feature is filled into its corresponding image placeholder position one by one according to its spatial order. After filling is complete, all image placeholders are replaced with their corresponding target visual features.
[0114] The data processing method provided in the embodiments of this specification determines the filling interval of the feature vector of each target image in the input sequence based on the offset information in the target index data, so that the filling position of the visual features corresponds precisely to the actual position of the image placeholder in the input sequence, ensuring that the arrangement order of visual features and text features in the filled target sequence is consistent with the structure of the input sequence.
[0115] See Figure 3 , Figure 3 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.
[0116] Specifically, this specification provides a visual token compression acceleration method designed specifically for the sglang inference engine. This method splits visual token compression into two collaborative stages: one on the CPU (Central Processing Unit) side and the other on the GPU (Graphics Processing Unit). The CPU side is responsible for quantity planning and sequence structure adjustment before the request enters the GPU, while the GPU side is responsible for performing actual feature filtering after visual feature extraction. Both the CPU and GPU sides are based on the same target number of visual tokens, structurally ensuring that the number of compressed visual tokens strictly matches the number of image placeholders in the input sequence, thereby eliminating the request failure problem caused by quantity mismatch in traditional solutions.
[0117] The following combination Figure 3 This data processing method will be described in detail.
[0118] When a data processing request is received from the caller, and the request contains multimodal data such as text and images, the server can reduce the number of visual tokens by working together with the central processing unit (request preprocessing results) and the image processing unit (ViT output stage) to improve the inference efficiency of subsequent model processing.
[0119] In practice, before the data processing request enters the GPU, the CPU needs to calculate the number of target visual terms and reduce the image placeholders in the initial sequence.
[0120] When a data processing request arrives at the server, the central processing unit (CPU) parses the target text and target image from the request. For the target text, the CPU calls a word segmenter to convert it into a sequence of text tokens. For each target image, the CPU reads the grid size information (t, h, w) of the target image from the request metadata, combines it with preset aggregation parameters (i.e., spatial merging units), and calculates the grid size using the formula n_orig=(t×h×w) / preset aggregation parameters. 2 The initial number of visual words, n_orig, is calculated. This number represents the total number of visual words that the visual encoder should theoretically output if it performs feature extraction on the target image.
[0121] The central processing unit performs compression calculations on the initial number of visual words based on the target reduction parameter `retain`. Specifically, it determines the target number of visual words `K` according to the rounding-up rule `K = ceil(retain × n_orig)`. The target reduction parameter ranges from greater than 0 to less than or equal to 1; a value of 1 indicates that no compression is performed.
[0122] The central processing unit constructs an initial sequence based on the text lexical sequence corresponding to the target text, the image placeholders corresponding to the initial number of visual lexical units, and their respective arrangement order. In this initial sequence, the number of image placeholders is equal to the initial number of visual lexical units.
[0123] In the initial sequence, the image placeholder intervals corresponding to each target image are located. Within each image placeholder interval, uniform sampling is used, that is, retaining positions are selected at equal intervals within the position interval of the image placeholder. Specifically, K positions corresponding to the target visual words are selected and marked as retaining positions, forming a retaining mask (keep_mask). The remaining image placeholders are removed from the initial sequence to obtain the updated input sequence. In the input sequence, the number and order of text words remain unchanged, the number of image placeholders is reduced to the number of target visual words, and the total sequence length is shortened accordingly.
[0124] The position matrix of the multimodal rotation position encoding and the position offset data of each target image are determined based on the input sequence. The position matrix records the position information of each word in the multimodal space, while the offset data records the start and end positions of the image placeholder of each target image in the input sequence. The length of the interval is equal to the number of target visual words corresponding to the target image.
[0125] After the CPU completes the sequence structure adjustment and offset data generation, the data processing request and offset data are sent to the scheduler. The scheduler enqueues the request, waiting for processing by the GPU. During the scheduler enqueueing phase, the offset data is passed to the GPU as part of the processing task, allowing the GPU to directly read the offset information from the task data when processing the task.
[0126] The GPU is used to extract features through a visual encoder and select features based on the number of target visual words. Specifically, the GPU needs to read the number of target visual words from the offset data, calculate the importance score based on the L2 norm, and perform a two-stage feature selection.
[0127] When processing a target image, the image processing unit first reads the number of target visual words from the offset data corresponding to the target image. That is, the number of target visual words is determined by the CPU, and the GPU is responsible for filtering according to the number of target visual words, thus structurally eliminating the possibility of inconsistency between the corresponding numbers on the CPU and the GPU.
[0128] The offset data records the start position off_start and end position off_end of the image placeholder in the input sequence. The image processing unit can obtain the number of target visual words K=off_end-off_start+1 by calculating the difference between the end position and the start position plus one.
[0129] See Figure 4 , Figure 4 A flowchart illustrating the visual feature selection process in a data processing method provided in one embodiment of this specification is shown.
[0130] The visual encoder (ViT) in the image processing unit divides the target image into multiple image blocks, and after encoding and calculation through multiple network layers, outputs the initial visual features of the initial number of visual words.
[0131] For initial visual features, importance scoring is typically performed using existing methods that rely on the attention weights of the CLS token output by the ViT terminal to assess the importance of each visual token. However, the ViT architecture for multimodal large models no longer uses CLS tokens; furthermore, for performance optimization, the attention calculation kernel does not expose the intermediate attention weight matrix. These two limitations together render attention weight-based metrics ineffective in real-world production environments.
[0132] Therefore, in this embodiment, the L2 norm of each initial visual feature is calculated along the feature dimension, and the value of the L2 norm is used as the importance score corresponding to each initial visual feature, thus obtaining n_orig importance scores. When calculating the L2 norm, this importance score does not depend on attention weights, and therefore is suitable for visual encoder architectures that do not contain CLS flags and where attention weights are unavailable.
[0133] In fact, as visual features propagate through the network layer by layer, features carrying effective semantic information continuously accumulate new information through residual connections, and the L2 norm of their hidden layer representation increases with the depth of the layer; while redundant features receive less attention in each layer, their hidden layer representation is hardly updated, and their norm remains at the initial level. Therefore, the L2 norm can be used as an effective measure of the information density of each visual lexical unit.
[0134] The image processing unit determines the allocation of the first quantity k_dom and the second quantity k_ctx based on the number of target visual words K. The first quantity k_dom is the number of target visual words K minus the second quantity k_ctx. The second quantity k_ctx is determined by the product of the supplementary sampling ratio ctx and the initial number of visual words n_orig.
[0135] When selecting Dominant tokens, the first number of visual features are selected from high to low importance scores. These visual features carry the semantic features of information-dense areas in the target image, such as key areas like faces, text, and object edges.
[0136] Next, contextual token selection is performed. Specifically, the first number of visual features already selected are excluded from the initial visual features, and the remaining visual features are used as a supplementary visual feature candidate set. From this set, a second number of visual features are selected using an equidistant sampling method. These visual features are used to supplement spatial coverage and prevent the loss of image background information due to the concentration of important words in a certain local area.
[0137] The first number of selected visual features are merged with the second number of visual features, and then rearranged according to the position order of each visual feature in the original sequence to obtain the target visual features corresponding to the target number of visual words.
[0138] In practical applications, in order to effectively preserve key information in the compressed visual features, this embodiment selects features based on the merged hidden state output by the last network layer of the visual encoder. That is, the image processing unit obtains the initial visual features corresponding to the target layer (i.e., the last network layer), which are the merged hidden state (out_hidden) output by PatchMerger (feature merging module).
[0139] The L2 norm is calculated along the feature dimension of the initial visual feature to obtain an importance vector of length equal to the number of initial visual words. Each value in this importance vector quantifies the semantic importance of the visual feature at the corresponding word position. Feature selection is then performed based on this importance vector to determine the target visual feature with the target number of visual words.
[0140] Optionally, in other embodiments of this specification, when feature selection is performed based on the merged hidden state output by the last network layer of the visual encoder to obtain a first number of first visual features and a second number of second visual features, retained index data can be generated according to the lexical positions of the first and second visual features. The retained index data records the position index of the visual features to be retained in the original sequence, and the number of indexes is equal to the number of target visual lexical units.
[0141] After obtaining the preserved index data, the image processing unit extracts feature vectors for corresponding word positions from the initial visual features corresponding to each level of the visual encoder. Each level independently performs the index extraction operation, obtaining compressed visual features with a word count equal to the target visual word count. It should be noted that "each level" here refers to the output of each layer corresponding to different network depths within the visual encoder. The features at different levels all have a word count equal to the initial visual word count, but the feature dimensions may differ depending on the level.
[0142] Each level shares the same retained index data, achieving unified slicing. This means that visual features retained at a certain location are retained across all levels; conversely, visual features discarded at a certain location are discarded across all levels. This operation ensures that the visual features retained after compression at each level remain strictly aligned spatially, providing a spatially consistent foundation for subsequent multi-level feature fusion. Simultaneously, this unified slicing operation ensures that all components of the Main and Deepstack branches are synchronously retained using the same set of indices. The Main branch performs core tasks (such as object classification or semantic discrimination), while the Deepstack branch provides auxiliary supervision or multi-level prediction fusion functions.
[0143] In practice, since the unified slicing operation selects along the token axis and does not modify the hidden layer dimension, the Main branch and the Deepstack branch receive the same set of preserved visual tokens after unified slicing. Based on this, if it is necessary to split the input features of the Deepstack branch along the hidden layer dimension, the independent compressed features corresponding to the shallow, middle, and deep layers can be restored separately and sent to the corresponding sub-modules in the Deepstack for processing. This hidden layer dimension splitting logic is executed within the Deepstack branch and is independent of the processing path of the Main branch, without interfering with each other. The Main branch does not need this splitting operation and can directly receive the concatenated complete fused features or features of a specific level for the main task inference.
[0144] In practice, since the index data is determined on the merged hidden state (i.e., out_hidden) of the visual encoder output, and out_hidden itself will be used as the shared input for the subsequent Main branch and Deepstack branch, this unified slicing operation naturally ensures that the two branches receive the same set of retained features, thus ensuring the consistency of the input feature distribution.
[0145] The compressed visual features corresponding to each level are concatenated along the feature dimension to generate the fused target visual features. After concatenation, the number of terms in the target visual features is still equal to the number of target visual terms, but it simultaneously contains shallow spatial texture information and deep semantic information, enriching the information dimension of the visual representation and improving the ability of subsequent models to understand image content.
[0146] In practical applications, the compressed visual features at each level can be normalized according to their respective feature dimensions before stitching to eliminate the impact of differences in feature magnitude between different levels on subsequent model processing. Furthermore, the stitched target visual features can be further mapped to a specific dimensional space through a linear transformation layer to match the input requirements of different downstream branch models.
[0147] It should be noted that if the image processing unit detects that the number of offset data indicated is inconsistent with the expected number derived from the image grid size, a safety fallback is triggered, and the uncompressed original features are returned directly to avoid processing failure due to data anomalies.
[0148] After GPU-side processing, a morphological alignment check is performed to ensure that the number of rows of the target visual features output by the image processing unit equals the number of image placeholders in the input sequence. In other words, the check verifies that the number of image placeholders retained in the input sequence and the number of target visual features output by the image processing unit are both the number of target visual words. If they match, the check passes, and the target visual features can be safely passed downstream; if they do not match, a fallback logic is triggered, and the uncompressed original features are returned.
[0149] After the morphological alignment verification is passed, the central processing unit determines the filling position of the target visual features of each target image in the target sequence based on the offset data corresponding to the input sequence. After the filling operation is completed, the image placeholders in the input sequence are replaced with the corresponding visual features, thus forming a complete target sequence. In this target sequence, text word embeddings and visual features are arranged alternately according to their original order in the input sequence, and can be directly fed into the multimodal large model for forward computation.
[0150] In this embodiment, the CPU samples images uniformly according to their placeholder positions, while the GPU sorts each initial visual feature based on its importance score. The number of images retained in each image is determined uniformly by the CPU, and the GPU is only responsible for filtering according to that number. This ensures that the number of rows of the target visual features output by the GPU is always equal to the number of image placeholders in the input sequence.
[0151] This embodiment uses a 448×448 pixel target image as an example to illustrate the end-to-end data morphology changes before and after visual word compression. The image's grid dimensions are time dimension 1, height dimension 28, and width dimension 28. The preset aggregation parameter is spatial merging unit 2, resulting in an initial number of 196 visual words. Simultaneously, the target text, after word segmentation, yields 81 text words. With a target reduction parameter of 0.6, the target number of visual words is 196 × 0.6 rounded up, which is 118. This means that 78 visual words need to be reduced from the initial 196. The following section compares the data morphology differences between the baseline scheme and the compression scheme according to each stage of the data processing chain.
[0152] In the sequence construction stage, the initial sequence constructed by the CPU in the basic scheme (i.e., the scheme without feature compression) contains 81 text terms and 196 image placeholders, with a total sequence length of 277. In the compression scheme, the CPU retains 118 image placeholders and removes 78 from the 196 image placeholders using a uniform sampling method, while keeping the number of text terms unchanged at 81. The total length of the updated input sequence is reduced to 199.
[0153] During the visual feature extraction stage, the workload of the GPU-based visual encoder in performing feature extraction on the target image is exactly the same regardless of whether compression is enabled. The target image is divided into 784 image blocks and input into the visual encoder. The initial visual features output are all in the shape [196, 4096], meaning 196 visual features, each with a feature dimension of 4096. This stage does not incur additional computational overhead or data format changes due to subsequent compression operations.
[0154] During the visual feature selection phase, the compression scheme takes effect. The image processing unit calculates the importance score of each visual feature based on the L2 norm and performs feature selection according to the target visual terminology of 118 transmitted from the CPU. It selects 118 feature vectors from the initial 196 feature vectors as target visual features, outputting a feature matrix of [118, 4096], which reduces the number of features by 78 compared to the [196, 4096] matrix of the basic scheme. The basic scheme does not perform any selection at this stage and outputs the complete 196 feature vectors.
[0155] In the feature filling stage, the total length of the input sequence in the basic scheme is 277, which includes 196 image placeholders. The 196 feature vectors output by the visual encoder are filled into these image placeholder positions one by one, forming a hybrid embedding sequence of [277, hidden layer dimension]. In the compression scheme, the total length of the input sequence is 199, which includes 118 image placeholders. The 118 feature vectors selected by the image processing unit are filled into these placeholder positions, forming a hybrid embedding sequence of [199, hidden layer dimension]. Compared with the basic scheme, the sequence length is reduced by 78.
[0156] In the target model processing stage, the basic scheme requires the model to process sequences with a length of 277 lexical units, while the compression scheme requires the model to process sequences with a length of 199 lexical units, reducing the sequence length by 78 units.
[0157] In practical applications, since the computational cost of the pre-filling stage is proportional to the sequence length, and the key-value cache usage is also proportional to the sequence length, the compression scheme achieves a reduction of approximately 28.5% in pre-filling computational cost and a reduction of approximately 28.5% in key-value cache usage compared to the basic scheme.
[0158] In summary, the computational workload of the visual encoder is the same on both paths. The compression scheme is introduced after the visual encoder has completed feature extraction. The inference acceleration benefit comes from the reduction in sequence length, and the corresponding key-value cache usage also decreases linearly. The data processing method provided in this embodiment achieves inference acceleration by reducing the input sequence length while ensuring the quality of visual features, based on visual lexical compression.
[0159] In practical applications, the data processing method provided in this embodiment achieves a zero-intrusion injection mechanism and deployment control in its implementation. See also... Figure 5 , Figure 5 A flowchart illustrating an implementation of a data processing method provided in one embodiment of this specification is shown.
[0160] In this embodiment, a hook module is used as the injection point entry point for the aforementioned data processing method. The specific injection process is as follows: During the plugin package installation phase, the installation script writes a pre-written hook file to the specified boot directory. This hook file contains the execution logic for the aforementioned data processing methods, but it is not yet activated at this stage.
[0161] Before starting the actual inference engine process, the service startup script checks the environment variable "SGLANG_VISIONZIP_RETAIN". If this environment variable exists and its value is less than 1.0, the startup script appends the aforementioned boot directory to the hook module's module search path. If this environment variable does not exist or its value is greater than or equal to 1.0, the boot directory is not appended.
[0162] When the interpreter starts, it automatically searches for hook modules. If the boot directory has been added to the hook module search path, the interpreter can find and import the hook file in that boot directory. Upon import, the hook file immediately executes the activation function defined within it. This activation function uses monkey patching (a technique that dynamically replaces class methods or function references at runtime) to replace the target class methods within the inference engine, injecting the data processing methods described earlier into the inference engine's request processing chain.
[0163] Since the hook module is executed automatically during the interpreter initialization phase, the above data processing method is installed before any business code of the inference engine runs, without requiring any modification to the source code of the inference engine.
[0164] It's worth noting that the deployment architecture of an inference engine typically includes multiple independent child processes, which also trigger the loading of hook modules upon startup. To prevent data processing methods in the aforementioned hook files from affecting unrelated processes, import statements for external dependencies in the hook files are wrapped in exception handling logic. For processes that do not hold inner classes of the inference engine, these import statements will trigger import errors and be caught, thus skipping the execution logic of the aforementioned data processing methods, and the behavior of that process remains unaffected.
[0165] The activation and deactivation of the above data processing method are uniformly controlled by a single environment variable "SGLANG_VISIONZIP_RETAIN", which is used to specify the retention ratio of visual lexical units (i.e., the target reduction parameter in the above embodiments).
[0166] When this environment variable is not set or its value is greater than or equal to 1.0, the service startup script does not inject the boot directory into the search path of the hook module, the hook module is not loaded, the above data processing method is not executed, and the compression function is turned off.
[0167] When the value of this environment variable is less than 1.0, the compression function is automatically enabled. At this time, the aforementioned hook module loading process is triggered, and the data processing method will be applied to the processing chain of the inference engine. The value of this environment variable also determines the retention ratio of visual words in each image, and the central processing unit uses this value as the basis for performing compression calculations when calculating the target number of visual words.
[0168] In addition, this embodiment provides optional supporting environment variables for fine-tuning. Specifically, the environment variable for supplementary sampling ratio is adjusted through the supporting environment variables to control the second quantity. All of the above-mentioned supporting environment variables are set with reasonable default values and take effect automatically when not explicitly configured.
[0169] To revert the compression feature, simply remove the "SGLANG_VISIONZIP_RETAIN" environment variable and restart the service to restore the system to its uncompressed state. The entire rollback process requires no modification to any code, configuration files, or redeployment of the plugin package, resulting in extremely low operational costs.
[0170] The data processing method provided in this specification does not require modification of the sglang source code, nor does it require users to modify the service entry code or inherit and replace specific model classes. All customized logic is injected at runtime through a hook mechanism, decoupled from the framework source code. Furthermore, this data processing method can operate independently of sglang's existing acceleration and optimization mechanisms, and can be used in combination without interference. The entire mechanism is encapsulated as an independent plugin sub-package, and upgrades or rollbacks can be completed through the installation and version management of the plugin package. All multimodal inference services deployed based on the same plugin package can enable compression through the same set of environment variables, achieving unified management and rapid deployment across services.
[0171] The L2-norm-based importance metric replaces the attention weight-based approach, avoiding the limitations of not including CLS flags and not exposing attention weights, thus making the compression scheme effective on various multimodal model architectures. Furthermore, the two-stage selection strategy of Dominant and Contextual balances information density and spatial coverage, avoiding semantic omissions or spatial deficiencies that might result from a single selection strategy.
[0172] By using a quantity alignment mechanism where the CPU determines the K value and encodes it into the offset data, and the GPU reads the K value from the offset data, the possibility of inconsistency between the two ends is eliminated at the data structure level, thus solving the request failure problem caused by changes in the number of tokens in the sglang inference engine architecture.
[0173] Visual token compression can be achieved by setting target reduction parameters, which can significantly reduce request response latency and reduce unit request cost. It can be used in real-time audit scenarios that are more sensitive to latency, such as live real-time frame audit and risk instant blocking, while maintaining accuracy. Furthermore, with the reduction of key-value cache usage, the freed-up GPU memory can be used to increase the concurrency limit, accommodate longer contexts, or use the spare GPU memory for other model deployments, thereby improving the overall cluster utilization.
[0174] This specification also provides a data processing system, including a central processing unit and an image processing unit, wherein... The central processing unit is configured to obtain the initial number of visual words in the target image in the data processing request; determine the target number of visual words based on the target reduction parameters and the initial number of visual words; and determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words. The image processing unit is used to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The central processing unit is further configured to fill the target visual features corresponding to the number of target visual lexical units into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
[0175] The central processing unit is the CPU in the above embodiments, and the image processing unit is the GPU in the above embodiments. Specific implementation methods can be found in the above embodiments, and are not limited thereto.
[0176] The data processing system provided in this specification eliminates the possibility of inconsistency between the two ends at the data structure level by using a number alignment mechanism where the CPU determines the number of target visual words and encodes them into offset data, and the GPU reads the number of target visual words from the offset data. This solves the problem of request failure caused by changes in the number of tokens in the sglang inference engine architecture.
[0177] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 6 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: The quantity acquisition module 602 is configured to acquire the initial number of visual lexical units of the target image in the data processing request; The sequence determination module 604 is configured to determine the target number of visual words based on the target reduction parameters and the initial number of visual words, and to determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words. The feature selection module 606 is configured to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The sequence filling module 608 is configured to fill the target visual features corresponding to the number of target visual words into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
[0178] Optionally, the quantity obtaining module 602 is further configured to: Receive a data processing request and obtain the target text and target image in the data processing request; The central processing unit obtains the text lexical sequence corresponding to the target text, and obtains the initial number of visual lexical units of the target image based on the size information of the target image and preset aggregation parameters.
[0179] Optionally, the sequence determination module 604 is further configured to: Based on the central processing unit, an initial sequence is constructed according to the text word sequence corresponding to the target text and the image placeholders corresponding to the initial number of visual words; Based on the central processing unit, the image placeholders corresponding to the target number of visual words are determined from the image placeholders corresponding to the initial number of visual words in the initial sequence using a uniform sampling method. The initial sequence is updated based on the image placeholders corresponding to the target number of visual lexical units to obtain an input sequence containing the text lexical sequence and the image placeholders corresponding to the target number of visual lexical units.
[0180] Optionally, the feature selection module 606 is further configured to: Based on the image processing unit, visual features are extracted from the target image to generate initial visual features corresponding to the initial number of visual words, and a first number of first visual features and a second number of second visual features are selected from the initial visual features. Based on the first number of first visual features and the second number of second visual features, target visual features corresponding to the target number of visual words are generated, wherein the sum of the first number and the second number is the target number of visual words.
[0181] Optionally, the feature selection module 606 is further configured to: Calculate the importance score corresponding to each initial visual feature along the feature dimension, and determine the first number of first visual features from the initial visual features corresponding to the number of initial visual words based on the importance score; The first number of first visual features are excluded from the initial visual features corresponding to the initial number of initial visual lexical units to obtain supplementary visual features; A second number of second visual features are determined from the supplementary visual features using a uniform sampling method.
[0182] Optionally, the feature selection module 606 is further configured to: Visual features are extracted from the target image based on the visual encoder in the image processing unit to obtain the initial visual features corresponding to the target level and the number of initial visual words; By selecting features from the initial visual features of the initial number of visual words, the retained index data of the target number of visual words is determined, wherein the retained index data is the word position corresponding to each retained visual feature; Based on the retained index data, feature vectors for corresponding word positions are extracted from the initial visual features of the initial number of initial visual words to obtain the target visual features corresponding to the target number of visual words.
[0183] Optionally, the sequence determination module 604 is further configured to: Based on the image placeholders corresponding to the target visual word count, target index data corresponding to the input sequence is obtained. The target index data includes offset data, which is determined according to the position of the image placeholders corresponding to the target visual word count in the input sequence. The image processing unit obtains the target visual word count based on the offset data.
[0184] Optionally, the sequence filling module 608 is further configured to: Based on the target index data corresponding to the input sequence, the target visual features corresponding to the number of target visual words are filled into the image placeholders of the input sequence to obtain the target sequence.
[0185] The data processing apparatus provided in one embodiment of this specification performs reduction calculations on the initial visual word data corresponding to the target image according to the target reduction parameters, determines the number of target visual words, and compresses the number of image placeholders in the input sequence to the number of target visual words. Image placeholders are reserved according to the compressed number of target visual words to construct an input sequence with a shorter sequence length. When visual features are extracted from the target image and initial visual features are generated, target visual features with the number of target visual words are selected from the initial visual features corresponding to the number of initial visual words, thereby reducing the number of visual features. The target visual features are then filled with the corresponding image placeholders in the input sequence to obtain the target sequence. This reduces the length of the sequence that needs to be processed subsequently, thereby reducing the amount of computation, key-value cache usage, and memory access overhead during the inference stage. This improves the inference throughput of a single card and reduces request response latency, alleviating the performance bottleneck caused by shuffling visual words in high-concurrency scenarios.
[0186] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.
[0187] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.
[0188] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0189] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0190] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.
[0191] The processor 720 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.
[0192] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.
[0193] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0194] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.
[0195] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0196] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.
[0197] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0198] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0199] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.
[0200] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0201] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Obtain the initial number of visual lexical units of the target image in the data processing request; Based on the target reduction parameters and the initial number of visual words, the target number of visual words is determined, and the input sequence corresponding to the data processing request is determined based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words; Visual features are extracted from the target image to generate initial visual features corresponding to the initial number of visual words, and target visual features corresponding to the target number of visual words are determined from the initial visual features. The target visual features corresponding to the number of target visual words are filled into the image placeholders of the input sequence to obtain the target sequence, and the processing result corresponding to the data processing request is obtained based on the target sequence.
2. The method as described in claim 1, wherein obtaining the initial number of visual lexical units of the target image in the data processing request includes: Receive a data processing request and obtain the target text and target image in the data processing request; The central processing unit obtains the text lexical sequence corresponding to the target text, and obtains the initial number of visual lexical units of the target image based on the size information of the target image and preset aggregation parameters.
3. The method as described in claim 2, further comprising, before determining the input sequence corresponding to the data processing request based on the number of target visual lexical units: Based on the central processing unit, an initial sequence is constructed according to the text word sequence corresponding to the target text and the image placeholders corresponding to the initial number of visual words; Determining the input sequence corresponding to the data processing request based on the number of target visual lexical units includes: Based on the central processing unit, the image placeholders corresponding to the target number of visual words are determined from the image placeholders corresponding to the initial number of visual words in the initial sequence using a uniform sampling method. The initial sequence is updated based on the image placeholders corresponding to the target number of visual lexical units to obtain an input sequence containing the text lexical sequence and the image placeholders corresponding to the target number of visual lexical units.
4. The method as described in claim 1, wherein extracting visual features from the target image to generate initial visual features corresponding to the initial number of visual words, and determining the target visual features corresponding to the target number of visual words from the initial visual features, comprises: Based on the image processing unit, visual features are extracted from the target image to generate initial visual features corresponding to the initial number of visual words, and a first number of first visual features and a second number of second visual features are selected from the initial visual features. Based on the first number of first visual features and the second number of second visual features, target visual features corresponding to the target number of visual words are generated, wherein the sum of the first number and the second number is the target number of visual words.
5. The method of claim 4, wherein selecting a first number of first visual features and a second number of second visual features from the initial visual features comprises: Calculate the importance score corresponding to each initial visual feature along the feature dimension, and determine the first number of first visual features from the initial visual features corresponding to the number of initial visual words based on the importance score; The first number of first visual features are excluded from the initial visual features corresponding to the initial number of initial visual lexical units to obtain supplementary visual features; A second number of second visual features are determined from the supplementary visual features using a uniform sampling method.
6. The method as described in claim 1 or 4, wherein extracting visual features from the target image to generate initial visual features corresponding to the initial number of visual words, and determining the target visual features corresponding to the target number of visual words from the initial visual features, comprises: Visual features are extracted from the target image based on the visual encoder in the image processing unit to obtain the initial visual features corresponding to the target level and the number of initial visual words; By selecting features from the initial visual features of the initial number of visual words, the retained index data of the target number of visual words is determined, wherein the retained index data is the word position corresponding to each retained visual feature; Based on the retained index data, feature vectors for corresponding word positions are extracted from the initial visual features of the initial number of initial visual words to obtain the target visual features corresponding to the target number of visual words.
7. The method according to any one of claims 1-5, wherein after determining the input sequence corresponding to the data processing request based on the number of target visual lexical units, it further includes: Based on the image placeholders corresponding to the target visual word count, target index data corresponding to the input sequence is obtained. The target index data includes offset data, which is determined according to the position of the image placeholders corresponding to the target visual word count in the input sequence. The image processing unit obtains the target visual word count based on the offset data.
8. The method of claim 7, wherein filling the target visual features corresponding to the number of target visual lexical units into the image placeholders of the input sequence to obtain the target sequence comprises: Based on the target index data corresponding to the input sequence, the target visual features corresponding to the number of target visual words are filled into the image placeholders of the input sequence to obtain the target sequence.
9. A data processing system, comprising a central processing unit and an image processing unit, wherein, The central processing unit is configured to obtain the initial number of visual words in the target image in the data processing request; determine the target number of visual words based on the target reduction parameters and the initial number of visual words; and determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words. The image processing unit is used to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The central processing unit is further configured to fill the target visual features corresponding to the number of target visual lexical units into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
10. A data processing apparatus, comprising: The quantity acquisition module is configured to obtain the initial number of visual lemmas of the target image in the data processing request; The sequence determination module is configured to determine the target number of visual words based on the target reduction parameters and the initial number of visual words, and to determine the input sequence corresponding to the data processing request based on the target number of visual words, wherein the input sequence contains image placeholders corresponding to the target number of visual words; The feature selection module is configured to extract visual features from the target image, generate initial visual features corresponding to the initial number of visual words, and determine the target visual features corresponding to the target number of visual words from the initial visual features; The sequence filling module is configured to fill the target visual features corresponding to the number of target visual words into the image placeholders of the input sequence to obtain the target sequence, and obtain the processing result corresponding to the data processing request based on the target sequence.
11. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 8.
12. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.
13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.
