Visual word layer perceptual adaptive compression method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a visual token layer perception adaptive compression method and device, an electronic device and a storage medium. The method comprises: in a plurality of backbone layers of a visual language model, determining at least two candidate layers as compression positions, wherein each candidate layer corresponds to a configured predictor for predicting the importance of visual tokens; based on calibration sample data, evaluating a first performance indicator of the visual language model in performing a target task and in each candidate layer performing visual token compression alone; based on each first performance indicator, determining a target layer as a target compression position from the at least two candidate layers. In this way, the target layer matching the target task can be determined, and since the visual token compression is performed in the target layer, the prediction performance of the visual language model can be basically maintained or improved when facing different tasks.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a visual word layer perception adaptive compression method and apparatus, electronic device and storage medium. Background Technology

[0002] A visual language model typically consists of a visual encoder, a text encoding module, and a large language model decoder. The image is first divided into image blocks by the visual encoder and encoded as visual words, while the text question or instruction is encoded as text words. Both are then input into the subsequent large language model for cross-modal understanding and language generation.

[0003] As the scale of visual language models continues to expand, especially in tasks such as high-resolution image understanding, fine-grained recognition, image-text question answering, and scene text understanding, the number of visual lexical units is usually far greater than the number of text lexical units. The influx of a large number of visual lexical units into large language models significantly increases the computational load and memory consumption during the pre-filling stage, leading to higher inference latency and service costs in actual deployment.

[0004] To address the problem of visual lexical redundancy, researchers have proposed various visual lexical compression methods. However, all of them set the compression position at a fixed layer in the visual language model. However, visual models can extract different information at different layers. Furthermore, the dominant information in an image varies depending on the task. Therefore, setting the compression position at a fixed layer may severely impact the model's predictive performance when dealing with different tasks. Summary of the Invention

[0005] In view of this, this disclosure proposes a visual lexical layer-aware adaptive compression scheme.

[0006] According to one aspect of this disclosure, a visual lexical layer-aware adaptive compression method is provided. The method includes: determining at least two candidate layers as compression positions among multiple backbone layers of a visual language model, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units; evaluating a first performance index of the visual language model in performing a target task and performing visual lexical compression individually on each candidate layer based on calibration sample data; and determining a target layer as the target compression position from the at least two candidate layers based on each of the first performance index.

[0007] In one possible implementation, the method further includes: controlling each backbone layer before the target layer and the target layer to process the image to obtain hidden representations of multiple visual words corresponding to the image; controlling a first predictor configured for the target layer to predict the importance score of the visual words based on the hidden representations; pruning the visual words based on the importance score to obtain retained visual words; concatenating the retained visual words with text words to obtain sequence data; and controlling the backbone layer after the target layer to process the sequence data to obtain a prediction result.

[0008] In one possible implementation, the control is configured for a first predictor of the target layer to predict the importance score of the visual lexical based on the hidden representation, including: determining parameters of the first predictor based on the identifier of the target task and the layer number of the target layer; controlling the first predictor to predict the importance score of the visual lexical based on the parameters.

[0009] In one possible implementation, the performance metrics include task accuracy and task time. The method further includes: evaluating a second performance metric for each candidate layer based on calibration sample data, assuming the visual language model does not undergo visual lexical compression; determining the target layer as the target compression position from the at least two candidate layers based on each of the first performance metrics includes: comparing each of the first performance metrics with the second performance metrics, and selecting the candidate layer with the largest decrease in task accuracy (less than a threshold) and the largest decrease in task time as the target layer.

[0010] In one possible implementation, the method further includes: for the target task, controlling a visual language model without a candidate layer to process training sample data and obtain a first sample result; for the target task, controlling a visual language model with a single candidate layer to process the training sample data and obtain a second sample result; determining a training loss based at least on a first difference between the second sample result and the first sample result; and adjusting the parameters of the predictor configured in the single candidate layer based on the training loss until a convergence condition is met.

[0011] In one possible implementation, the method further includes: determining a second difference between the second sample result and the benchmark result; determining a third difference between the retention ratio of visual lexical units by the single candidate layer and the target retention ratio; and determining the training loss based at least on the first difference between the second sample result and the first sample result, including: determining the training loss based on the first difference, the second difference, and the third difference.

[0012] In one possible implementation, the performance metrics include at least one of the following metrics: task accuracy, and / or task time, and / or video memory utilization, and / or visual word compression rate.

[0013] According to another aspect of this disclosure, a visual lexical layer perceptual adaptive compression device is provided, the device 20 comprising:

[0014] Compression position determination unit 21 is used to determine at least two candidate layers as compression positions in multiple backbone layers of a visual language model, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units.

[0015] The first performance evaluation unit 22 is used to evaluate the first performance index of the visual language model in performing the target task and performing visual word compression separately in each of the candidate layers based on calibration sample data.

[0016] The target layer determination unit 23 is used to determine the target layer as the target compression position from the at least two candidate layers based on each of the first performance indicators.

[0017] In one possible implementation, the device 20 further includes:

[0018] The hidden representation determination unit is used to control the processing of the image by each backbone layer before the target layer and the target layer to obtain hidden representations of multiple visual words corresponding to the image;

[0019] An importance score prediction unit is used to control a first predictor configured for the target layer to predict the importance score of the visual word based on the hidden representation;

[0020] A visual lexical pruning unit is used to prune the visual lexical based on the importance score to obtain the retained visual lexical;

[0021] Sequence data units are used to concatenate retained visual lexical units with textual lexical units to obtain sequence data;

[0022] The prediction result determination unit is used to control the backbone layer after the target layer to process the sequence data and obtain the prediction result.

[0023] In one possible implementation, the importance score prediction unit is further configured to:

[0024] Based on the identifier of the target task and the layer number of the target layer, the parameters of the first predictor are determined;

[0025] The first predictor is controlled to predict the importance score of the visual lexical based on the parameters.

[0026] In one possible implementation, the performance metrics include: task accuracy and task time, and the device 20 further includes:

[0027] The second performance index determination unit is used to evaluate the second performance index of the visual language model without visual word compression for each candidate layer based on calibration sample data.

[0028] The target layer determination unit 23 is further configured to:

[0029] Each of the first performance indicators is compared with the second performance indicator, and the candidate layer with the largest decrease in task accuracy (less than the threshold) and the largest decrease in task time is selected as the target layer.

[0030] In one possible implementation, the device 20 further includes:

[0031] The first sample result determination unit is used to control the visual language model without a candidate layer to process the training sample data and obtain the first sample result for the target task.

[0032] The second sample result determination unit is used to control a visual language model with a single candidate layer set for the target task, process the training sample data, and obtain the second sample result.

[0033] The trainer parameter adjustment unit is used to determine the training loss based at least on the first difference between the second sample result and the first sample result, and adjust the parameters of the predictor configured in the single candidate layer based on the training loss until the convergence condition is met.

[0034] In one possible implementation, the device 20 further includes:

[0035] The second difference determination unit is used to determine the second difference between the second sample result and the benchmark result;

[0036] The third difference determination unit is used to determine the third difference between the retention ratio of visual words in a single candidate layer and the target retention ratio.

[0037] The trainer parameter adjustment unit is further used for:

[0038] The training loss is determined based on the first difference, the second difference, and the third difference.

[0039] In one possible implementation, the performance metrics include at least one of the following metrics: task accuracy, and / or task time, and / or video memory utilization, and / or visual word compression rate.

[0040] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0041] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0042] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0043] In this embodiment, at least two candidate layers are first determined. Using calibration samples, the first performance index of the visual language model is evaluated for each candidate layer individually when visual word compression is performed, for the target task. The first performance index is used as the evaluation criterion to select the target layer from the candidate layers. In this way, a target layer matching the target task can be determined. Since visual word compression is performed at the target layer, the predictive performance of the visual language model can be basically maintained or improved when facing different tasks.

[0044] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0045] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0046] Figure 1 This is a flowchart illustrating the visual lexical layer-aware adaptive compression method provided in an embodiment of this disclosure.

[0047] Figure 2 This is a schematic diagram of the structure of the visual lexical layer perception adaptive compression device provided in an embodiment of this disclosure.

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device for visual lexical layer-aware adaptive compression provided in an embodiment of this disclosure. Detailed Implementation

[0049] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0050] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0051] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0052] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0053] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0054] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0055] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0056] To address the problem of visual lexical redundancy, researchers have proposed various visual lexical compression methods. However, all of them set the compression location as a fixed layer in the visual language model. For ease of description below, the layer used for visual lexical compression will be named the lexical layer. Using a fixed lexical layer means that information on the same layer will be compressed regardless of the task being performed. This can lead to situations where information that needs to be compressed has not yet been formed at the lexical layer, while information crucial for completing the target task is compressed instead, reducing the accuracy of the prediction results; or the information that needs to be compressed may have already been formed before the layer, resulting in late compression and unnecessary computation. This degrades the predictive performance of the visual language model.

[0057] Therefore, a method for adaptively determining the lexical layer based on the target task is needed, which can basically maintain the predictive performance of the visual language model when facing different tasks.

[0058] Figure 1 This is a flowchart illustrating the visual lexical layer-aware adaptive compression method provided in an embodiment of this disclosure.

[0059] S11, in the multiple backbone layers of the visual language model, at least two candidate layers are determined as compression positions, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units.

[0060] In this embodiment, the backbone layer can be uniformly sampled to obtain candidate layers. For example, the backbone layer can be divided into three segments: front, middle, and back, and uniform sampling can be performed on each segment. Alternatively, layers generating various types of information can be used as candidate layers, such as layers that generate image texture and details, layers that generate semantics, and layers that generate context alignment information. Alternatively, each backbone layer can be determined as a candidate layer. Candidate layers can serve as compression points to compress the hidden representations of visual lexical units on the candidate layers. These hidden representations can be vectors.

[0061] The predictor can predict the importance of visual words based on their hidden representations in candidate layers. Each candidate layer can share a single predictor, and each candidate layer can correspond to a set of predictor parameters. When using a predictor corresponding to a candidate layer, predictor parameters matching that candidate layer can be loaded into the predictor. Alternatively, a predictor can be configured individually for each candidate layer. This disclosure does not limit this approach. The predictor can be a lightweight predictor. A lightweight predictor can employ linear layers, multilayer perceptrons, or lightweight attention modules, or it can employ convolutional modules, gating modules, lightweight Transformer modules, or other model structures that can output visual word importance scores.

[0062] S12, Based on calibration sample data, evaluate the first performance metric of the visual language model in performing the target task and performing visual lexical compression individually in each of the candidate layers.

[0063] In this embodiment of the disclosure, one or more data to be predicted can be selected from the target task. The number of calibration sample data is much smaller than the number of data to be predicted in the target task.

[0064] For each candidate layer, the following steps are performed: Calibration sample data is input into the visual language model, compressed only through the current candidate layer, to obtain the output prediction result, and the first performance metric of the visual language model corresponding to the current candidate layer is evaluated. In this way, each candidate layer can be assigned its own first performance metric.

[0065] S13, based on each of the first performance indicators, determine the target layer as the target compression position from the at least two candidate layers.

[0066] In this embodiment of the disclosure, a target first performance index representing the optimal performance can be selected, and the candidate layer corresponding to the target first performance index can be used as the target layer. Alternatively, a first performance index with a value higher than a performance threshold can be used as a candidate performance index, and a target first performance index can be obtained by further selection from one or more candidate performance indices, and the candidate layer corresponding to the target first performance index can be used as the target layer.

[0067] In one possible implementation, the performance metrics include at least one of the following: task accuracy, and / or task time, and / or GPU memory usage, and / or visual word compression rate. The specific metrics included in the first performance metric can be determined based on the requirements of the target task. This ensures that the visual language model meets the requirements of the target task when processing it.

[0068] In this embodiment, at least two candidate layers are first determined. Using calibration samples, the first performance index of the visual language model is evaluated for each candidate layer individually when visual word compression is performed, for the target task. The first performance index is used as the evaluation criterion to select the target layer from the candidate layers. In this way, a target layer matching the target task can be determined. Since visual word compression is performed at the target layer, the predictive performance of the visual language model can be basically maintained or improved when facing different tasks.

[0069] In one possible implementation, the method further includes: controlling each backbone layer before the target layer and the target layer to process the image to obtain hidden representations of multiple visual words corresponding to the image; controlling a first predictor configured for the target layer to predict the importance score of the visual words based on the hidden representations; pruning the visual words based on the importance score to obtain retained visual words; concatenating the retained visual words with text words to obtain sequence data; and controlling the backbone layer after the target layer to process the sequence data to obtain a prediction result.

[0070] After determining the target layer, the image can be input into the visual language model. The backbone layers before the target layer, as well as the target layer itself, can process the image to obtain the hidden representations corresponding to each visual word output by the target layer. For ease of description, the predictor configured for the target layer is named the first predictor. The first predictor is invoked to predict the importance score of each visual word based on the hidden representations. The importance score characterizes the importance of the visual word. This image is included in the prediction data for the target task.

[0071] In this embodiment, importance scores can be used as the basis for compressing visual words to obtain retained visual words. For example, visual words with importance scores greater than a first score threshold can be retained. Another example is the removal of visual words with importance scores less than a second score threshold, leaving the remaining visual words as retained visual words. Yet another example is the selection of visual words with the highest importance scores according to a preset ratio. The retained visual words are then concatenated with text words to obtain sequence data. The backbone layer following the target layer processes the sequence data to obtain prediction results.

[0072] When a visual language model makes predictions for a target task, it can compress visual lexical units at the target layer based on the first predictor. The information that needs to be compressed in the target task is generated precisely at the target layer and can be accurately compressed, reducing the probability of incorrectly compressing important information and / or delaying the compression of information that needs to be compressed. This improves the predictive performance of the visual language model when processing any target task. Moreover, compressing only visual lexical units can reduce redundant information while avoiding damage to semantic expression, thus improving reasoning efficiency and accuracy.

[0073] In one possible implementation, the control is configured for a first predictor of the target layer to predict the importance score of the visual lexical based on the hidden representation, including: determining parameters of the first predictor based on the identifier of the target task and the layer number of the target layer; controlling the first predictor to predict the importance score of the visual lexical based on the parameters.

[0074] When multiple candidate layers share the same predictor, predictor parameters corresponding to the target task and target layer can be selected based on the target task identifier and the target layer number. These parameters are then loaded into the first predictor, which controls the prediction of visual word importance scores.

[0075] In this embodiment of the disclosure, multiple candidate layers can share the same predictor, reducing memory footprint and maintaining or improving the predictive performance of the visual language model for the target task.

[0076] In one possible implementation, the performance metrics include task accuracy and task time. The method further includes: evaluating a second performance metric for each candidate layer based on calibration sample data, assuming the visual language model does not undergo visual lexical compression; determining the target layer as the target compression position from the at least two candidate layers based on each of the first performance metrics includes: comparing each of the first performance metrics with the second performance metrics, and selecting the candidate layer with the largest decrease in task accuracy (less than a threshold) and the largest decrease in task time as the target layer.

[0077] In this embodiment of the disclosure, the following steps are performed for each candidate layer: calibration sample data is input into the visual language model, compressed only by the current candidate layer, to obtain the output prediction result, and a first performance metric of the visual language model corresponding to the current candidate layer is evaluated. That is, the first task accuracy and the first task time. Thus, each candidate layer can correspond to its own first task accuracy and first task time.

[0078] Furthermore, the calibration samples are input into the visual language model, and the second performance metric of the visual language model is obtained without lexical compression. That is, the accuracy of the second task and the time taken for the second task.

[0079] Each first performance metric is compared with a second performance metric to determine the decrease in task accuracy and the decrease in task time for each candidate layer. A pre-defined magnitude threshold is set, which represents the maximum decrease in task accuracy caused by visual lexical compression for the target task. Therefore, the candidate layer with the largest decrease in task accuracy (less than the magnitude threshold) and the largest decrease in task time is selected as the target layer.

[0080] In the embodiments of this disclosure, the selected target layer can minimize prediction time and improve the prediction efficiency of the model while meeting the task accuracy requirements.

[0081] In one possible implementation, the method further includes: for the target task, controlling a visual language model without a candidate layer to process training sample data and obtain a first sample result; for the target task, controlling a visual language model with a single candidate layer to process the training sample data and obtain a second sample result; determining a training loss based at least on a first difference between the second sample result and the first sample result; and adjusting the parameters of the predictor configured in the single candidate layer based on the training loss until a convergence condition is met.

[0082] For the target task, a matching predictor can be trained for each candidate layer (or the matching predictor parameters can be obtained).

[0083] The training samples can be test data selected for the target task. For the target task, the training samples are input into a visual language model without a candidate layer, resulting in the first sample result output by the visual language model without a candidate layer after predicting the training samples. Furthermore, for the target task, and for each candidate layer separately, the training samples are input into a visual language model with a single candidate layer, resulting in the second sample result output by the visual language model with a single candidate layer after predicting the training samples.

[0084] The first difference between the second sample result and the first sample result can characterize the first difference in the prediction results output by the visual language model after word compression by a single candidate layer. This first difference is generated by word compression by the candidate layer, and thus can be used to measure the degree to which word compression by the candidate layer causes a change in the prediction result. This first difference can be used as a training loss to adjust the parameters of the predictor configured in the single candidate layer until a convergence condition is met. The convergence condition can be that the number of epochs of parameter adjustment reaches a preset number of epochs. The convergence condition can be that the first difference is less than a preset difference threshold for n consecutive times.

[0085] In this way, a corresponding predictor can be trained for each candidate layer for the target task. During the use of the visual language model, the trained predictor can be configured for the target layer to improve the prediction performance of the visual language model when processing any target task.

[0086] In one possible implementation, the method further includes: determining a second difference between the second sample result and the benchmark result; determining a third difference between the retention ratio of visual lexical units by the single candidate layer and the target retention ratio; and determining the training loss based at least on the first difference between the second sample result and the first sample result, including: determining the training loss based on the first difference, the second difference, and the third difference.

[0087] In this embodiment of the disclosure, during the training of each predictor, a second difference between the second sample result and the baseline result can be determined. The baseline result can be the correct result in the label corresponding to the training sample. The second difference can be used to measure the degree to which the candidate layer causes a decrease in the accuracy of the prediction result due to lexical compression.

[0088] During the training of each predictor, a third difference can be determined between the retention rate of visual lexical units by the candidate layer and the target retention rate. This third difference characterizes the gap between the compression capability of the candidate layer and the target.

[0089] For ease of understanding, the training loss is represented by formula (1).

[0090] (1)

[0091] in, Indicates training loss, Indicates the second difference. Indicates the first difference, Indicates the third difference, , These are the weighting coefficients.

[0092] In this embodiment, because the training loss includes a first difference, the compressed output, guided by the importance score of the predictor output, is close to the uncompressed output, reducing the number of visual words without decreasing the amount of information. Because the training loss includes a second difference, even when visual words are compressed (compression guided by the importance score of the predictor output), the accuracy of the task is not reduced. Because the training loss includes a third difference, the compression rate of visual words can reach the target, thereby reducing computational load and improving the inference efficiency of the visual language model.

[0093] Therefore, the predictor trained using the training loss of this embodiment does not reduce the amount of information after visual lexical compression, thus maintaining the accuracy of reasoning and improving reasoning efficiency.

[0094] In one example, the backbone layers of a visual language model can be sampled to obtain candidate layers. For example, layers 4, 8, 12, and 16 can be used as candidate layers. For the target task, training sample data is used to train predictors for each candidate layer. During predictor training, the training loss can include at least the first difference. The training loss can include the first difference, second difference, and third difference. The predictor parameters are tuned based on the training loss until the convergence condition is met.

[0095] During the inference phase, one or more data points can be arbitrarily selected from the target task's prediction data as calibration sample data. Based on the calibration sample data, for each candidate layer, the first performance metric of the visual language model under visual lexical compression is evaluated. The candidate layer that optimizes the first performance metric can be selected as the target candidate layer. Alternatively, based on the calibration sample data, the second performance metric of the visual language model without visual lexical compression is evaluated. Based on the comparison results of each first and second performance metric, the candidate layer with the shortest task time after compression, while achieving the standard task accuracy, is determined as the target layer.

[0096] For example, for fine-grained recognition tasks, the target layer can be a candidate layer located at the front or middle to preserve local texture, edge and region details.

[0097] For example, for visual reasoning tasks, the target layer can be a later candidate layer to perform visual lexical compression after semantic information and object relationships are formed.

[0098] For example, for a text question-answering task, the target layer can be a candidate layer located in the middle, so as to perform visual lexical compression after the context alignment information is formed.

[0099] Configure the corresponding predictor for the target layer. For example, a predictor trained for the target task can be configured for the target layer. Another example is determining the predictor parameters based on the target layer number and the target task identifier. The predictor parameters are obtained by training the predictor as described above. Load the predictor parameters into the predictor and configure it for the target layer.

[0100] The data to be predicted is input into a visual language model that has already configured a predictor for the target layer. The target layer and the backbone layer preceding it process the images in the data to obtain hidden representations of visual words. The predictor configured in the target layer can predict the importance scores of visual words based on these hidden representations. Visual words can be selected for retention based on their importance scores, according to a preset number or proportion. The retained visual words and text words are then concatenated to obtain sequence data. The backbone layer following the target layer processes this sequence data to obtain the prediction results. In this way, the predictive performance of the visual language model can be maintained or improved for different tasks.

[0101] Figure 2 A schematic diagram of the structure of a visual lexical layer perceptual adaptive compression device provided in an embodiment of this disclosure. The device 20 includes:

[0102] Compression position determination unit 21 is used to determine at least two candidate layers as compression positions in multiple backbone layers of a visual language model, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units.

[0103] The first performance evaluation unit 22 is used to evaluate the first performance index of the visual language model in performing the target task and performing visual word compression separately in each of the candidate layers based on calibration sample data.

[0104] The target layer determination unit 23 is used to determine the target layer as the target compression position from the at least two candidate layers based on each of the first performance indicators.

[0105] In one possible implementation, the device 20 further includes:

[0106] The hidden representation determination unit is used to control the processing of the image by each backbone layer before the target layer and the target layer to obtain hidden representations of multiple visual words corresponding to the image;

[0107] An importance score prediction unit is used to control a first predictor configured for the target layer to predict the importance score of the visual word based on the hidden representation;

[0108] A visual lexical pruning unit is used to prune the visual lexical based on the importance score to obtain the retained visual lexical;

[0109] Sequence data units are used to concatenate retained visual lexical units with textual lexical units to obtain sequence data;

[0110] The prediction result determination unit is used to control the backbone layer after the target layer to process the sequence data and obtain the prediction result.

[0111] In one possible implementation, the importance score prediction unit is further configured to:

[0112] Based on the identifier of the target task and the layer number of the target layer, the parameters of the first predictor are determined;

[0113] The first predictor is controlled to predict the importance score of the visual lexical based on the parameters.

[0114] In one possible implementation, the performance metrics include: task accuracy and task time, and the device 20 further includes:

[0115] The second performance index determination unit is used to evaluate the second performance index of the visual language model without visual word compression for each candidate layer based on calibration sample data.

[0116] The target layer determination unit 23 is further configured to:

[0117] Each of the first performance indicators is compared with the second performance indicator, and the candidate layer with the largest decrease in task accuracy (less than the threshold) and the largest decrease in task time is selected as the target layer.

[0118] In one possible implementation, the device 20 further includes:

[0119] The first sample result determination unit is used to control the visual language model without a candidate layer to process the training sample data and obtain the first sample result for the target task.

[0120] The second sample result determination unit is used to control a visual language model with a single candidate layer set for the target task, process the training sample data, and obtain the second sample result.

[0121] The trainer parameter adjustment unit is used to determine the training loss based at least on the first difference between the second sample result and the first sample result, and adjust the parameters of the predictor configured in the single candidate layer based on the training loss until the convergence condition is met.

[0122] In one possible implementation, the device 20 further includes:

[0123] The second difference determination unit is used to determine the second difference between the second sample result and the benchmark result;

[0124] The third difference determination unit is used to determine the third difference between the retention ratio of visual words in a single candidate layer and the target retention ratio.

[0125] The trainer parameter adjustment unit is further used for:

[0126] The training loss is determined based on the first difference, the second difference, and the third difference.

[0127] In one possible implementation, the performance metrics include at least one of the following metrics: task accuracy, and / or task time, and / or video memory utilization, and / or visual word compression rate.

[0128] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0129] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0130] This disclosure also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0131] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0132] Figure 3 This is a schematic diagram of the structure of an electronic device for visual lexical layer perceptual adaptive compression provided in an embodiment of this disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. (Refer to...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0133] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0135] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0136] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0137] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0138] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0142] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual lexical layer-aware adaptive compression method, characterized in that, The method includes: In the multiple backbone layers of the visual language model, at least two candidate layers are determined as compression positions, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units. Based on calibration sample data, the first performance metric of the visual language model is evaluated when performing the target task and visual lexical compression is performed individually in each of the candidate layers. Based on each of the first performance metrics, a target layer is determined from the at least two candidate layers as the target compression location.

2. The method according to claim 1, characterized in that, The method further includes: The main layers before the target layer and the target layer process the image to obtain hidden representations of multiple visual terms corresponding to the image; Control the first predictor configured for the target layer to predict the importance score of the visual lexical based on the hidden representation; Based on the importance score, the visual lexical units are pruned to obtain the retained visual lexical units; The retained visual lexical units are concatenated with the textual lexical units to obtain sequence data; The backbone layer following the target layer processes the sequence data to obtain the prediction results.

3. The method according to claim 2, characterized in that, The control is a first predictor configured for the target layer, which predicts the importance score of the visual lexical unit based on the hidden representation, including: Based on the identifier of the target task and the layer number of the target layer, the parameters of the first predictor are determined; The first predictor is controlled to predict the importance score of the visual lexical based on the parameters.

4. The method according to claim 1, characterized in that, The performance metrics include: task accuracy and task time; the method further includes: Based on calibration sample data, for each of the candidate layers, the second performance index of the visual language model without visual lexical compression is evaluated. The step of determining the target layer as the target compression position from the at least two candidate layers based on each of the first performance indicators includes: Each of the first performance indicators is compared with the second performance indicator, and the candidate layer with the largest decrease in task accuracy (less than the threshold) and the largest decrease in task time is selected as the target layer.

5. The method according to claim 1, characterized in that, Also includes: For the target task, the visual language model without candidate layers is controlled to process the training sample data and obtain the first sample result; For the target task, a visual language model with a single candidate layer was configured to process the training sample data and obtain a second sample result. Based at least on the first difference between the second sample result and the first sample result, a training loss is determined, and based on the training loss, the parameters of the predictor configured in the single candidate layer are adjusted until the convergence condition is met.

6. The method according to claim 5, characterized in that, The method further includes: Determine the second difference between the second sample results and the baseline results; Determine the third difference between the retention ratio of visual lexical units in the individual candidate layer and the target retention ratio; The determination of training loss based at least on the first difference between the second sample result and the first sample result includes: The training loss is determined based on the first difference, the second difference, and the third difference.

7. The method according to claim 1, characterized in that, The performance metrics include at least one of the following metrics: task accuracy, and / or task time, and / or video memory usage, and / or visual word compression rate.

8. A visual word layer perception adaptive compression device, characterized in that, The device includes: A compression position determination unit is used to determine at least two candidate layers as compression positions among multiple backbone layers of a visual language model, wherein each candidate layer is configured with a predictor for predicting the importance of visual lexical units. The first performance evaluation unit is used to evaluate the first performance index of the visual language model in performing the target task and performing visual word compression separately in each of the candidate layers based on calibration sample data. The target layer determination unit is used to determine the target layer as the target compression position from the at least two candidate layers based on each of the first performance indicators.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images