A graphics API recommendation method based on multi-scene lightweight adapter

Abstract

This invention relates to a graphics API recommendation method based on a multi-scene lightweight adapter, comprising the following steps: acquiring a public image dataset D, and constructing a recommendation model M based on a Transformer decoder; selecting any image data from D for data preprocessing to obtain standardized image data, and then performing image block segmentation and feature embedding operations sequentially to obtain an initial visual feature sequence; inputting the input into the first Transformer decoder to obtain the core branch backbone features at layer l; inputting the input into MLP branches and branches respectively to obtain the self-attention feature matrix and the preliminary dimensionality reduction projection feature matrix; inputting the input into the output to obtain a low-dimensional matrix, and inputting the input to obtain a high-dimensional matrix, calculating the final feature matrix after passing through layers, and sequentially passing it through all subsequent Transformer decoders to obtain a graphics API probability sequence; then training the recommendation model, and obtaining the trained recommendation model M' after completion.

Description

Technical Field

[0001] This invention relates to the technical fields of graphics API recommendation, computer vision, multi-label image classification, and efficient parameter fine-tuning, and particularly to a graphics API recommendation method based on a multi-scene lightweight adapter. Background Technology

[0002] With the widespread application of data visualization in scientific research analysis, engineering design, business decision-making, and educational demonstrations, the demand for rapid chart creation from developers and non-professional users is constantly growing. Graphical Application Programming Interfaces (APIs) are core tools for generating charts, controlling styles, and expressing visual information through programming languages. However, different chart types often correspond to different graphical APIs. Developers typically need to search, compare, and experiment with a large number of candidate APIs to find one that can replicate the chart type, which results in high implementation costs and long trial-and-error cycles.

[0003] Plot2API proposed the SPGNN network to solve the problem of image API recommendation. It treats the image API recommendation task as a multi-label classification task, extracting relevant semantic visual information through a semantic parsing module and combining it with a convolutional neural network (CNN) to process image information and recommend image APIs. MSTIL addresses the data imbalance problem in Plot2API by employing a three-step optimization strategy of transfer learning, data augmentation, and a novel loss function, effectively mitigating the overfitting problem of SPGNN. However, it only initially implements the function of recommending APIs based on image features, is limited to one language, and uses only one modeling method (i.e., CNN).

[0004] Existing research on graph API recommendation extracts image features and establishes a mapping relationship between images and API labels to achieve automatic recommendation from graphs to APIs. However, these methods have significant limitations in real-world applications.

[0005] First, existing graphics API recommendation methods are usually centered around a single programming language. However, different programming languages ​​have independent graphics API systems. A model trained on a Python graphics API dataset cannot recommend R graphics APIs. If you want to recommend R graphics APIs, you need to retrain the entire model using an R graphics API dataset. This will cause the parameter size, training cost, and deployment overhead to expand rapidly with the number of languages.

[0006] Second, existing models often employ a single modeling mechanism in the process of extracting graphic features, either focusing on the global topological structure or on local detailed features, lacking the ability to collaboratively represent the global topological structure and local detailed features of a graph. In complex graphic API recommendation tasks, category discrimination depends on both the overall layout and local details; therefore, the performance of single-feature modeling methods degrades significantly when dealing with complex graphics. Summary of the Invention

[0007] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to implement API recommendation for complex multilingual graphics in a lightweight and efficient manner.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A method for recommending graphics APIs based on a multi-scenario lightweight adapter includes the following steps:

[0010] S1: Obtain a publicly available image dataset D. Each image sample in D includes image data and image ground truth labels. The image ground truth labels refer to the actual graphics API used to draw the image sample and the target programming language scenarios corresponding to these graphics APIs. ;

[0011] A recommendation model M is constructed based on Transformer, comprising several cascaded Transformer decoders and a recommendation module. In addition to the original structure of each Transformer decoder, an adapter unit is added after the MSA layer of each Transformer decoder, parallel to the MLP layer. The adapter unit includes the following cascaded shared downprojection matrix modules. Local perception enhancement unit module And shared projection matrix module The shared projection matrix module mentioned above The local perception enhancement unit module is used to obtain the preliminary dimensionality-reduced projection feature matrix. Composed of a convolutional network module (CNN), the shared up-projection matrix is ​​used to obtain a high-dimensional output matrix;

[0012] The recommendation module is used to output a recommended list of graphics APIs;

[0013] S2: Select any image data from D. ,right Data preprocessing is performed to obtain standardized image data. Then Image patch segmentation and feature embedding are performed sequentially to obtain an initial visual feature sequence. ;

[0014]

[0015]

[0016]

[0017] in, Represents the position encoding matrix, This represents an extended sequence matrix containing global tokens. express… Indicates the first after flattening Local vectors Represents the linear projection weight matrix. Represents a learnable bias vector; Represents a learnable parameter vector;

[0018] S3: Will The first Transformer decoder in input M passes through the LayerNorm layer, MSA module, and LayerNorm layer of each layer in the first Transformer decoder in sequence to obtain the result at the 1st epoch. Core branch trunk features of the layer ;

[0019] S4: Will Inputs into the MLP layer and adapter unit respectively , The self-attention feature matrix is ​​obtained through the MLP layer. , through Obtain the preliminary dimension-reduced projection feature matrix ,in, Represents the first... layer, The expression is as follows:

[0020]

[0021]

[0022]

[0023] in, This represents the globally shared learnable matrix of the Transform. The layer-specific projection scaling matrix is ​​represented. Indicates the first Layer-specific bias parameters, Indicates length is The whole Column vectors D' represents the outer product; D' represents Spatial dimension, Indicates the first Lower layer The first in elements and , Indicates the first Lower layer The first in One element; one learnable scalar parameter;

[0024] S5: Will enter Then the low-dimensional matrix is ​​output. ;

[0025] S6: Will enter Obtain the high-dimensional matrix The expression is as follows:

[0026]

[0027] in, Indicates the first The layer is modulated using a dedicated up-projection transformation matrix. Indicates the first Layer bias vector;

[0028] S7: Calculation process The final feature matrix of the layer The expression is as follows:

[0029] ;

[0030] S8: Will After passing through all subsequent Transformer encoders in sequence, we obtain information about... Graph API probability sequence The value of K is determined by To determine, set a probability threshold. Input P into the recommendation module, and select P that is greater than... The probability corresponding to the graphical API composition The graphical API predicts and recommends a list and outputs it, and then makes a judgment. Greater than the threshold The expression is as follows:

[0031]

[0032] in, This represents the category index of the graphics API corresponding to the k-th probability value in P;

[0033] S9: Iterate through all data in D, repeating S2-S8, to obtain the graph API prediction recommendation list corresponding to all image data. For each image in D, label its own graph API prediction tag according to its corresponding graph API prediction recommendation list. Pair the ground truth label and predicted label of each image in D as a training sample. All training samples constitute the supervised loss training set. ;

[0034] S10: Constructing the loss function ,Will As input to L, using Train M, freezing the structural model parameters of M except for the adapter unit during training. Update the model parameters through gradient descent backpropagation. Stop training when the maximum number of iterations is reached or the loss function converges. At this point, the trained recommendation model M' is obtained.

[0035] S11: Select the image X to be recommended, and input X into M' to obtain a list of recommended image APIs for X.

[0036] Preferably, standardized image data is obtained in S2. The steps are as follows:

[0037] S2-1: Will Mapped to the target image using bilinear interpolation. ;

[0038] S2-2: Using continuously uniformly distributed random variables to... The image is obtained by randomly flipping it horizontally. The calculation expression is as follows:

[0039]

[0040] in, Indicates the flip indicator function;

[0041] S2-3: To Dimensional permutation and range continuity are performed to obtain Then on Z-score normalization yields the normalized tensor. The expression is as follows:

[0042]

[0043]

[0044] in, This represents the expected distribution vector of channel c. This represents the standard deviation distribution vector of channel c;

[0045] S2-4: Utilize Obtain standardized images The expression is as follows:

[0046]

[0047] in, This indicates the fill value for the occluded area. This indicates that the triggering control random variable is... A two-dimensional spatial set representing the occluded area. Represents the coordinates of any discrete pixel on image I.

[0048] Preferably, the loss function is constructed in S10. The steps are as follows:

[0049] S10-1: Constructing a loss function using binary cross-entropy loss The calculation formula is as follows:

[0050]

[0051] in, Indicates the first The sample at the th Unnormalized prediction outputs for each API category This represents the corresponding real label;

[0052] S10-2: Define the confidence factor for prediction accuracy Then use right Adaptive recalibration is performed to obtain The calculation formula is as follows:

[0053]

[0054] in, Represents the balance coefficient. Indicates the focusing coefficient;

[0055] S10-3: Based on and Obtain the overall objective loss function The ultimate doubly discrete summative expansion is expressed as follows:

[0056]

[0057] in, This represents the total number of training samples. This indicates the total number of graphics API categories.

[0058] Compared with the prior art, the present invention has at least the following advantages:

[0059] 1. This invention introduces an inter-layer shared projection matrix within the adapter and, combined with layer-specific scaling and bias parameters, constructs a lightweight adaptation structure based on a parameter reuse strategy. This method no longer learns complete and independent projection matrices for each layer separately. Instead, it restores the differentiated adjustment capabilities of each layer through a small number of layer-specific parameters while sharing basic projection capabilities across layers. This technical solution effectively solves the problem of rapid parameter expansion with increasing layer count during multi-scene graphics API recommendation in existing technologies. It overcomes the shortcomings of high storage overhead, high training costs, and uneconomical deployment caused by repeatedly learning a large number of similar mapping relationships, significantly improving the model's parameter utilization and storage economy.

[0060] 2. This invention addresses the commonalities in graphical representation requirements across different programming language scenarios, while highlighting differences in underlying API implementations. It employs a unified modeling approach combining shared basis parameters and layer-specific adjustment parameters. Specifically, a shared projection matrix models relatively stable high-level graphical semantics and visual representation patterns across different scenarios, while layer-specific scaling and bias parameters compensate for adaptation differences between layers in various scenarios. This method overcomes the shortcomings of existing technologies that often require independent training of complete models across different programming language scenarios. It avoids the problems of repeated training, storage, and deployment caused by scenario switching, enabling the same model framework to support multi-scenario graphical API recommendations at a lower cost, thus significantly enhancing the method's generalization and cross-scenario scalability. Simultaneously, it solves the problems of lightweight programming language switching and improved capabilities for modeling complex images.

[0061] 3. This invention further introduces a local perception enhancement unit based on the parameter reuse and adaptation branch. This unit includes a local spatial feature extraction module, a channel response recalibration module, and a channel feature fusion module. It can supplement the modeling of local spatial information in the input features and adaptively adjust the discriminative importance of different channels. This technical configuration effectively solves the problems of existing methods that mainly rely on global self-attention mechanisms and have insufficient modeling of local discriminative information. It overcomes the defect that relying solely on global features in complex graphic scenes can easily lead to the loss of local key information, thereby improving the model's ability to recognize complex graphic structures and its API discrimination ability.

[0062] 4. This invention, based on the global semantic perception capability provided by the pre-trained backbone network, utilizes local perception enhancement units to supplement local spatial discrimination information, forming a local and global collaborative modeling mechanism for complex graphics. This scheme overcomes the shortcomings of existing single feature extraction mechanisms, which struggle to simultaneously consider the overall data organization and local geometric patterns. It enables the model to accurately grasp the overall layout relationships of graphics while effectively focusing on key local structures influencing API selection, thereby improving the stability, robustness, and recognition accuracy in complex graphics recommendation tasks.

[0063] 5. This invention employs a pre-trained visual Transformer as the backbone network and keeps the backbone parameters frozen when transferring to the graphics API recommendation task, training only a small number of parameters in the adapter branch. This method can fully inherit the general visual priors contained in large-scale pre-trained models, while avoiding the problems of high training cost, large memory overhead, unstable convergence, and high risk of overfitting caused by updating all parameters. Therefore, compared with the full parameter fine-tuning method, this invention significantly reduces the computational resource consumption and parameter tuning difficulty during model training, improves the feasibility of model training, and is especially suitable for graphics API recommendation tasks with limited sample size and varied scenarios.

[0064] 6. In its method design, this invention modularly inserts parameter reuse adapters and local perception enhancement units into the backbone encoder, thereby achieving task adaptation while maintaining structural compatibility. Due to the smaller number of new parameters, the model size is effectively controlled, resulting in higher efficiency in training, storage, transmission, and deployment. This technique overcomes the shortcomings of existing technologies, such as excessively heavy models, high deployment costs, and high barriers to engineering application, making this method more suitable for practical application in real-world systems and easier to promote across multiple terminals, platforms, and scenarios.

[0065] 7. This invention introduces a local perception enhancement mechanism, which can better handle input situations where the local shape of the graphics is complex, the structural combination is diverse, and the fine-grained patterns significantly affect semantic judgment. This method overcomes the problem that existing technologies are more geared towards standard digital graphics design and lack adaptability in real-world complex scenarios. It extends the recommendation of graphics APIs from idealized experimental inputs to complex input conditions that are closer to the actual user needs, thus enhancing the application value of the method.

[0066] 8. This invention, starting from the needs of graphic expression and visual structural features, unifies the commonalities and differences of scenes into a single modeling framework. Shared parameters are responsible for modeling commonalities, while layer-specific parameters are responsible for recovering differences. This design aligns with the essential characteristic of multi-scene graphics API recommendation tasks: "shared high-level expression, differentiated low-level interface adaptation." Therefore, this invention overcomes the shortcomings of existing technologies that overly rely on independent modeling of individual scenes, resulting in a method that not only has better parameter efficiency but also stronger structural rationality and task matching.

[0067] 9. The parameter multiplexing adapter and local perception enhancement unit in this invention both have clear structural boundaries and can be embedded as independent functional modules into different types of visual backbone networks. This design not only facilitates implementation in existing frameworks but also allows for subsequent expansion to more programming language scenarios, more graphics categories, and more downstream recommendation tasks based on task requirements. This technology overcomes the problems of high model coupling, difficult transfer, and inconvenient upgrades in existing technologies, giving this invention good compatibility, portability, and sustainable evolution capabilities.

[0068] 10. This invention achieves a good balance between lightweight design and recommendation performance by compressing redundant parameters, maintaining general graph representation capabilities through parameter reuse strategies, ensuring adaptability through layer-specific parameters, and supplementing complex graph representation capabilities through local perception enhancement units. This approach overcomes the shortcomings of some existing lightweight methods that, while reducing parameter size, simultaneously weaken model expressiveness and recommendation effectiveness. This invention maintains strong graph API recommendation capabilities with lower resource consumption, thus possessing higher overall technical value. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the overall framework of the algorithm of this invention.

[0070] Figure 2 The results are experimental results of this invention under In21K weights.

[0071] Figure 3 This presents the experimental results of the present invention under MAE weighting. Detailed Implementation

[0072] The present invention will now be described in further detail.

[0073] See Figures 1-3 A method for recommending graphics APIs based on a multi-scenario lightweight adapter includes the following steps:

[0074] S1: Obtain a publicly available image dataset D. Each image sample in D includes image data and image ground truth labels. The image ground truth labels refer to the actual graphics API used to draw the image sample and the target programming language scenarios corresponding to these graphics APIs. ;

[0075] A recommendation model M is constructed based on Transformer, comprising several cascaded Transformer decoders and a recommendation module. In addition to the original structure of each Transformer decoder, an adapter unit is added after the MSA layer of each Transformer decoder, parallel to the MLP layer. The adapter unit includes the following cascaded shared downprojection matrix modules. Local perception enhancement unit module And shared projection matrix module The shared projection matrix module mentioned above The local perception enhancement unit module is used to obtain the preliminary dimensionality-reduced projection feature matrix. It is composed of a convolutional network module (CNN). The shared up-projection matrix is ​​used to obtain a high-dimensional output matrix. The MSA layer and MLP (FFN) layer in the Transformer network, as well as the convolutional network module (CNN), are existing technologies. The convolutional network module (CNN) includes a cascaded 3x3 deep convolutional network, a residual connection layer, an SENet module (channel attention mechanism module), a 1x1 deep convolutional network, a GeLU activation function, and a 1x1 deep convolutional network.

[0076] The recommendation module is used to output a recommended list of graphics APIs;

[0077] S2: Select any image data from D. ,right Data preprocessing is performed to obtain standardized image data. Then Image patch segmentation and feature embedding are performed sequentially to obtain an initial visual feature sequence. ;

[0078]

[0079]

[0080]

[0081] in, Represents the position encoding matrix, This represents an extended sequence matrix containing global tokens. express… Indicates the first after flattening Local vectors Represents the linear projection weight matrix. Represents a learnable bias vector; Represents a learnable parameter vector;

[0082] Image patch partitioning based on non-overlapping sliding windows

[0083] Set the spatial resolution (Patch Size) of the square image patch to The image is divided into sections along the height direction. Each slice is divided into slices along its width. There are slices. The total number of image patches generated is . For any index in the spatial grid ,in Represents row index, Represents the column index. We will use the first... Each image patch is defined as a local three-dimensional tensor. The extraction equation is:

[0084]

[0085] Among them, local spatial offset , Channel Index .

[0086] One-dimensional flattening of local tensors

[0087] Each local 3D tensor Arrange the data in continuous memory along the channel, height, and width dimensions, and flatten it into a one-dimensional feature vector. During the flattening process, a one-dimensional vector Internal The index mapping relationship of elements is defined as follows: ,in Therefore, the flattening equation can be expressed as:

[0088]

[0089] Indexing the two-dimensional space Mapped to a one-dimensional sequence index Define the index conversion formula as follows: .at this time, ,and Therefore, the first after being flattened Let the local vectors be denoted as .

[0090] Linear Projection

[0091] Set the hidden layer feature dimension of the Transformer backbone network Define a learnable linear projection weight matrix. and learnable bias vectors For each flattened image patch vector Perform the affine transformation independently and map it to The projected feature vector is obtained from the hidden layer feature space. :

[0092]

[0093] Sequence concatenation and class token prepending

[0094] All of Indexed by spatial sequence The order of the rows is concatenated into a two-dimensional sequence matrix. :

[0095]

[0096] To aggregate global image-level features, a separate learnable parameter vector is introduced as a classification token, denoted as . Classification tokens Prepositioned in sequence matrix The first row forms an extended sequence matrix containing the global token. ,in :

[0097] ;

[0098] One-dimensional absolute positional encoding injection

[0099] To overcome the permutation equivariance of the Transformer self-attention mechanism and enable it to perceive the original spatial topological relationships of image patches, an independent, learnable one-dimensional absolute position encoding matrix is ​​defined. Each row in the positional coding matrix The corresponding sequence Number of positions (of which) The global spatial location information of the location encoding matrix is ​​obtained by performing element-wise matrix addition between the location encoding matrix and the extended sequence matrix to obtain the final initial visual feature sequence. (Right now );

[0100] Standardized image data is obtained in S2. The steps are as follows:

[0101] S2-1: Will Mapped to the target image using bilinear interpolation. Bilinear interpolation is an existing technology.

[0102] For any discrete pixel coordinates in the target image ,in , Its continuous floating-point projection coordinates on the original image Defined as:

[0103]

[0104] Let the four nearest known integer coordinates of the projected point on the original map be... , , , The mathematical definition of its position and weight coefficient is:

[0105]

[0106]

[0107] The scaled feature output equation obtained by multiplying bilinear interpolation matrices in two-dimensional space is:

[0108] ;

[0109] S2-2: Using continuously uniformly distributed random variables to... The image is obtained by randomly flipping it horizontally. The calculation expression is as follows:

[0110]

[0111] in, Indicates the flip indicator function;

[0112] Introducing a continuously uniformly distributed random variable Uses the underlying default trigger threshold. Define the flip indicator function. :when hour, ;otherwise Through affine transformation Perform a centrally symmetric mapping on the axis to output the image. The analytical expression is: ;

[0113] S2-3: To Dimensional permutation and range continuity are performed to obtain Then on Z-score normalization yields the normalized tensor. The expression is as follows:

[0114]

[0115]

[0116] in, This represents the expected distribution vector of channel c. This represents the standard deviation distribution vector of channel c;

[0117] Change the pixel value range from discrete Linear compression to the continuous real number field Simultaneously, the tensor's dimensional space is transposed, changing the shape from... Convert to channel-priority form suitable for underlying operators Construct a new channel-priority coordinate system. Continuous tensor The calculation formula is:

[0118]

[0119] Let the expected distribution vectors of the three channels of the image be... The standard deviation distribution vector is For tensors Each channel dimension Perform translation and variance scaling operations independently to obtain a normalized tensor. The calculation formula is as follows:

[0120] ;

[0121] S2-4: Utilize Obtain standardized images The expression is as follows:

[0122]

[0123] in, This indicates the fill value for the occluded area. This indicates that the triggering control random variable is... A two-dimensional spatial set representing the occluded area. Represents the coordinates of any discrete pixel on image I.

[0124] Random occlusion based on area and aspect ratio sampling

[0125] Introducing random variables for trigger control (Default trigger probability) ).

[0126] when At that time, the erasure logic is executed. Let the total area of ​​the target image be... : Randomly sample the proportion of occlusion area from a uniform distribution Calculate the area of ​​the obstructed region. The aspect ratio of the occluded area is randomly sampled from a uniform distribution. Calculate the height of the occluding rectangle. With width (If the boundary is exceeded, resampling or truncation will be performed). The coordinates of the top-left corner of a rectangle are randomly sampled from a uniform distribution: and .

[0127] Define the two-dimensional spatial set of the occlusion region as Let the occlusion fill value be... .

[0128] The final output normalized image tensor It is obtained through the following piecewise function:

[0129] .

[0130] S3: Will The first Transformer decoder in input M passes through the LayerNorm layer, MSA module, and LayerNorm layer of each layer in the first Transformer decoder in sequence to obtain the result at the 1st epoch. Core branch trunk features of the layer ; Initial visual feature sequence Inputting the first layer (layernorm) of the Transformer yields the feature sequence matrix. Then Input the MSA module and output the feature matrix. Then Inputting the second LayerNorm layer yields the core splitting feature matrix. ; After passing through layer l, we obtain... ;

[0131] S4: Will Inputs into the MLP layer and adapter unit respectively , The self-attention feature matrix is ​​obtained through the MLP layer. , through Obtain the preliminary dimension-reduced projection feature matrix ,in, Represents the first... layer, The expression is as follows:

[0132]

[0133]

[0134]

[0135] in, This represents the globally shared learnable matrix of the Transform. The layer-specific projection scaling matrix is ​​represented. Indicates the first Layer-specific bias parameters, Indicates length is The whole Column vectors D' represents the outer product; D' represents Spatial dimension, Indicates the first Lower layer The first in elements and , Indicates the first Lower layer The first in One element; a learnable scalar parameter; D' is the dimension of an extremely compressed low-dimensional bottleneck space, such as D'=64). and All of them are learnable scalar parameters;

[0136] Define the projection basis under a cross-layer shared down-projection basis.

[0137] To explicitly reuse the shared general graph representation capabilities across different target languages ​​in multi-layer networks, while avoiding exponential parameter inflation during cross-language adaptation, a learnable weight matrix is ​​defined that is globally shared across all Transformer layers. .

[0138] in, The main hidden layer feature dimension. For extremely compressed low-dimensional bottleneck space dimensions (such as ).

[0139] matrix The first in line, number Column elements are denoted as , where feature index Bottleneck Index This matrix carries a universal visual mapping prior for all cross-language tasks.

[0140] Layer-Specific Scaling Modulation

[0141] Because network layers of different depths extract features with different granularities and semantic abstraction levels (shallow layers emphasize local structure, while deeper layers emphasize higher-order semantics), layer-specific parameters must be introduced to differentiate the treatment of the shared base. The response intensity.

[0142] For the Each layer defines its own, extremely lightweight layer-specific downprojection scaling matrix. .

[0143] matrix The first in line, number Column elements are denoted as ,in .

[0144] In the underlying algebraic space, a shared basis is used. With scaling matrix Perform matrix multiplication to obtain the effective transformation matrix of this layer after intensity modulation. :

[0145]

[0146] Its corresponding number line, number The underlying cumulative expansion formula for column elements is:

[0147]

[0148] Fine-grained shift compensation via layer-specific bias

[0149] Scaling alone cannot fully recover the absolute offsets of each layer in the feature distribution locations and semantic response centers. Therefore, based on the modulation matrix, a further step is to introduce a third... Layer-specific bias parameters .

[0150] Based on a rigorous mathematical formalization, the layer-specific projection matrix of this layer is thoroughly constructed as follows: :

[0151]

[0152] (Mathematical note: To maintain efficiency for extreme parameters and satisfy the rules of matrix addition, in the underlying tensor operations of deep learning frameworks, ...) In fact, it is a dimension The one-dimensional row vector bias. During computation, it is distributed along the sequence dimension via an automatic broadcasting mechanism. Adding it to the projected feature matrix provides fine-grained distribution translation correction. However, in the purely algebraic definition of operator weights, its logical equivalent is directly applying a shift to the transformation matrix.

[0153] Tensor Dimensionality Reduction & Manifold Contraction of High-Dimensional Feature Manifolds

[0154] The splitting matrix after the second layer normalization (i.e., includes) Each token has [number] tokens. The input sequence of dimensional features, and the down projection matrix constructed in this layer. Perform a fully connected affine transformation.

[0155] This completes the dimensionality reduction mapping from the high-dimensional backbone space to the low-dimensional adapter bottleneck space, yielding the preliminary dimensionality reduction projection feature matrix of the adaptation branch. (Right now ).

[0156] S5: Will enter Then the low-dimensional matrix is ​​output. ;

[0157] Class token detachment and spatial sequence separation will reduce the dimension of the projected feature matrix. Strict slicing is performed along the row dimension (sequence dimension). The 0th row is extracted as the global classification token vector. :

[0158] Extract rows 1 to 196 as a pure spatial image patch feature sequence :

[0159]

[0160] Topological Reconstruction from 1D to 2D Grid: To restore the physical adjacency of image patches in the original chart image in two-dimensional space, the one-dimensional sequence must be transformed into a 2D grid. Folding backwards yields a 3D spatial tensor. The known number of mesh height divisions is... The width is divided into several parts. ( Define the reshaped space tensor as... (Right now (Following the Channel-First memory layout of the underlying convolution operator). Matrix The first in OK( ), No. List( ) elements Mapping to a 3D tensor absolute physical coordinates The equation is:

[0161]

[0162] Zero-Padding for Spatial Resolution Preservation The convolution kernel does not change the physical size of the feature map during scanning (preservation) ), must be on the tensor Fill the space boundary. Define a fill constant. Construct a space with a size of (Right now Zero-filled topological manifolds Its spatial piecewise mapping equation is rigorously expressed as:

[0163]

[0164] The range of fill coordinates is: .

[0165] Based on channel decoupling Depth-Decoupled Convolution Computation The core of Depthwise Convolution lies in breaking the mixing of information across channels. Each input channel performs an inner product with only one corresponding two-dimensional convolution kernel, thereby extracting "local line trends and boundary shapes" with extremely low parameter count. The local spatial weight tensor of depthwise convolution is defined. and channel bias vector .in Indicates the first In the nth channel convolution kernel The weighting coefficients of each relative position ( Let the output local spatial topological tensor after the convolution sliding window inner product operation be... (Right now For any physical point on the target mesh. The product sum expansion equation of its underlying discrete two-dimensional convolution is:

[0166] The Identity Residual Addition for Local Extraction module aims to maintain the stability of the feature flow while incorporating local spatial information and to avoid excessive perturbation of the original projected features by the convolution operation. After performing depthwise convolution, the output spatial tensor of the depthwise convolution is... With the original input space tensor of this module Perform element-wise tensor addition. This yields a preliminary local spatial feature tensor. :

[0167]

[0168] The SE channel attention mechanism uses global average pooling for spatial squeezing to evaluate each feature channel (total) along the channel dimension. The importance of the input tensor to the current hand-drawn chart discrimination lies first in the need to compress the two-dimensional spatial distribution information into a global channel descriptor. Define its first The statistics for each channel are Global average pooling is performed independently along the spatial dimension to obtain the channel statistics description vector. Its first Statistics of each channel The exact summation equation is:

[0169]

[0170] At this point, the local spatial features are compressed into channel description vectors. This vector condenses the global receptive field information of 64 channels across the entire image.

[0171] Double layer Channel Dependency Excitation via Convolution To learn the nonlinear interactions between channels and control model complexity, Convolutions generates channel weights through a bottleneck structure of "dimensionality reduction followed by dimensionality increase".

[0172] Define the first layer Weight matrix of convolution (corresponding to dimensionality reduction of fully connected layers) With bias (in (For compression dimensions). Define the second layer. The weight matrix of convolution (corresponding to the dimensionality increase of fully connected layers) With bias By introducing the ReLU activation function and the Sigmoid gate function, strictly constrained... Interval channel normalized weight vector Its multi-layer composite matrix mapping equation is:

[0173]

[0174] Let vector The Each component is It represents the first adaptive learning of the network. The "importance discrimination coefficient" of each feature channel.

[0175] Channel-wise gating and recalibration of feature tensors utilizes the generated weight vectors For the initial local spatial feature tensor Perform adaptive feature recalibration. Define the recalibrated output tensor as... Its underlying element-wise multiplication analytical equation is:

[0176]

[0177] in, , , At this point, channels containing redundant information are greatly suppressed ( The channels with high discrimination ability for hand-drawn lines and geometric contours are effectively amplified. ).

[0178] Compression mapping dimensionality reduction of the channel feature fusion module (Channel Mixer: First Convolution for Compression) channel feature fusion adopts compression mapping strategy (i.e.) To filter redundant information. Define the first layer of channel fusion. Convolution weight matrix With bias Recalibration characteristics Perform a linear transformation along the channel dimension to obtain the intermediate compressed tensor. :

[0179]

[0180] Nonlinear activation of channel feature fusion (Channel Mixer: GeLU Activation) on intermediate compression tensors Apply Gaussian error linear units (GeLU) element-wise. Let the tensor... The The spatial location of each channel The element at is Introducing an error function The activated feature tensor is obtained. :

[0181]

[0182] Channel feature fusion upscaling and ultimate low-level residual summation (Channel Mixer: Second) Convolution & Residual Addition defines the second-layer channel fusion. Convolution weight matrix With bias Restore the activation features to the bottleneck dimension. The convolution output tensor is obtained. :

[0183]

[0184] Core Residual Addition: To ensure smooth backpropagation of gradients in deep networks and preserve the original response information after recalibration by the SE module, the output tensor of the channel fusion module is added. With the input tensor of this module By performing rigorous element-wise residual tensor addition, we obtain the spatial output tensor after local perception enhancement. :

[0185]

[0186] After the reverse sequence flattening and global reconstruction of the spatial mesh are completed, the three-dimensional tensor must be... Convert it back to a Transformer-compatible sequence format. First, convert it to... Inversely flattened into a two-dimensional sequence matrix Its memory mapping relationship is as follows:

[0187]

[0188] Subsequently, the aforementioned stripped classification tokens Reassemble the sequence into the beginning to obtain the complete sequence matrix:

[0189]

[0190] To maintain consistency with the training phase, a random dropout operation is performed on the recombined complete sequence matrix, ultimately yielding the low-dimensional output matrix of step S5. :

[0191] ;】

[0192] S6: Will enter Obtain the high-dimensional matrix The expression is as follows:

[0193]

[0194] in, Indicates the first The layer is modulated using a dedicated up-projection transformation matrix. Indicates the first Layer bias vector;

[0195] Construction of Cross-Layer Shared Up-Projection Basis

[0196] To maximize parameter reuse efficiency, the upprojection mapping does not introduce a new, large weight matrix, but instead directly calls the shared basis defined in the dimensionality reduction stage at the algebraic level. (in (The main feature dimension).

[0197] Perform a matrix transpose operation on it to obtain a shared upper projection basis across layers. (Right now ).

[0198] Let the original matrix be... The line, number Column elements are ( , Then the transpose matrix The line, number The elements of a column are strictly defined as .

[0199] Dimensional Modulation via Layer-Specific Up-Projection Scaling Matrix

[0200] Similar to the dimensionality reduction process, it must be the first... The top projection of the layer introduces a dedicated lightweight scaling matrix. (Right now ), to differentiate the layer's response to the shared base The response amplitude.

[0201] Let matrix The line, number Column elements are ( ).

[0202] (Note from the underlying mathematics department: To satisfy the dimension matching principle of matrix multiplication in linear algebra) The scaling matrix needs to be applied to the transpose basis from the left.

[0203] Performing the inner product of the scaling matrix and the transpose basis yields the first... The dedicated upprojection transformation matrix of the modulated layer :

[0204]

[0205] The first in the matrix line, number Column elements The underlying cumulative analytical expression is:

[0206] High-dimensional manifold mapping and bias compensation for adapted branches

[0207] Definition of the first Layer-specific upprojection bias vector (Right now Let its first... Each component is .

[0208] Low-dimensional sequence matrices containing local features With the modulated upprojection matrix Perform a fully connected affine transformation along the sequence dimension. Broadcast plus bias vector .

[0209] Thus, the inverse dilatation mapping from the adapter bottleneck space back to the main manifold space is completed, yielding the final high-dimensional output matrix of the adapting branch. (Right now );

[0210] S7: Calculation process The final feature matrix of the layer The expression is as follows:

[0211] ;

[0212] In the In the spatial structure of a layered Transformer, we have three matrices of equal dimension: The initial backbone features are extracted and appended with the first layer of residuals after multi-head self-attention (MSA). Features of the feedforward network after the original Transformer's massive two-layer linear dilation and GELU activation; After passing through the Plot-Adapter branch, the local line trends and hand-drawn boundary shapes are integrated, and the enhanced local perception features of the higher dimension are perfectly reconstructed. Tensor addition is performed on these three matrices to obtain the final output of the transformer encoder at layer l. ;

[0213] S8: Will This process proceeds sequentially through all subsequent Transformer encoders, which are all Transformer encoders except the first Transformer decoder. These subsequent Transformer encoders share the same data processing procedure as the first Transformer decoder, meaning the output of the first Transformer decoder is processed sequentially. Equivalent to the input of the second Transformer decoder , get about Graph API probability sequence The value of K is determined by To determine, for example: if In the Python language scenario, K takes the value of 12, resulting in a 13-dimensional probability vector sequence. A probability threshold is then set. Input P into the recommendation module, and select P that is greater than... The probability corresponding to the graphical API composition The graphical API predicts and recommends a list and outputs it, and then makes a judgment. Greater than the threshold The expression is as follows:

[0214]

[0215] in, This represents the category index of the graphics API corresponding to the k-th probability value in P;

[0216] S9: Iterate through all data in D, repeating S2-S8, to obtain the graph API prediction recommendation list corresponding to all image data. For each image in D, label its own graph API prediction tag according to its corresponding graph API prediction recommendation list. Pair the ground truth label and predicted label of each image in D as a training sample. All training samples constitute the supervised loss training set. ;

[0217] S10: Constructing the loss function ,Will As input to L, using Train M, freezing the structural model parameters of M except for the adapter unit during training. Update the model parameters through gradient descent backpropagation. Stop training when the maximum number of iterations is reached or the loss function converges. At this point, the trained recommendation model M' is obtained.

[0218] 1. Constructing the loss function in S10 The steps are as follows:

[0219] S10-1: Constructing a loss function using binary cross-entropy loss The calculation formula is as follows:

[0220]

[0221] in, Indicates the first The sample at the th Unnormalized prediction outputs for each API category This represents the corresponding real label;

[0222] S10-2: Define the confidence factor for prediction accuracy Then use right Adaptive recalibration is performed to obtain The calculation formula is as follows:

[0223]

[0224] in, Represents the balance coefficient. Indicates the focusing coefficient;

[0225] S10-3: Based on and Obtain the overall objective loss function The ultimate doubly discrete summative expansion is expressed as follows:

[0226]

[0227] in, This represents the total number of training samples. This indicates the total number of graphics API categories.

[0228] Multi-label binary cross-entropy loss calculation

[0229] During the model training phase, the batch size is set to... For the first The sample at the th The unnormalized prediction output (Logit) for each API category is denoted as... Its corresponding ground truth is denoted as .

[0230] First, the basic class-by-class binary cross-entropy loss (BCE Loss) is calculated. Based on the principle of extreme values ​​of information entropy, The mathematical formal definition of is:

[0231] ;

[0232] Combating long-tail imbalance in multi-label Focal Loss: Dynamic Reweighting

[0233] To address the issue that simple samples (such as high-frequency basic graphs) dominate the gradient direction while features of difficult samples (such as rare special structures) are overwhelmed, a modulation mechanism of Focal Loss is introduced.

[0234] First, define the confidence factor for prediction accuracy. : When the model's predictions are closer to the true labels, , When the prediction is seriously wrong, .

[0235] Subsequently, an equilibrium coefficient was introduced. With focus coefficient ,pass This nonlinear adjustment factor adaptively recalibrates the basic cross-entropy loss: ;

[0236] For including One sample The entire training batch of each category, based on and Obtain the overall objective loss function The ultimate double discrete cumulative expansion; the objective function's partial derivatives are obtained through backpropagation, and the gradient flow will follow this... A complex mathematical step is transmitted back along the way, precisely updating... All learnable parameters.

[0237] S11: Select the image X to be recommended, and input X into M' to obtain a list of recommended image APIs for X.

[0238] The operational procedures for each stage of implementation are as follows:

[0239] 1. Training Phase

[0240] During the training phase, a training sample set of charts aligned with the graphics API category set is first constructed. Preferably, manual redrawing can be organized based on standard chart reference samples to form hand-drawn chart samples with differences in real hand-drawn style, and each sample is labeled using a multi-label method. Subsequently, all training images are uniformly preprocessed and input into the pre-trained visual Transformer backbone network. The backbone network is kept frozen, and only the adaptation branch, local perception enhancement unit, and classification head are trained.

[0241] In each selected Transformer encoding layer, the input features are first mapped to the bottleneck space through a cross-layer shared projection matrix. Then, a layer-specific scaling matrix and a layer-specific bias term are used to construct the differential mapping for the current layer. Subsequently, local perceptual enhancement operations are performed within the bottleneck space. First, local neighborhood information is extracted using depthwise convolution. Then, channel responses are recalibrated using SE channel attention. Finally, nonlinear mixing is completed through a channel fusion structure, and the enhancement result is mapped back to the original dimension via upprojection. Multi-label Focal Loss is used to optimize the model parameters during training.

[0242] 2. Reasoning Stage

[0243] During the inference phase, the user inputs a hand-drawn chart image and specifies the target programming language. The system performs the same preprocessing as in the training phase on the input image, then uses a shared visual backbone and parameter reuse adaptation structure to obtain a joint global-local feature representation. Subsequently, it loads the lightweight parameter set and classification head corresponding to the target programming language, outputs predicted probabilities for candidate image APIs, and outputs one or more recommended APIs based on thresholds or ranking rules.

[0244] 3. Multilingual extensions

[0245] When scaling to new target programming languages, this invention eliminates the need to copy and retrain the entire backbone model. Instead, it allows for rapid adaptation to new language scenarios by incrementally training the layer-specific scaling matrix, layer-specific bias terms, and classification head corresponding to the target language, while maintaining the frozen backbone and shared projection matrix. This implementation significantly reduces model maintenance costs in multilingual deployments.

[0246] 4. Module replacement

[0247] Without departing from the overall concept of this invention, the local spatial feature extraction module can use other lightweight local modeling operators instead of depthwise convolution; the channel response recalibration module can use other channel attention mechanisms; and the channel feature fusion module can use other lightweight channel hybrid structures. Any implementation that achieves local spatial enhancement, channel response adjustment, and feature fusion in the adaptation branch, and works in conjunction with parameter reuse strategies to complete cross-language graphics API recommendations for hand-drawn charts, should fall within the scope of this invention.

[0248] Experimental content and results

[0249] The above effects were demonstrated through experimental comparison.

[0250] To verify the effectiveness of the lightweight CNN module in the proposed Plot-Adapter, this section loads ImageNet-21K and MAE pre-trained weights under a unified ViT-B / 16 backbone network framework, and conducts a systematic comparative analysis with full parameter fine-tuning, linear probing, and two typical PEFT methods: VPT and AdaptFormer.

[0251] from Figure 2 and Figure 3 The overall experimental results show that, under the same fine-tuning settings, the Plot-Adapter, which incorporates a CNN module, achieved higher mAP in most experiments compared to fine-tuning methods such as Linear, VPT, and AdaptFormer, demonstrating a more stable performance advantage overall.

[0252] As shown in Table 1, under the In21K pre-trained weights, Plot-Adapter achieved the highest performance among the fine-tuning methods on all three datasets. On the Python-13 dataset, Plot-Adapter achieved a peak mAP of 68.78%, a significant improvement over VPT's peak mAP of 66.95%, AdaptFormer's peak mAP of 67.02%, and Linear's peak mAP of 66.90%. On the R-32 dataset, Plot-Adapter achieved a peak mAP of 44.62%, also higher than VPT's peak mAP of 42.76%, AdaptFormer's peak mAP of 42.67%, and Linear's peak mAP of 41.34%. On the HDpy-13 hand-drawn dataset, Plot-Adapter achieved a peak mAP of 83.40%, a significant improvement over VPT's peak mAP of 76.44% and AdaptFormer's peak mAP of 78.59%.

[0253] Table 1. Experimental results under In21K weights

[0254] Fine-tuning method Python-13 (mAP%) R-32 (mAP%) HDpy-13 (mAP%) Full-tuning 67.65 43.73 80.12 Linear 66.90 41.34 76.23 VPT-4 66.84 42.53 76.18 VPT-8 66.95 42.76 75.90 VPT-16 65.85 41.53 76.44 AdaptFormer-4 67.02 42.41 77.09 AdaptFormer-16 66.62 41.78 77.67 AdaptFormer-64 66.71 42.67 78.59 Plot-Adapter-4 68.26 44.62 82.05 Plot-Adapter-16 68.16 44.30 82.89 Plot-Adapter-64 68.78 44.32 83.40

[0255] As shown in Table 2, a similar trend can be observed under the MAE pre-trained weight condition. Plot-Adapter achieves a highest mAP of 69.12% on the Python-13 dataset, higher than VPT's highest mAP of 61.53%, AdaptFormer's highest mAP of 62.05%, and Linear's mAP of 62.78%. On the R-32 dataset, Plot-Adapter's highest mAP of 40.92% also outperforms VPT's highest mAP of 39.16% and AdaptFormer's highest mAP of 39.83%, and is close to the performance of the Linear method. On the HDpy-13 dataset, Plot-Adapter achieves a highest mAP of 79.90%, also higher than VPT's highest mAP of 76.81%, AdaptFormer's highest mAP of 75.38%, and Linear's mAP of 76.65%.

[0256] Table 2 Experimental results under MAE weighting

[0257] Fine-tuning method Python-13 (mAP%) R-32 (mAP%) HDpy-13 (mAP%) Full-tuning 64.49 40.73 78.73 Linear 62.78 39.78 76.65 VPT-4 60.14 38.67 76.81 VPT-8 61.53 39.13 74.00 VPT-16 61.16 39.16 75.24 AdaptFormer-4 61.40 38.85 75.38 AdaptFormer-16 62.05 39.83 75.32 AdaptFormer-64 61.13 39.41 75.22 Plot-Adapter-4 69.10 40.89 79.24 Plot-Adapter-16 69.12 40.35 79.09 Plot-Adapter-64 69.09 40.92 79.90

[0258] Overall, while keeping the backbone network frozen and only updating the fine-tuning modules, Plot-Adapter exhibits more stable performance advantages across different pre-trained weights and datasets. The results show that introducing a lightweight CNN module with local spatial modeling capabilities into the adapter structure effectively enhances the model's ability to represent the spatial structure information of chart images. Compared to the PEFT method, which relies solely on cue vectors or simple nonlinear mappings for feature adjustment, the convolution operator can aggregate features within the local pixel neighborhood, thus better capturing spatial structure patterns in chart images. This alleviates the performance constraints imposed by freezing the backbone network to some extent, resulting in more stable performance gains for the model.

[0259] Further observation of the experimental results in Tables 1 and 2 reveals that, regardless of the Python-13, R-32, or HDpy-13 datasets, and regardless of whether In21K or MAE pre-trained weights are used as the backbone network initialization method, the mAP of Full-tuning is consistently higher than that of Linear and all configurations of VPT and AdaptFormer. This indicates that, without being limited by the parameter update range, allowing the target task data to update the backbone network as a whole typically yields stronger task adaptability than conventional PEFT.

[0260] More notably, even with the backbone network completely frozen, Plot-Adapter still outperforms Full-tuning in multiple experimental settings. With In21K pre-trained weights, Plot-Adapter's highest mAP on all three datasets surpasses the corresponding Full-tuning results; this performance also holds true with MAE pre-trained weights. Compared to simply fine-tuning all parameters of the original ViT, Plot-Adapter introduces an additional CNN local feature modeling branch outside the ViT backbone, fusing local spatial features with global representation information to obtain feature representations more suitable for chart vision tasks, achieving superior performance even with the backbone network frozen.

[0261] Table 3 Comparison of parameter counts between Plot-Adapter and other fine-tuning methods

[0262] Fine-tuning method Average number of parameters (M) Full-tuning 85.81 Linear 85.81 VPT-4 0.04 VPT-8 0.07 VPT-16 0.15 AdaptFormer-4 0.08 AdaptFormer-16 0.30 AdaptFormer-64 1.19 Plot-Adapter-4 0.06 Plot-Adapter-16 0.08 Plot-Adapter-64 0.19

[0263] Table 3 presents the statistical results of the number of trainable parameters for different fine-tuning methods. Overall, Plot-Adapter remains within the lightweight range of parameter-efficient fine-tuning methods, with a parameter size far lower than that of Full-tuning and Linear methods, and similar in magnitude to VPT. However, from the perspective of the method design in this chapter, a more practically meaningful comparison is with AdaptFormer, as both belong to adapter-based parameter-efficient fine-tuning methods. Therefore, the parameter size comparison between Plot-Adapter and AdaptFormer reflects that this chapter maintains the parameter efficiency advantage of the adapter structure even after introducing the lightweight CNNBlock.

[0264] Specifically, with a bottleneck dimension of 4, the number of parameters for AdaptFormer-4 and Plot-Adapter-4 are 0.08M and 0.06M, respectively; with a bottleneck dimension of 16, they are 0.30M and 0.08M, respectively; and with a bottleneck dimension of 64, they are 1.19M and 0.19M, respectively. It can be seen that under all bottleneck dimension settings examined, the number of trainable parameters for Plot-Adapter is lower than that for AdaptFormer, and this difference widens further as the bottleneck dimension increases. Furthermore, combining Tables 1 and 2, Plot-Adapter's mAP is also superior to that of AdaptFormer. This result indicates that the lightweight CNNBlock introduced in this chapter does not sacrifice performance for increased parameter size, but rather effectively enhances the model's ability to extract local structural features while maintaining its lightweight characteristics.

[0265] Compared to AdaptFormer, Plot-Adapter exhibits both lower parameter count and higher performance, demonstrating the effectiveness of the local augmentation structure adopted in this chapter. By introducing functionally clear and well-defined local augmentation units into the lightweight adaptation branch, the model can retain its global modeling capabilities while addressing the shortcomings in representing local spatial patterns. On the other hand, it also maintains the requirements of lightweight adaptation for cross-language tasks, achieving improvements in both performance and parameter efficiency.

[0266] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for recommending graphics APIs based on a multi-scenario lightweight adapter, characterized in that: Includes the following steps: S1: Obtain a publicly available image dataset D. Each image sample in D includes image data and image ground truth labels. The image ground truth labels refer to the actual graphics API used to draw the image sample and the target programming language scenarios corresponding to these graphics APIs. ; A recommendation model M is constructed based on Transformer, comprising several cascaded Transformer decoders and a recommendation module. In addition to the original structure of each Transformer decoder, an adapter unit is added after the MSA layer of each Transformer decoder, parallel to the MLP layer. The adapter unit includes the following cascaded shared downprojection matrix modules. Local perception enhancement unit module And shared projection matrix module The shared projection matrix module mentioned above The local perception enhancement unit module is used to obtain the preliminary dimensionality reduction projection feature matrix. Composed of a convolutional network module (CNN), the shared up-projection matrix is ​​used to obtain a high-dimensional output matrix; The recommendation module is used to output a recommended list of graphics APIs; S2: Select any image data from D. ,right Data preprocessing is performed to obtain standardized image data. Then Image patch segmentation and feature embedding are performed sequentially to obtain an initial visual feature sequence. ; in, Represents the position encoding matrix, This represents an extended sequence matrix containing global tokens. express… Indicates the first after flattening Local vectors Represents the linear projection weight matrix. Represents a learnable bias vector; Represents a learnable parameter vector; S3: Will In the first Transformer decoder of input M, the result is obtained by sequentially passing through the LayerNorm layer, the MSA module, and the LayerNorm layer of each layer in the first Transformer decoder. Core branch trunk features of the layer ; S4: Will Inputs into the MLP layer and adapter unit respectively , The self-attention feature matrix is ​​obtained through the MLP layer. , through Obtain the preliminary dimension-reduced projection feature matrix ,in, Represents the first... layer, The expression is as follows: in, This represents the globally shared learnable matrix of the Transform. The layer-specific projection scaling matrix is ​​represented. Indicates the first Layer-specific bias parameters, Indicates length is The whole Column vectors D' represents the outer product; D' represents Spatial dimension, Indicates the first Lower layer The first in elements and , Indicates the first Lower layer The first in One element; one learnable scalar parameter; S5: Will enter Then the low-dimensional matrix is ​​output. ; S6: Will enter Obtain the high-dimensional matrix The expression is as follows: in, Indicates the first The layer is modulated using a dedicated up-projection transformation matrix. Indicates the first Layer bias vector; S7: Calculation process The final feature matrix of the layer The expression is as follows: ； S8: Will After passing through all subsequent Transformer encoders in sequence, we obtain information about... Graph API probability sequence The value of K is determined by To determine and set a probability threshold. Input P into the recommendation module and select P that is greater than... The probability corresponding to the graphical API composition The graphical API predicts and recommends a list and outputs it, and then makes a judgment. Greater than the threshold The expression is as follows: in, This represents the category index of the graphics API corresponding to the k-th probability value in P; S9: Iterate through all data in D, repeating S2-S8, to obtain the graph API prediction recommendation list corresponding to all image data. For each image in D, label its own graph API prediction tag according to its corresponding graph API prediction recommendation list. Pair the ground truth label and predicted label of each image in D as a training sample. All training samples constitute the supervised loss training set. ; S10: Constructing the loss function ,Will As input to L, using Train M, freezing the structural model parameters of M except for the adapter unit during training. Update the model parameters through gradient descent backpropagation. Stop training when the maximum number of iterations is reached or the loss function converges. At this point, the trained recommendation model M' is obtained. S11: Select the image X to be recommended, and input X into M' to obtain a list of recommended image APIs for X.

2. The method for recommending graphics APIs based on a multi-scenario lightweight adapter as described in claim 1, characterized in that: Standardized image data is obtained in S2. The steps are as follows: S2-1: Will Mapped to the target image using bilinear interpolation. ; S2-2: Using continuously uniformly distributed random variables to... The image is obtained by randomly flipping it horizontally. The calculation expression is as follows: in, Indicates the flip indicator function; S2-3: To Dimensional permutation and range continuity are performed to obtain Then on Z-score normalization yields the normalized tensor. The expression is as follows: in, This represents the expected distribution vector of channel c. This represents the standard deviation distribution vector of channel c; S2-4: Utilize Obtain standardized images The expression is as follows: in, This indicates the fill value for the occluded area. This indicates that the triggering control random variable is... A two-dimensional spatial set representing the occluded area. Represents the coordinates of any discrete pixel on image I.

3. The method for recommending graphics APIs based on a multi-scenario lightweight adapter as described in claim 2, characterized in that: The loss function is constructed in S10. The steps are as follows: S10-1: Constructing a loss function using binary cross-entropy loss The calculation formula is as follows: in, Indicates the first The sample at the th Unnormalized prediction outputs for each API category This represents the corresponding real label; S10-2: Define the confidence factor for prediction accuracy Then use right Adaptive recalibration is performed to obtain The calculation formula is as follows: in, Represents the balance coefficient. Indicates the focusing coefficient; S10-3: Based on and Obtain the overall objective loss function The ultimate doubly discrete summative expansion is expressed as follows: in, This represents the total number of training samples. This indicates the total number of graphics API categories.

Images