UI effect diagram recognition method, system and medium based on multimodal large model

Abstract

This application discloses a method, system, and medium for UI effect image recognition based on a multimodal large model. The method includes the following steps: constructing a multimodal pre-trained model based on image encoding, multimodal projection, and text encoding; performing fine-tuning training based on historical image data and the multimodal pre-trained model to obtain the multimodal large model; constructing a vector database based on a built-in component data structure; obtaining a UI effect image to be recognized, and outputting first recognition data based on the multimodal large model and the UI effect image to be recognized; and using a retrieval algorithm to search the vector database for the first recognition data and output generation information corresponding to the UI effect image to be recognized. The beneficial effect of this application is to ensure the consistency and efficiency of UI effect image generation.

Description

Technical Field

[0001] The present application relates to the field of image recognition and generation technology, and in particular to a method, system and medium for recognizing UI effect diagrams based on a multimodal large model. Background Art

[0002] The complexity of UI rendering recognition and generation stems primarily from the diversity of design elements, the complexity of layout and rendering, and the complexity of interactions. These complexities increase the difficulty of recognition and generation, making the process cumbersome.

[0003] Related technologies typically use model training to identify and generate UI renderings. However, insufficient model generalization is a common challenge in machine learning and deep learning. For UI rendering recognition and generation tasks, insufficient model generalization can lead to poor performance under new data or scenarios.

[0004] At the same time, due to the large number of components and their complex functions, developers often find it difficult to fully understand and apply these components, resulting in inconsistent UI renderings and reducing user experience.

[0005] The patent, "Method for Generating Web Code from a UI Based on Generative Adversarial and Convolutional Neural Networks," published with publication number CN110377282A on October 25, 2019, specifically discloses the following steps: constructing a mapping relationship between the display effects of HTML elements and their source code; identifying the locations of HTML elements in image I; generating complete HTML code for I; calculating the similarity Sim1 between the manually written HTML code and the generated HTML code; calculating the similarity Sim2 between image I and image I1 generated using the generated HTML code; balancing Sim1 and Sim2 to Sim0, determining the relationship between Sim0 and a threshold t; if Sim0 is less than t, repeating the above process; otherwise, executing the next step; and obtaining a generative model M from image to HTML code after training in the previous step. Inputting the UI image to be processed into model M yields the corresponding HTML code. This solution relies on a multi-step deep learning model, which not only requires a large amount of training data but also involves complex pre-processing and post-processing steps, increasing development costs and making it difficult to ensure consistent generation results. Secondly, existing deep learning models are easily limited by training data when processing UI renderings, resulting in poor generalization ability of the model and difficulty in maintaining stable performance in different application scenarios.

[0006] The patent, "A UI Generation Method Based on Multimodal AI," published with publication number CN115756455A and date of publication on March 7, 2023, specifically discloses obtaining a requirements document for the target website's official website. Based on the requirements document, the requirements are refined into functional aspects corresponding to the different functions of the specific UI interface. Feature extraction is then performed on different functional aspects to form feature aspects. Feature sets provided by the different functional aspects are fused to establish connections between the features to provide comprehensive coverage of the requirements. The fused feature sets are then encapsulated into feature vectors required by a neural network prediction model. The feature vectors are then entered into a UI automatic generation tool powered by multimodal AI technology to generate UI images, predicting and generating UI images corresponding to the functional requirements. The generated UI images for different functional requirements are then reviewed, and any deficiencies are locally optimized based on the input features for secondary generation. Finally, the generated UI images are delivered for use. During the UI rendering generation process, different designers or developers use different components and styles. This solution cannot ensure a consistent UI design, impacting the user experience and interface aesthetics. Summary of the Invention

[0007] In response to the problem in the prior art that UI image recognition and generation cannot balance efficiency and consistency, the present application provides a UI effect image recognition method, system and medium based on a multimodal large model. The multimodal large model is obtained by constructing a multimodal pre-training model based on image coding, multimodal projection and text coding, and fine-tuning the training with historical image data. While achieving comprehensive recognition of charts, components and text content in UI effect images, the recognition accuracy and efficiency are improved. The retrieval algorithm is used to search the vector database constructed based on the built-in component data structure for the first recognition data, and the component vector that matches the first recognition data is quickly found. Based on these vectors, the generation information of the corresponding UI effect image to be identified is obtained, thereby ensuring the consistency and efficiency of UI effect image generation.

[0008] To achieve the above-mentioned technical objectives, the present application provides a technical solution, which is a UI effect diagram recognition method based on a multimodal large model, comprising the following steps: constructing a multimodal pre-training model based on image encoding, multimodal projection and text encoding; performing fine-tuning training based on historical image data and the multimodal pre-training model to obtain a multimodal large model; constructing a vector database based on a built-in component data structure; obtaining a UI effect diagram to be identified, and outputting first recognition data based on the multimodal large model and the UI effect diagram to be identified; using a retrieval algorithm to search the vector database for the first recognition data, and outputting generation information corresponding to the UI effect diagram to be identified.

[0009] Furthermore, the fine-tuning training is performed based on the historical image data and the multimodal pre-training model to obtain the multimodal large model, including: obtaining historical image data and performing data preprocessing on the historical image data; using LoRA to adjust the weight matrix of the multimodal pre-training model; inputting the historical image data after data preprocessing into the adjusted multimodal pre-training model, performing fine-tuning training, and obtaining the multimodal large model.

[0010] Furthermore, the use of a retrieval algorithm to search the vector database for the first recognition data and output generation information corresponding to the UI effect diagram to be identified also includes: using RAG to search the vector database for the first recognition data based on a preset prompt template to obtain component data with the highest matching similarity; obtaining structural content data of the UI effect diagram to be identified, integrating the component data and structural content data to obtain generation information corresponding to the UI effect diagram to be identified.

[0011] Furthermore, the fine-tuning training is performed based on historical image data and the multimodal pre-trained model to obtain the multimodal large model, which also includes: constructing a triplet containing a chart, a corresponding text description, and related questions; comparing the candidate model and the output results of the multimodal large model corresponding to the triplet based on the evaluation model, and outputting the evaluation results of the multimodal large model; and judging whether the multimodal large model needs iterative training based on the evaluation results.

[0012] Furthermore, the constructing of a vector database based on the built-in component data structure includes: obtaining the relationship between graph elements to construct a knowledge graph; obtaining the built-in component data structure, and constructing a vector database based on the knowledge graph and the built-in component data structure.

[0013] Furthermore, the construction of a vector database based on the knowledge graph and the built-in component data structure also includes: converting the built-in component data structure into JSON object data; extracting key attributes from each JSON object data and converting them into feature vectors; standardizing the feature vectors; and constructing associations for the standardized feature vectors based on the knowledge graph to obtain a vector database.

[0014] Furthermore, the fine-tuning training performed based on historical image data and the multimodal pre-training model also includes: obtaining key features of the historical image data, and matching fine-tuning levels based on the key features of the historical image data; and using LoRA to perform fine-tuning training on the multimodal pre-training model based on the fine-tuning levels and historical image data.

[0015] Furthermore, the obtaining of key features of historical image data and the matching fine-tuning level based on the key features of historical image data include: obtaining historical image data and identifying key features of the historical image data; if the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning level; if the key features are complex structural features, the intermediate convolutional layer of the image encoding is used as the fine-tuning level; if the key features are global semantic features, the top convolutional layer or the fully connected layer of the image encoding is used as the fine-tuning level.

[0016] Furthermore, the use of LoRA to perform fine-tuning training on the multimodal pre-training model according to the fine-tuning level and historical image data also includes: using LoRA to adjust the weight matrix of the multimodal pre-training model according to the fine-tuning level; performing image encoding and text encoding on the historical image data to obtain visual features and language features; mapping visual features and language features based on the cross-attention module, and calculating attention weights; obtaining fused visual features based on the attention weights and the gated fusion mechanism; updating fine-tuning parameters based on the loss function and the fused visual features, and performing fine-tuning training.

[0017] Furthermore, the acquiring of historical image data and the data preprocessing of the historical image data include: performing image normalization and size adjustment on the picture data in the historical image data; and performing information parsing and format conversion on the structural content data in the historical image data.

[0018] Furthermore, the resizing of the image data in the historical image data includes: obtaining the resolution of the multimodal pre-trained model, and obtaining an adjustment value based on the resolution of the multimodal pre-trained model and the original resolution of the image data; when the adjustment value is negative, calling the nearest neighbor interpolation to perform resizing on the image data; when the adjustment value is positive, calculating the image volume, and calling the corresponding order interpolation method according to the image volume gradient.

[0019] Another technical solution provided by the present application is a UI effect diagram recognition system based on a multimodal large model, which is used to implement the UI effect diagram recognition method based on the multimodal large model as mentioned above, including: a question and answer annotation module, which obtains historical image data based on user annotations and questions and answers; a fine-tuning training module, which performs fine-tuning training based on historical image data and a multimodal pre-trained model to obtain a multimodal large model; a vector database, which pre-stores a built-in component data structure; an identification generation module, which obtains a UI effect diagram to be identified, outputs first identification data based on the multimodal large model and the UI effect diagram to be identified, uses a retrieval algorithm to search the vector database for the first identification data, and outputs generation information corresponding to the UI effect diagram to be identified.

[0020] Another technical solution provided by the present application is a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processing device, the UI effect diagram recognition method based on the multimodal large model as described above is implemented.

[0021] The beneficial effects of this application are: 1. With the continuous increase of historical design drawing data and the enrichment of component libraries, the model can be continuously fine-tuned to improve the accuracy and efficiency of recognition. At the same time, the vector database can be adaptively adjusted without changing the trained multimodal big data, so it can adapt to different application scenarios and needs, and is more practical.

[0022] 2. For different historical image data, different fine-tuning methods are selected and the remaining layers are frozen at the same time. This reduces the amount of training calculations while being more targeted, achieving simultaneous improvements in efficiency and accuracy.

[0023] 3. By partially fine-tuning specific layers of the image encoding, the model can focus on learning features closely related to chart recognition, improving its understanding of chart details and thus enhancing recognition accuracy. This also avoids comprehensive fine-tuning of the entire image encoding, reducing computing resource consumption and training time, and achieving efficient fine-tuning.

[0024] 4. By achieving a deep fusion of visual and language information at the projection layer, the model is able to simultaneously understand and process both the visual features and textual information in a chart. Through cross-modal attention and gating mechanisms, the model is able to capture the complex connections between vision and language, generating more accurate and rich chart descriptions. This improves the model's understanding of charts and expands its application scope, enabling it to handle a wider variety of charts and more complex textual descriptions, thereby enhancing its generalization and practicality.

[0025] 5. By fine-tuning a small number of prefix parameters to adapt to downstream tasks, the model can quickly adapt to different graph recognition tasks without requiring large-scale adjustments to the entire model. This avoids model performance degradation or overfitting problems caused by excessive fine-tuning, reduces computing resource consumption, and makes model fine-tuning more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the UI effect diagram recognition method based on a multimodal large model in this application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific implementation method described here is only an optimal embodiment of this application, which is only used to explain this application and does not limit the scope of protection of this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] like Figure 1 As shown in the first embodiment of the present application, a UI effect diagram recognition method based on a multimodal large model includes the following steps:

[0029] Build a multimodal pre-training model based on image encoding, multimodal projection, and text encoding;

[0030] Perform fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model;

[0031] Build vector database based on built-in component data structure;

[0032] Obtain a UI effect diagram to be identified, and output first recognition data based on the multimodal large model and the UI effect diagram to be identified;

[0033] A search algorithm is used to search a vector database for the first recognition data, and generation information corresponding to the UI effect diagram to be recognized is output.

[0034] In this embodiment, a multimodal large model is obtained by constructing a multimodal pre-training model based on image coding, multimodal projection and text coding, and fine-tuning the training with historical image data. While achieving comprehensive recognition of charts, components and text content in UI renderings, the recognition accuracy and efficiency are improved. A retrieval algorithm is used to search the vector database constructed based on the built-in component data structure for the first recognition data, and the component vectors matching the first recognition data are quickly found. Based on these vectors, the generation information of the corresponding UI rendering to be identified is obtained to ensure the consistency and efficiency of UI rendering generation. With the continuous increase of historical design drawing data and the enrichment of the component library, the model is continuously fine-tuned to improve the recognition accuracy and efficiency. At the same time, the vector database is adaptively adjusted. It can adapt to different application scenarios and needs without changing the trained multimodal big data, and is more practical.

[0035] A multimodal pre-training model is constructed based on image encoding, multimodal projection, and text encoding. LLaVA-3 can be used as the multimodal pre-training model. By combining image encoding and text encoding, the model can simultaneously process and understand information from different modalities, adapting to situations where UI renderings contain both image information and text information. Based on multimodal projection, data from different modalities is mapped into the same semantic space, allowing the model to directly compare and fuse information from different modalities in the semantic space, outputting recognition results for UI renderings, and improving recognition efficiency and accuracy.

[0036] Fine-tuning training is performed based on historical image data and a multimodal pre-trained model to obtain a large multimodal model, including:

[0037] Acquire historical image data and perform data preprocessing on the historical image data;

[0038] Use LoRA to adjust the weight matrix of the multimodal pre-trained model;

[0039] The historical image data after data preprocessing is input into the adjusted multimodal pre-training model, and fine-tuning training is performed to obtain a large multimodal model.

[0040] LoRA (Low-Rank Adaptation) approximates the model's weight updates by adding a low-rank matrix instead of directly updating the original high-dimensional weight matrix. This enables rapid adjustment and adaptation of the model without significantly increasing computing and storage costs, thereby improving the accuracy and efficiency of UI rendering recognition.

[0041] Acquiring historical image data and performing data preprocessing on the historical image data include:

[0042] Performing image normalization and size adjustment on the image data in the historical image data;

[0043] Perform information parsing and format conversion on the structural content data in the historical image data.

[0044] Historical image data includes at least image data and structural content data containing component locations and attribute information. To ensure that image details are clearly captured during training, the pixel values ​​in the historical image data are normalized and resized. This improves the accuracy of subsequent training, facilitates rapid model convergence, and simultaneously increases model training efficiency.

[0045] The step of resizing the image data in the historical image data includes:

[0046] Obtain the resolution of the multimodal pre-trained model, and obtain an adjustment value based on the resolution of the multimodal pre-trained model and the original resolution of the image data;

[0047] Performs resizing on the image data based on a scaling algorithm and an adjustment value.

[0048] Normalization is performed on the image's pixel values, linearly mapping them from their original range to the range of [0, 1] or [-1, 1] to meet the model input requirements. Image data is resized based on the resolution of the multimodal pre-trained model's image encoding and the difference in the original resolution of the image data to maintain image clarity and detail. For example, when the original resolution of the image is greater than the resolution of the multimodal pre-trained model, the image is scaled down using a scaling algorithm. When the original resolution is less than the resolution of the multimodal pre-trained model, the image is scaled up using a scaling algorithm.

[0049] In this embodiment, performing size adjustment on the image data based on the scaling algorithm and the adjustment value further includes:

[0050] The corresponding scaling algorithm is called based on the adjustment value to perform size adjustment on the image data.

[0051] The scaling algorithm includes at least nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. Other high-order interpolation methods may also be included in the remaining embodiments. For example, when the original resolution of the image is greater than the resolution of the multimodal pre-training model, the adjustment value is a negative value, and when the original resolution of the image is less than the resolution of the multimodal pre-training model, the adjustment value is a positive value. At this time, when the adjustment value is a negative value, the nearest neighbor interpolation is called to perform size adjustment on the image data. When the adjustment value is a positive value, bilinear interpolation or bicubic interpolation is called to perform size adjustment on the image data. The nearest neighbor interpolation directly selects the source pixel value closest to the target pixel, and the calculation speed is fast, but it is easy to cause the edge of the image to be jagged when the image is enlarged. Therefore, when the image is reduced, the nearest neighbor interpolation is used to achieve fast scaling and reduce the amount of calculation. Bilinear interpolation calculates the weighted average of the four source pixels surrounding the target pixel to obtain the value of the target pixel, while bicubic interpolation considers the influence of more surrounding pixels to generate a smoother and more detailed image. Although it has a large computational load, it has a good scaling effect and is suitable for compensating pixels when enlarging an image. Therefore, when enlarging an image, bilinear interpolation or bicubic interpolation is used to achieve precise scaling. In other embodiments, resizing the image data based on the scaling algorithm and the adjustment value also includes:

[0052] When the adjustment value is negative, the nearest neighbor interpolation is called to perform size adjustment on the image data;

[0053] When the adjustment value is positive, the image volume is calculated and the corresponding order interpolation method is called according to the image volume gradient.

[0054] Different gradient thresholds are set according to the computing capacity of the system. For example, when the image size is less than the first gradient threshold, bicubic interpolation is called to resize the image data. When the image size is greater than the first gradient threshold, bilinear interpolation is called to resize the image data. Therefore, when the image size is small, a higher-order interpolation method is used for scaling to improve the image processing accuracy. When the image size is large, a lower-order interpolation method is used for scaling to improve the image processing efficiency. At this time, when the image size is large, there is more training data. Even if the scaling accuracy of the lower-order interpolation method is not as good as that of the higher-order interpolation method, the mutual compensation of a large amount of data can also ensure the accuracy of the final model construction. When the image size is small, the image scaling accuracy is improved by a higher-order interpolation method, so that more details can be extracted from each image during the model training process, compensating for the problem of small data size, and ensuring the accuracy of the final model.

[0055] Structural content data includes component locations and attribute information. Typically, image structural content data is stored in HTML files. We parse the HTML files in historical image data to extract the location information (x, y coordinates) and attribute information for each component. Attribute information includes at least dimensions (width, height). These location and attribute information are then converted into JSON format to facilitate model processing and improve the readability and scalability of the structural content data.

[0056] In this embodiment, the multimodal pre-trained model is composed of a CLIP image encoder, a multimodal projector, and a Vicuna text encoder. Using LoRA to adjust the weight matrix of the multimodal pre-trained model includes:

[0057] Introducing low-rank matrices into the weight matrices of multimodal pre-trained models.

[0058] The pre-processed historical image data is input into the adjusted multimodal pre-trained model, and fine-tuning training is performed to obtain a large multimodal model including:

[0059] The historical image data after data preprocessing is input into the adjusted multimodal pre-training model, forward propagation and backpropagation are performed, the low-rank matrix of the multimodal pre-training model is updated, and a multimodal large model is obtained.

[0060] A low-rank matrix is ​​introduced into the weight matrix of the multimodal pre-training model to reduce the dimension of the model parameters. The introduction of the low-rank matrix is ​​expressed as:

[0061] ;

[0062] in, To introduce the weight matrix of the low-rank matrix, is the original weight matrix, is a low-rank matrix, and is a matrix of lower dimension.

[0063] Through back propagation, the low-rank matrix in the model parameters is updated so that it can better adapt to the new task. The loss function of back propagation is:

[0064] ;

[0065] Among them, MSE represents mean square error, which is used for position loss, CrossEntropy represents cross entropy loss, which is used for classification loss. is the weight coefficient, N is the number of samples, is the actual observation value of the i-th sample, is the model prediction value of the i-th sample, is the actual probability distribution of the i-th sample, is the predicted probability distribution of the i-th sample.

[0066] The loss function consists of at least position loss and classification loss. Position loss, or localization error, measures the difference between the model's predicted icon or component location and its true location. Classification loss, or type error, measures the difference between the model's predicted icon or component type and its true type. During fine-tuning, the model considers both position loss and classification loss and updates model parameters through a backpropagation algorithm to minimize the weighted sum of these two losses, thereby improving the recognition accuracy of large multimodal models.

[0067] LoRA's low-rank decomposition technology is used to significantly reduce the computational cost and memory usage of fine-tuning, while ensuring the model's efficient performance in chart recognition tasks. This allows the model to leverage the advantages of existing large pre-trained models while quickly adapting to new tasks through a small amount of parameter updates. Especially in chart recognition scenarios, it takes into account both the specialized requirements of the task and the efficient use of resources, providing strong technical support for the development of intelligent, multimodal chart recognition assistants.

[0068] In this example, the low-rank matrix introduced by LoRA has a rank of 128 and a scaling factor of 256. The scaling factor is used to control the weight impact during fine-tuning training. The CLIP model (vit-large-patch14-336 version) is used as the image encoding architecture to process input image data and extract image visual features. The penultimate layer of the CLIP model is selected as the output layer to generate image visual features. A two-layer MLP plus GELU activation function is used as the multimodal projection architecture type. The learning rate for multimodal projection is set to convert the image visual features into a language model. The Vicuna-13B v1.5 model is used as the text encoding architecture. The training cycle is set to 1 epoch, the training batch size per device is 16, and the evaluation batch size per device is 4. One step of gradient is accumulated before each backpropagation. The model is saved every 50,000 steps. The maximum number of saved models is 1, and if exceeded, the old checkpoint is overwritten. The global learning rate is set to 0.0002, and the warm-up phase ratio is 3%, with the learning rate gradually increasing. We used a cosine annealing learning rate scheduler, enabled gradient checkpointing, reduced video memory usage, used bfloat16 precision for training to reduce video memory requirements, accelerated floating-point computations on TF32-capable hardware, set the maximum model input length to 2048 tokens, and used four parallel threads for data loading. With this setup, we fine-tuned the multimodal pre-trained model to produce a large multimodal model.

[0069] After fine-tuning the multimodal large model, the validation set is used to evaluate the model performance to ensure that the model can accurately identify and locate elements in the UI rendering. After pre-processing the historical image data, the historical image data is divided into a training set and a validation set according to a preset ratio. The training set is used to fine-tune the multimodal pre-trained model, and the validation set is used to verify and evaluate the multimodal large model. The verification evaluation includes classification evaluation and position evaluation. The cross-entropy loss is used for classification evaluation. The input data is forward propagated through the model to obtain the predicted probability distribution of each category. The true label is converted into a one-hot encoding format (One-Hot Encoding), and the cross-entropy loss is calculated using the cross-entropy formula:

[0070] ;

[0071] in, is the true label, is the predicted probability output by the model.

[0072] Position loss is used for position evaluation. Position loss is used to evaluate the performance task in the chart positioning prediction position. The model outputs the bounding box coordinates of each target (center point coordinates, width and height), obtains the bounding box coordinates of the real target, and uses smooth L1 loss or MSE to calculate the difference between the predicted bounding box and the real bounding box, that is, the position loss:

[0073] ;

[0074] in, are the true bounding box coordinates, are the predicted bounding box coordinates.

[0075] The classification loss and position loss are weightedly combined to calculate the total loss of the model, and the recognition accuracy of the current multimodal large model is judged based on the total loss.

[0076] In other cases, quantitative metrics are used to evaluate the image recognition capabilities of large multimodal models. Advanced language models are used to measure the quality of generated responses. Fine-tuning training is performed based on historical image data and multimodal pre-trained models. Obtaining a large multimodal model also includes:

[0077] Construct a triplet containing a diagram, its corresponding text description, and a related question;

[0078] Comparing the candidate models with the output results of the multimodal large model corresponding to the triplets based on the evaluation model, and outputting the evaluation results of the multimodal large model;

[0079] Determine whether the multimodal large model requires iterative training based on the evaluation results.

[0080] Specifically, a triplet is constructed containing a chart, its corresponding text description, and a related question. The candidate model (e.g., LLM) generates an answer based on the question and chart content, thereby creating a reference prediction based on the question and text description, which is generated using a text generation model. After obtaining the answers from the candidate model and the multimodal large model, we provide the question, chart information (in the form of text description), and the answers generated by the two models to the evaluator (e.g., text generation model). The evaluator will score the generated answers based on their relevance, accuracy, clarity, and level of detail, using a scoring system of 1 to 10, where higher scores indicate better overall performance. As shown in Table 1, this is an evaluation score sheet for one case. The recognition performance of the multimodal large model is obtained based on the evaluation score sheet, so that the parts with poor recognition performance can be adjusted in a targeted manner.

[0081] Table 1 Evaluation score sheet

[0082] ;

[0083] Based on the validation results, the training parameters and training set are adjusted, and multiple iterations of training and validation are performed until the large multimodal model achieves the expected performance. The expected performance metric can be a preset total loss threshold. When the total loss of the large multimodal model is less than the preset total loss threshold, the large multimodal model is considered to have achieved the expected performance metric. Based on the validation results, training parameters such as the learning rate and batch size can be adjusted to bring the output values ​​of the large multimodal model closer to the actual values. Data augmentation techniques (such as rotation, scaling, and cropping) can also be used to expand the training data to improve the model's generalization ability. The adjusted parameters and augmented data are re-entered into the model for iterative training. After each iteration of training, the validation set is used for evaluation until the model performance reaches the expected target.

[0084] Building a vector database based on built-in component data structures includes:

[0085] Obtain the relationship between graph elements to build a knowledge graph;

[0086] Obtain the built-in component data structure and build a vector database based on the knowledge graph and the built-in component data structure.

[0087] In UI renderings, there are logical relationships between chart elements. The chart elements and the relationships between them are represented in a structured manner based on the knowledge graph. Through nodes and edges, the knowledge graph can reflect the layout of the chart, the logical relationships between elements, and the overall semantic information.

[0088] For example, in this embodiment, the nodes of the knowledge graph represent elements: including the title, axis, data series, data points, labels, etc. of the chart, and each node contains the attributes of the element (type, coordinates, text content, etc.). The edges of the knowledge graph represent relationships: the relationship between elements, such as "axis-data series", "data series-data point", "data point-label", etc. The knowledge graph is converted into a fixed-length vector V_input that can capture global and local features through the graph convolutional network (GCN) embedding model, capturing the global and local features of the chart, thereby learning the representation of the chart structure and the relationship between elements. For example:

[0089] node:

[0090] -N1: Title ("Sales Trends")

[0091] -N2: X axis ("month")

[0092] - N3: Y axis ("sales")

[0093] -N4: Data Series ("Product A")

[0094] -N5: data point 1 ("value, coordinate")

[0095] -N6: Data point 2 ("value, coordinate")

[0096] side:

[0097] -N1 and N4: Title-related data series

[0098] -N2 and N5 / N6: X-axis associated data points

[0099] -N3 and N5 / N6: Y-axis associated data points

[0100] -N4 and N5 / N6: Data series contains data points.

[0101] Built-in component data structures include UI chart components and their data structures. UI chart components include at least Spc charts, dashboards, line charts, scatter charts, curve charts, bar charts, bubble charts, funnel charts, donut charts, Gantt charts, histograms, spider charts, progress bars, area charts, and pie charts. Detailed property descriptions are provided for each component, including component type, component ID, display name, label, image name, position (x, y coordinates), width, and height. The built-in component data structures are organized into JSON format and vectorized.

[0102] Building a vector database based on the knowledge graph and built-in component data structures also includes:

[0103] Convert built-in component data structures into JSON object data;

[0104] Extract key attributes from each JSON object data and convert them into feature vectors;

[0105] Normalize the feature vector;

[0106] The normalized feature vectors are correlated according to the knowledge graph to obtain a vector database.

[0107] Extract key attributes (e.g. component type, location, size, etc.) from each JSON object and convert them into feature vectors. For example, vectorize component location information:

[0108] ;

[0109] in, is the characteristic vector of component position information, x is the component x coordinate, y is the component y coordinate, is the component width, is the component height.

[0110] The feature vectors are normalized to ensure that all features are on the same scale. In this embodiment, the normalization is achieved by subtracting the mean and dividing by the standard deviation:

[0111] ;

[0112] in, is the normalized eigenvector, is the mean, is the standard deviation.

[0113] The normalized feature vectors are stored in a vector database. Each vector is associated with its corresponding JSON object based on the knowledge graph to facilitate subsequent retrieval and call.

[0114] In this embodiment, a basic vector database is constructed using database technologies for efficient vector retrieval, such as FAISS (Facebook AI Similarity Search) or Milvus. All standardized feature vectors are imported into the basic vector database, and their correlation is established. The vector data is indexed using a suitable indexing method (such as IVF, HNSW, etc.) to obtain a vector database, ensuring that efficient retrieval performance can still be maintained on large-scale data sets.

[0115] Searching the vector database for the first recognition data using a retrieval algorithm and outputting generation information corresponding to the UI effect image to be recognized further includes:

[0116] Using RAG to search the vector database for the first recognition data based on a preset prompt template to obtain the component format data with the highest matching similarity;

[0117] The structural content data of the UI effect diagram to be identified is obtained, and the component format data and the structural content data are integrated to obtain generation information corresponding to the UI effect diagram to be identified.

[0118] The UI effect diagram to be identified includes the image data to be identified and the structural content data. The first identification data includes at least the component type and the component position. The component type and the position of each component are identified according to the multimodal large model, and the relationship between each component is obtained based on the relationship information between the elements in the vector database, so as to call the component format data. It can be understood that although the description of historical image data is used in this embodiment, it is actually a collection of several UI effect diagrams with corresponding labeled component data. Taking the sketch export file as an example, the attribute information (such as color, size, etc.) of each point, line, and surface is extracted, and compared and integrated with the data identified by the fine-tuning model. The structural content data is pre-stored in the HTML file, and the type, size, and positioning of basic components such as lines, rectangles, and text are identified by parsing HTML. It is converted into built-in basic components such as lines, rectangles, and text through business code, and then the complete page is realized together with the complex charts identified by LLM.

[0119] In this embodiment, the generation information of the UI effect diagram to be identified includes at least component format data and structure content data. Since any UI effect diagram is identified and obtains the corresponding standard component format data and structure content data, the consistency and efficiency of the regeneration of any UI design diagram can be guaranteed.

[0120] In other cases, the generated information for the UI rendering to be identified includes at least the page code corresponding to the component format data and the structure content data. In this case, the code corresponding to each component format data is pre-stored in the vector database, and the code corresponding to the structure content data is also pre-stored in the system. The code for the component format data and the code for the structure content data are integrated to obtain the page code, making it easier to directly deploy the required UI rendering to the page specified by the system, automatically completing the restoration and display of the UI rendering, significantly improving the quality and efficiency of UI rendering generation, while ensuring the consistency and efficiency of the design.

[0121] In this embodiment, a large amount of design drawing data is fine-tuned through multimodal large model technology, effectively improving the generalization ability of the model. At the same time, the process of traditional deep learning solutions is simplified, reducing the dependence on large amounts of training data and complex pre-processing and post-processing steps, reducing development costs and improving development efficiency. A vector database is established, which describes in detail the JSON data structure and attribute descriptions of the system's built-in element components. The vector database enables the model to deeply understand and correctly apply these components, thereby ensuring the consistency and efficiency of the design when generating UI renderings. RAG (Retrieval-Augmented Generation) is adopted, so that the model can use the retrieval-augmented generation method when generating UI renderings, ensuring the consistency of the style and structure of the generated design drawings, and effectively improving the user experience and interface aesthetics.

[0122] As a second embodiment of the present application, performing fine-tuning training based on historical image data and a multimodal pre-trained model further includes:

[0123] Obtain key features of historical image data and fine-tune the hierarchy based on matching of key features of historical image data;

[0124] Use LoRA to perform fine-tuning training on the multimodal pre-trained model based on the fine-tuning level and historical image data.

[0125] Key features include basic visual features, complex structural features, and global semantic features. At this time, the key features of historical image data are obtained, and the fine-tuning level based on the key features of historical image data matching includes:

[0126] Acquire historical image data and identify key features of the historical image data;

[0127] If the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning layer;

[0128] If the key feature is a complex structural feature, the intermediate convolutional layer of the image encoding is used as the fine-tuning layer;

[0129] If the key feature is a global semantic feature, the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning layer.

[0130] Basic visual features include color, texture, and edges. When historical image data contains components with unique appearances, such as distinctive color combinations or texture patterns, these basic visual features stand out and serve as key features. Complex structural features include shape, internal structure, or layout. When historical image data contains complex relationships between components, these complex structural features stand out and serve as key features. Global semantic features include component functions or domain-specific terminology. When historical image data contains component functional descriptions, these global semantic features stand out and serve as key features. The hierarchical structure of image encoding determines the level of abstraction of the features extracted. Low-level layers (initial convolutional layers) learn basic visual features such as edges, texture, and color. Mid-level layers (intermediate convolutional layers) capture more complex shapes and local structures. High-level layers (top convolutional layers or fully connected layers) learn global semantic information and high-level features. Different fine-tuning methods are used for different historical image data, while freezing the remaining layers. This reduces the training computational load while enhancing targetedness, achieving simultaneous improvements in efficiency and accuracy.

[0131] Fine-tuning a multimodal pre-trained model using LoRA based on the fine-tuning hierarchy and historical image data also includes:

[0132] When the intermediate convolutional layer of the image encoding is used as the fine-tuning layer:

[0133] Use LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level;

[0134] Perform image encoding and text encoding on historical image data to obtain visual features and language features;

[0135] Map visual features and language features based on the cross-attention module and calculate the attention weight;

[0136] Obtain fused visual features based on attention weights and gated fusion mechanism;

[0137] Based on the loss function and the fused visual features, the fine-tuning parameters are updated and fine-tuning training is performed.

[0138] In the multimodal projection layer, a cross-attention module is designed to map visual and linguistic features to the same dimension through linear transformation. Attention weights are then calculated to determine the correlation between the visual and linguistic features. The visual and linguistic features are fused using the attention weights and a gating mechanism. The gating mechanism dynamically adjusts the importance of each feature during the fusion process based on the attention weights, thereby generating fused visual features that contain rich cross-modal information. An appropriate loss function is defined to measure the difference between the model output and the true label. The fused visual features are used to calculate gradients through backpropagation and update the parameters of the low-rank matrix after LoRA decomposition. The updated parameters are then used along with the rest of the original model to fine-tune the entire model. During training, the model's performance on the validation set is continuously monitored, and hyperparameters such as the learning rate and regularization parameter are adjusted as needed to optimize performance.

[0139] In this embodiment, the extracted visual features and language features are input into the cross-attention module. The module maps the relationship between them by calculating the dot product or other similarity metrics between the two, and assigns an attention weight related to the language feature to each element in the visual feature according to the mapping result. The calculated attention weight and the gated fusion mechanism are used to fuse the visual features and the language features. The gating mechanism can dynamically adjust the importance of each feature in the fusion process according to the attention weight, so as to show the degree of training of the model for the correlation between language and visual features and the nonlinear relationship between features based on the fused visual features, and use the loss function to judge whether the expected training requirements are met. If so, the fine-tuning training is terminated. If not, the fine-tuning parameters are updated based on the difference between the fused visual features and the actual visual features, and the fine-tuning training is continued.

[0140] In this embodiment, updating the fine-tuning parameters based on the loss function and the fused visual features, and performing fine-tuning training further includes:

[0141] When the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning layer:

[0142] The prefix vector parameters of the text encoding are updated using prefix tuning technology and fine-tuning training is performed.

[0143] Using prefix tuning techniques, only a small number of LLM parameters are fine-tuned to avoid overfitting and reduce computational resource consumption. For each Transformer layer, a prefix key and value vector are initialized and the prefix length is set. In multi-head attention, the expanded key and value are updated based on the initialized prefix key and value vectors and the current layer, while the query vector remains unchanged. Forward propagation is performed based on the attention scores and attention outputs, calculating the cross-entropy loss and updating only the parameters of the prefix vector, keeping most of the pre-trained model parameters unchanged.

[0144] The chart recognition task has its own unique visual features, such as chart type, text and symbols, color, and style. By partially fine-tuning specific layers of the image encoding, the model can focus on learning these features that are closely related to chart recognition. This targeted fine-tuning can significantly improve the model's understanding of chart details, thereby improving recognition accuracy. At the same time, it avoids comprehensive fine-tuning of the entire image encoding, reducing computing resource consumption and training time, and achieving efficient fine-tuning. Furthermore, by achieving a deep fusion of visual and language information at the projection layer, the model can simultaneously understand and process the visual features and textual information in the chart. Through cross-modal attention and gating mechanisms, the model can capture the complex connections between vision and language, thereby generating more accurate and rich chart descriptions, improving the model's understanding of charts, and expanding the model's application range, enabling the model to handle a wider variety of charts and more complex textual descriptions, thereby improving the model's generalization and practicality. Furthermore, by fine-tuning a small number of prefix parameters to adapt to downstream tasks, the model can quickly adapt to different graph recognition tasks without large-scale adjustments to the entire model. This avoids model performance degradation or overfitting problems caused by excessive fine-tuning, reduces the consumption of computing resources, and makes model fine-tuning more efficient.

[0145] In other embodiments, any fine-tuning method can also be combined, such as through the collaboration of image encoding fine-tuning and cross-modal fusion, the model can more accurately extract the visual features of the chart and at the same time the visual features can be combined with text information to generate a more accurate and rich chart description. Through the collaboration of cross-modal fusion and text encoding fine-tuning methods, the model can generate more accurate and fluent text descriptions while ensuring the stability and efficiency of the model during the fine-tuning process. By combining partial fine-tuning of image encoding, cross-modal fusion of enhanced projection layers, and the use of parameter-efficient text encoding fine-tuning methods, the model training achieves a balance between efficiency and accuracy, which can not only quickly adapt to different chart recognition tasks, but also generate accurate and rich chart descriptions, meeting the needs of practical applications.

[0146] As a third embodiment of the present application, a UI effect diagram recognition system based on a multimodal large model includes at least:

[0147] Question and answer annotation module, which obtains historical image data based on user annotations and questions and answers;

[0148] The fine-tuning training module performs fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model;

[0149] Vector database, pre-stored built-in component data structure;

[0150] The recognition generation module obtains the UI effect diagram to be identified, outputs first recognition data based on the multimodal large model and the UI effect diagram to be identified, uses a retrieval algorithm to search the vector database for the first recognition data, and outputs generation information corresponding to the UI effect diagram to be identified.

[0151] In this embodiment, the question-answer annotation module is connected to the fine-tuning training module, and the fine-tuning training module and the vector database are connected to the recognition generation module. The recognition generation module includes at least an interaction unit for obtaining the UI effect image to be recognized.

[0152] The fine-tuning training module includes at least a visual encoder corresponding to image encoding, a multimodal projector corresponding to multimodal projection, and a language model corresponding to text encoding. In this embodiment, the language model is an LLM model. By partially fine-tuning the visual encoder, a specific layer of the visual encoder is fine-tuned to capture task-related visual features, and an advanced cross-modal mechanism is introduced to enable a deeper fusion of visual and language information at the projection layer, thereby improving the model's understanding and generation capabilities. Using prefix tuning technology, only a small number of LLM parameters are fine-tuned to avoid overfitting and reduce the consumption of computing resources. While improving the efficiency of model fine-tuning, the accuracy of model fine-tuning is improved, thereby improving the accuracy of subsequent models for UI renderings.

[0153] As a fourth embodiment of the present application, a computer-readable storage medium is used to store a computer program or instruction. When the computer program or instruction is executed by a processing device, the above-mentioned multimodal large model-based UI effect diagram recognition method is implemented. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0154] The specific implementation methods described above are the preferred implementation methods of the UI effect diagram recognition method, system and medium based on the multimodal large model of this application, and are not intended to limit the specific implementation scope of this application. The scope of this application includes but is not limited to this specific implementation method. All equivalent changes made in accordance with the shape and structure of this application are within the scope of protection of this application.

Claims

1. A UI effect diagram recognition method based on a multimodal large model, characterized by: The steps include: Build a multimodal pre-training model based on image encoding, multimodal projection, and text encoding; Perform fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model; Build vector database based on built-in component data structure; Obtain a UI effect diagram to be identified, and output first recognition data based on the multimodal large model and the UI effect diagram to be identified; Searching a vector database for the first recognition data using a retrieval algorithm, and outputting generation information corresponding to the UI effect diagram to be recognized; The performing fine-tuning training based on the historical image data and the multimodal pre-training model further includes: Acquire historical image data and identify key features of the historical image data; If the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a complex structural feature, the intermediate convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a global semantic feature, the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning layer; When the intermediate convolutional layer of the image encoding is used as the fine-tuning layer: Use LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level; Perform image encoding and text encoding on historical image data to obtain visual features and language features; Map visual features and language features based on the cross-attention module and calculate the attention weight; Obtain fused visual features based on attention weights and gated fusion mechanism; Update fine-tuning parameters based on the loss function and fused visual features, and perform fine-tuning training; When the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning layer: The prefix vector parameters of the text encoding are updated using prefix tuning technology and fine-tuning training is performed.

2. The method for recognizing UI effect diagrams based on a multimodal large model according to claim 1, wherein: The performing of fine-tuning training based on historical image data and the multimodal pre-trained model to obtain a multimodal large model includes: Acquire historical image data and perform data preprocessing on the historical image data; Use LoRA to adjust the weight matrix of the multimodal pre-trained model; The historical image data after data preprocessing is input into the adjusted multimodal pre-training model, and fine-tuning training is performed to obtain a large multimodal model.

3. The UI effect diagram recognition method based on a multimodal large model according to claim 1, characterized in that: The step of searching the vector database for the first recognition data using a search algorithm and outputting generation information corresponding to the UI effect diagram to be recognized further includes: Using RAG to search the vector database for the first recognition data based on a preset prompt template to obtain the component data with the highest matching similarity; The structural content data of the UI effect diagram to be identified is obtained, and the component data and the structural content data are integrated to obtain generation information corresponding to the UI effect diagram to be identified.

4. The UI effect diagram recognition method based on a multimodal large model according to claim 1, characterized in that: The performing fine-tuning training based on the historical image data and the multimodal pre-trained model to obtain the multimodal large model further includes: Construct a triplet containing a diagram, its corresponding text description, and a related question; Comparing the candidate models with the output results of the multimodal large model corresponding to the triplets based on the evaluation model, and outputting the evaluation results of the multimodal large model; Determine whether the multimodal large model requires iterative training based on the evaluation results.

5. The UI effect diagram recognition method based on a multimodal large model according to claim 1, characterized in that: The constructing of a vector database based on the built-in component data structure includes: Obtain the relationship between graph elements to build a knowledge graph; Obtain the built-in component data structure and build a vector database based on the knowledge graph and the built-in component data structure.

6. The UI effect diagram recognition method based on a multimodal large model according to claim 5, characterized in that: The constructing of a vector database based on the knowledge graph and the built-in component data structure further includes: Convert built-in component data structures into JSON object data; Extract key attributes from each JSON object data and convert them into feature vectors; Normalize the feature vector; The normalized feature vectors are correlated according to the knowledge graph to obtain a vector database.

7. The UI effect diagram recognition method based on a multimodal large model according to claim 1, characterized in that: The method of using LoRA to perform fine-tuning training on the multimodal pre-trained model according to the fine-tuning level and historical image data further includes: Use LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level; Perform image encoding and text encoding on historical image data to obtain visual features and language features; Map visual features and language features based on the cross-attention module and calculate the attention weight; Obtain fused visual features based on attention weights and gated fusion mechanism; Based on the loss function and the fused visual features, the fine-tuning parameters are updated and fine-tuning training is performed.

8. The UI effect diagram recognition method based on a multimodal large model according to claim 2, characterized in that: The acquiring of historical image data and the preprocessing of the historical image data include: Performing image normalization and size adjustment on the image data in the historical image data; Perform information parsing and format conversion on the structural content data in the historical image data.

9. The UI effect diagram recognition method based on a multimodal large model according to claim 8, characterized in that: The resizing of the image data in the historical image data includes: Obtain the resolution of the multimodal pre-trained model, and obtain an adjustment value based on the resolution of the multimodal pre-trained model and the original resolution of the image data; When the adjustment value is negative, the nearest neighbor interpolation is called to perform size adjustment on the image data; When the adjustment value is positive, the image volume is calculated and the corresponding order interpolation method is called according to the image volume gradient.

10. A UI effect diagram recognition system based on a multimodal large model, for implementing the UI effect diagram recognition method based on a multimodal large model according to any one of claims 1 to 9, characterized in that: include: Question and answer annotation module, which obtains historical image data based on user annotations and questions and answers; The fine-tuning training module performs fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model; Vector database, pre-stored built-in component data structure; The recognition generation module obtains the UI effect diagram to be identified, outputs first recognition data based on the multimodal large model and the UI effect diagram to be identified, uses a retrieval algorithm to search the vector database for the first recognition data, and outputs generation information corresponding to the UI effect diagram to be identified.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or instruction. When the computer program or instruction is executed by a processing device, the UI effect diagram recognition method based on a multimodal large model as described in any one of claims 1 to 9 is implemented.

Images