UI effect picture recognition method and system based on multi-modal large model and medium

Abstract

The invention discloses a UI effect picture recognition method and system based on a multi-modal large model, and a medium. The method comprises the following steps: constructing a multi-modal pre-training model based on image coding, multi-modal projection and text coding; performing fine tuning training according to the historical image data and the multi-modal pre-training model to obtain a multi-modal large model; constructing a vector database based on the built-in component data structure; obtaining a to-be-recognized UI effect picture, and outputting first recognition data based on the multi-modal large model and the to-be-recognized UI effect picture; and searching a vector database for the first identification data by using a retrieval algorithm, and outputting generation information corresponding to the UI effect picture to be identified. The method has the beneficial effect that the generation consistency and efficiency of the UI effect picture are ensured.

Description

Technical Field

[0001] The present application relates to the field of image recognition generation technology, and in particular to a method, system and medium for recognizing UI effect diagrams based on a multimodal large model. Background Art

[0002] The complexity of the UI rendering recognition and generation process mainly comes from the diversity of design elements, the complexity of layout and rendering, and the complexity of interaction. These complexities increase the difficulty of recognition and generation, making the process cumbersome.

[0003] In related technologies, model training is usually used to identify and generate UI renderings. However, insufficient model generalization is a common challenge in the field of machine learning and deep learning. For UI rendering recognition and generation tasks, insufficient model generalization may lead to poor performance of the model under new data or new scenarios.

[0004] At the same time, due to the large number of components and complex functions, developers often find it difficult to fully understand and apply these components, resulting in inconsistent UI renderings and reduced user experience.

[0005] Patent "Method for generating web code from UI based on generative adversarial and convolutional neural network", publication number: CN110377282A, publication date: October 25, 2019, specifically discloses: constructing a mapping relationship between the display effect of HTML elements and their source code; identifying the location of HTML elements in image I; generating I complete HTML code; obtaining the similarity Sim1 between the manually written HTML code and the generated HTML code; obtaining the similarity Sim2 between image I and image I1 generated by the generated HTML code; balancing Sim1 and Sim2 to Sim0, judging the relationship between Sim0 and threshold t, if Sim0 is less than t, repeating the above process, otherwise, executing the next step; after the previous step of training, obtaining the generation model M from image to HTML code, and inputting the UI image to be processed into the model M to obtain the corresponding HTML code. This solution relies on a multi-step deep learning model, which not only requires a large amount of training data, but also involves complex pre-processing and post-processing steps, increasing the development cost, and it is difficult to ensure the consistency of the generation effect. Secondly, existing deep learning models are easily limited by training data when processing UI renderings, resulting in poor generalization ability of the model and difficulty in maintaining stable performance in different application scenarios.

[0006] Patent "A UI Generation Method Based on Multimodal AI", publication number: CN115756455A, publication date: March 7, 2023, specifically discloses obtaining the demand document of the official website of the target website, and according to the demand document, refining the demand into functional sections by comparing the different functions of the specific UI interface, and extracting different dimensional features for different functional sections to form feature sections; performing fusion operations on the feature sets provided by different functional sections to establish the connection between the features to form a comprehensive coverage of the demand, and encapsulating the fused feature set results into the feature vector required by the neural network prediction model; entering the feature vector into the UI automatic generation tool driven by multimodal AI technology to perform the UI image generation function, predicting and generating UI images corresponding to the functional requirements; reviewing the generated UI images with different functional requirements, and performing local optimization with reference to the input features for secondary generation of the deficiencies, and finally delivering the generated different UIs for use. In the process of generating UI renderings, different designers or developers use different components and styles. This solution cannot ensure the consistency of the generated UI design, which affects the user experience and interface aesthetics. Summary of the invention

[0007] In order to solve the problem in the prior art that efficiency and consistency cannot be taken into account in UI image recognition and generation, the present application provides a UI effect image recognition method, system and medium based on a multimodal large model. The multimodal large model is obtained by constructing a multimodal pre-training model based on image coding, multimodal projection and text coding, and fine-tuning the training with historical image data. While realizing comprehensive recognition of charts, components and text content in UI effect images, the recognition accuracy and efficiency are improved. The retrieval algorithm is used to search a vector database constructed based on a built-in component data structure for the first recognition data, so as to quickly find the component vector matching the first recognition data, and obtain the generation information of the corresponding UI effect image to be recognized based on these vectors, so as to ensure the consistency and efficiency of UI effect image generation.

[0008] To achieve the above-mentioned technical objectives, a technical solution provided by the present application is a method for recognizing UI effect diagrams based on a multimodal large model, comprising the following steps: constructing a multimodal pre-trained model based on image encoding, multimodal projection and text encoding; performing fine-tuning training according to historical image data and the multimodal pre-trained model to obtain a multimodal large model; constructing a vector database based on a built-in component data structure; obtaining a UI effect diagram to be recognized, and outputting first recognition data based on the multimodal large model and the UI effect diagram to be recognized; using a retrieval algorithm to search the vector database for the first recognition data, and outputting generation information corresponding to the UI effect diagram to be recognized.

[0009] Furthermore, the fine-tuning training is performed based on the historical image data and the multimodal pre-trained model to obtain the multimodal large model, including: obtaining historical image data and performing data preprocessing on the historical image data; using LoRA to adjust the weight matrix of the multimodal pre-trained model; inputting the historical image data after data preprocessing into the adjusted multimodal pre-trained model, performing fine-tuning training, and obtaining the multimodal large model.

[0010] Furthermore, the use of a retrieval algorithm to search the vector database for the first recognition data and outputting generation information corresponding to the UI effect diagram to be identified also includes: using RAG to search the vector database for the first recognition data based on a preset prompt template to obtain component data with the highest matching similarity; obtaining structural content data of the UI effect diagram to be identified, integrating the component data and the structural content data to obtain generation information corresponding to the UI effect diagram to be identified.

[0011] Furthermore, the fine-tuning training is performed based on the historical image data and the multimodal pre-trained model to obtain the multimodal large model, which also includes: constructing a triplet including a chart, a corresponding text description, and related questions; comparing the candidate models and the output results of the multimodal large model corresponding to the triplet based on the evaluation model, and outputting the evaluation results of the multimodal large model; and judging whether the multimodal large model needs iterative training based on the evaluation results.

[0012] Furthermore, the constructing of a vector database based on a built-in component data structure includes: acquiring relationships between graph elements to construct a knowledge graph; acquiring a built-in component data structure, and constructing a vector database based on the knowledge graph and the built-in component data structure.

[0013] Furthermore, the construction of a vector database based on the knowledge graph and the built-in component data structure also includes: converting the built-in component data structure into JSON object data; extracting key attributes from each JSON object data and converting them into feature vectors; standardizing the feature vectors; and constructing associations for the standardized feature vectors based on the knowledge graph to obtain a vector database.

[0014] Furthermore, the fine-tuning training performed according to the historical image data and the multimodal pre-trained model also includes: obtaining key features of the historical image data, and matching the fine-tuning levels based on the key features of the historical image data; and using LoRA to perform fine-tuning training on the multimodal pre-trained model according to the fine-tuning levels and the historical image data.

[0015] Furthermore, the key features of the historical image data are obtained, and the fine-tuning level is matched based on the key features of the historical image data, including: obtaining the historical image data, and identifying the key features of the historical image data; if the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning level; if the key features are complex structural features, the intermediate convolutional layer of the image encoding is used as the fine-tuning level; if the key features are global semantic features, the top convolutional layer or the fully connected layer of the image encoding is used as the fine-tuning level.

[0016] Furthermore, the use of LoRA to perform fine-tuning training on the multimodal pre-trained model according to the fine-tuning level and historical image data also includes: using LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level; performing image encoding and text encoding on the historical image data to obtain visual features and language features; mapping visual features and language features based on a cross-attention module, and calculating attention weights; obtaining fused visual features based on attention weights and a gated fusion mechanism; updating fine-tuning parameters based on the loss function and the fused visual features, and performing fine-tuning training.

[0017] Furthermore, the obtaining of historical image data and data preprocessing of the historical image data include: performing image normalization and size adjustment on the picture data in the historical image data; and performing information parsing and format conversion on the structural content data in the historical image data.

[0018] Furthermore, the resizing of the picture data in the historical image data includes: obtaining the resolution of the multimodal pre-trained model, and obtaining an adjustment value according to the resolution of the multimodal pre-trained model and the original resolution of the picture data; when the adjustment value is a negative value, calling the nearest neighbor interpolation to perform resizing on the picture data; when the adjustment value is a positive value, calculating the picture volume, and calling the corresponding order interpolation method according to the picture volume gradient.

[0019] Another technical solution provided by the present application is a UI effect diagram recognition system based on a multimodal large model, which is used to implement the UI effect diagram recognition method based on the multimodal large model as mentioned above, including: a question and answer annotation module, which obtains historical image data based on user annotations and questions and answers; a fine-tuning training module, which performs fine-tuning training based on historical image data and a multimodal pre-trained model to obtain a multimodal large model; a vector database, which pre-stores a built-in component data structure; an identification generation module, which obtains a UI effect diagram to be identified, outputs first identification data based on the multimodal large model and the UI effect diagram to be identified, uses a retrieval algorithm to search the vector database for the first identification data, and outputs generation information corresponding to the UI effect diagram to be identified.

[0020] Another technical solution provided by the present application is a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processing device, the UI effect diagram recognition method based on the multimodal large model as described above is implemented.

[0021] The beneficial effects of this application are: 1. With the continuous increase of historical design drawing data and the enrichment of component libraries, the model can be continuously fine-tuned to improve the accuracy and efficiency of recognition. At the same time, the vector database can be adaptively adjusted without changing the trained multimodal big data to adapt to different application scenarios and needs, which is more practical.

[0022] 2. For different historical image data, different fine-tuning forms are selected and the remaining layers are frozen at the same time, which reduces the amount of training calculations while being more targeted, thereby achieving simultaneous improvements in efficiency and accuracy.

[0023] 3. By partially fine-tuning specific layers of the image encoding, the model can be more focused on learning these features that are closely related to chart recognition, improving the model's ability to understand chart details, thereby improving recognition accuracy. At the same time, it avoids comprehensive fine-tuning of the entire image encoding, reduces the consumption of computing resources and training time, and achieves efficient fine-tuning.

[0024] 4. By realizing the deep fusion of visual and language information in the projection layer, the model can simultaneously understand and process the visual features and text information in the chart. Through cross-modal attention and gating mechanisms, the model can capture the complex relationship between vision and language, thereby generating more accurate and rich chart descriptions, improving the model's ability to understand charts, and expanding the scope of application of the model, so that the model can process more types of charts and more complex text descriptions, improving the generalization ability and practicality of the model.

[0025] 5. By fine-tuning a small number of prefix parameters to adapt to downstream tasks, the model can quickly adapt to different graph recognition tasks without large-scale adjustments to the entire model, avoiding model performance degradation or overfitting problems caused by excessive fine-tuning, and reducing the consumption of computing resources, making model fine-tuning more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the UI effect diagram recognition method based on a multimodal large model in this application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in conjunction with the drawings and examples. It should be understood that the specific implementation method described here is only an optimal embodiment of the present application, which is only used to explain the present application and does not limit the scope of protection of the present application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0028] like Figure 1 As shown, as the first embodiment of the present application, the UI effect diagram recognition method based on the multimodal large model includes the following steps: Construct a multimodal pre-training model based on image encoding, multimodal projection, and text encoding; Perform fine-tuning training based on historical image data and multimodal pre-trained models to obtain a multimodal large model; Build a vector database based on built-in component data structures; Obtain a UI effect diagram to be identified, and output first identification data based on the multimodal large model and the UI effect diagram to be identified; The vector database is searched for the first recognition data using a retrieval algorithm, and generation information corresponding to the UI effect diagram to be recognized is output.

[0029] In this embodiment, a multimodal large model is obtained by fine-tuning the multimodal pre-training model based on image coding, multimodal projection and text coding, and fine-tuning the training with historical image data. While realizing comprehensive recognition of charts, components and text content in UI renderings, the recognition accuracy and efficiency are improved. The retrieval algorithm is used to search the vector database constructed based on the built-in component data structure for the first recognition data, and the component vector matching the first recognition data is quickly found. Based on these vectors, the generation information of the corresponding UI rendering to be identified is obtained to ensure the consistency and efficiency of UI rendering generation. With the continuous increase of historical design drawing data and the enrichment of component libraries, the model is continuously fine-tuned to improve the recognition accuracy and efficiency. At the same time, the vector database is adaptively adjusted to adapt to different application scenarios and needs without changing the trained multimodal big data, which is more practical.

[0030] A multimodal pre-training model is constructed based on image coding, multimodal projection, and text coding. LLaVA-3 can be selected as a multimodal pre-training model. By combining image coding and text coding, the model can simultaneously process and understand information from different modalities, thereby adapting to the situation where the UI rendering contains both image information and text information, and mapping data of different modalities to the same semantic space based on multimodal projection, so that the model can directly compare and fuse information of different modalities in the semantic space, output recognition results of UI renderings, and improve recognition efficiency and accuracy.

[0031] Fine-tuning training is performed based on historical image data and multimodal pre-trained models to obtain a large multimodal model including: Acquire historical image data, and perform data preprocessing on the historical image data; Use LoRA to adjust the weight matrix of the multimodal pre-trained model; The historical image data after data preprocessing is input into the adjusted multimodal pre-trained model, and fine-tuning training is performed to obtain a multimodal large model.

[0032] LoRA (Low-Rank Adaptation) adds a low-rank matrix to approximate the weight update of the model instead of directly updating the original high-dimensional weight matrix, thereby achieving rapid adjustment and adaptation of the model without significantly increasing the computing and storage costs, and improving the recognition accuracy and efficiency of UI effect diagrams.

[0033] Acquiring historical image data and performing data preprocessing on the historical image data include: Performing image normalization and size adjustment on the picture data in the historical image data; Perform information parsing and format conversion on the structural content data in the historical image data.

[0034] The historical image data at least includes image data and structural content data including component locations and attribute information. In order to ensure that the details of the image can be clearly obtained during the training process, the pixel values ​​of the images in the historical image data are normalized and resized, thereby improving the accuracy of subsequent training and assisting the model to converge quickly, and simultaneously improving the efficiency of model training.

[0035] The step of resizing the image data in the historical image data includes: Obtain the resolution of the multimodal pre-trained model, and obtain an adjustment value according to the resolution of the multimodal pre-trained model and the original resolution of the image data; Resizing is performed on the image data based on the scaling algorithm and the adjustment value.

[0036] Normalization is performed based on the pixel values ​​of the image, and the pixel values ​​are linearly mapped from the original range to the range of [0, 1] or [-1, 1] to meet the model input requirements. The image data is resized by scaling the image data based on the resolution of the image encoding of the multimodal pre-trained model and the difference in the original resolution of the image data to maintain the clarity and details of the image. For example, when the original resolution of the image is greater than the resolution of the multimodal pre-trained model, the image is reduced using a scaling algorithm, and when the original resolution of the image is less than the resolution of the multimodal pre-trained model, the image is enlarged using a scaling algorithm.

[0037] In this embodiment, resizing the image data based on the scaling algorithm and the adjustment value further includes: The corresponding scaling algorithm is called based on the adjustment value to perform size adjustment on the image data.

[0038] The scaling algorithm includes at least nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. Other high-order interpolation methods may also be included in the remaining embodiments. For example, when the original resolution of the image is greater than the resolution of the multimodal pre-trained model, the adjustment value is a negative value, and when the original resolution of the image is less than the resolution of the multimodal pre-trained model, the adjustment value is a positive value. At this time, when the adjustment value is a negative value, the nearest neighbor interpolation is called to perform size adjustment on the image data, and when the adjustment value is a positive value, the bilinear interpolation or bicubic interpolation is called to perform size adjustment on the image data. The nearest neighbor interpolation directly selects the source pixel value closest to the target pixel, and the calculation speed is fast, but it is easy to cause the edge of the image to be jagged when the image is enlarged. Therefore, when the image is reduced, the nearest neighbor interpolation is used to achieve fast scaling and reduce the amount of calculation. Bilinear interpolation obtains the value of the target pixel by calculating the weighted average of the four source pixels around the target pixel, and bicubic interpolation considers the influence of more surrounding pixels to generate a smoother and more detailed picture. It has a large amount of calculation, but has a good scaling effect and is suitable for compensating pixels when enlarging a picture. Therefore, when enlarging a picture, bilinear interpolation or bicubic interpolation is used to achieve precise scaling. In other embodiments, resizing the picture data based on the scaling algorithm and the adjustment value also includes: When the adjustment value is negative, the nearest neighbor interpolation is called to perform size adjustment on the image data; When the adjustment value is positive, the image volume is calculated and the corresponding order interpolation method is called according to the image volume gradient.

[0039] Different gradient thresholds are set according to the computing capacity of the system. For example, when the amount of pictures is less than the first gradient threshold, bicubic interpolation is called to resize the picture data. When the amount of pictures is greater than the first gradient threshold, bilinear interpolation is called to resize the picture data. Therefore, when the amount of pictures is small, a higher-order interpolation method is used for scaling to improve the picture processing accuracy. When the amount of pictures is large, a lower-order interpolation method is used for scaling to improve the picture processing efficiency. At this time, when the amount of pictures is large, there is more training data. Even if the scaling accuracy of the lower-order interpolation method is not as good as that of the higher-order interpolation method, the mutual compensation of a large amount of data can also ensure the accuracy of the final model construction. When the amount of pictures is small, the picture scaling accuracy is improved by a higher-order interpolation method, so that more details can be extracted from each picture during the model training process, compensating for the problem of small data volume, and ensuring the accuracy of the final model.

[0040] Structural content data includes component location and attribute information. Usually, the structural content data of an image is stored in an HTML file. The HTML file in the historical image data is parsed to extract the location information (x, y coordinates) and attribute information of each component. The attribute information includes at least size information (width, height). The location information and attribute information are converted into JSON format to facilitate model processing and improve the readability and scalability of the structural content data.

[0041] In this embodiment, the multimodal pre-trained model is composed of a CLIP image encoder, a multimodal projector, and a Vicuna text encoder. Using LoRA to adjust the weight matrix of the multimodal pre-trained model includes: Introducing low-rank matrices into the weight matrices of multimodal pre-trained models.

[0042] The historical image data after data preprocessing is input into the adjusted multimodal pre-trained model, and fine-tuning training is performed to obtain a multimodal large model including: The historical image data after data preprocessing is input into the adjusted multimodal pre-trained model, forward propagation and back propagation are performed, the low-rank matrix of the multimodal pre-trained model is updated, and a multimodal large model is obtained.

[0043] A low-rank matrix is ​​introduced into the weight matrix of the multimodal pre-training model to reduce the dimension of the model parameters. The introduction of the low-rank matrix is ​​expressed as: ; in, To introduce the weight matrix of the low-rank matrix, is the original weight matrix, is a low-rank matrix, and is a matrix of lower dimension.

[0044] Through back propagation, the low-rank matrix in the model parameters is updated so that it can better adapt to the new task. The loss function of back propagation is: ; Among them, MSE stands for mean square error, which is used for location loss, CrossEntropy stands for cross entropy loss, which is used for classification loss, is the weight coefficient, N is the number of samples, is the actual observed value of the ith sample, is the model prediction value of the i-th sample, is the actual probability distribution of the ith sample, is the predicted probability distribution of the ith sample.

[0045] The loss function includes at least position loss and classification loss. Position loss is the positioning error, which measures the difference between the icon or component position predicted by the model and the actual position. Classification loss is the type error, which measures the difference between the chart or component type predicted by the model and the actual type. During fine-tuning, the model considers both position loss and classification loss, and updates the model parameters through the back-propagation algorithm to minimize the weighted sum of the two losses, thereby improving the recognition accuracy of multimodal large models.

[0046] LoRA's low-rank decomposition technology is used to significantly reduce the computational cost and memory usage of fine-tuning, while ensuring the model's efficient performance in chart recognition tasks. This allows the model to take advantage of existing large pre-trained models and quickly adapt to new tasks through a small amount of parameter updates. Especially in chart recognition scenarios, it takes into account both the specialized requirements of the task and the efficient use of resources, providing strong technical support for the development of intelligent, multimodal chart recognition assistants.

[0047] In this embodiment, the rank of the low-rank matrix introduced by LoRA is 128, and the scaling factor is 256. The scaling factor is used to control the weight influence during fine-tuning training. The CLIP model (vit-large-patch14-336 version) is used as the image encoding structure to process the input image data and extract the image visual features. The penultimate layer of the CLIP model is selected as the output layer to generate the image visual features. The two-layer MLP plus GELU activation function is used as the multimodal projection structure type, and the learning rate for multimodal projection is set to convert the image visual features into a language model. The Vicuna-13B v1.5 model is used as the text encoding structure. The training cycle is set to 1 epoch, the training batch size of each device is 16, the evaluation batch size of each device is 4, and 1 step of gradient is accumulated before each back propagation. The model is saved every 50,000 steps. The upper limit of the number of models saved is 1. If it exceeds, the old checkpoint is overwritten. The global learning rate is set to 0.0002, the proportion of the warm-up phase is 3%, and the learning rate is gradually increased. Use the cosine annealing learning rate scheduler, enable gradient checkpoints, reduce video memory usage, use bfloat16 precision for training to reduce video memory requirements, accelerate floating point calculations on hardware that supports TF32, the maximum input length of the model is 2048 tokens, and use 4 parallel threads when loading data. With this setting, fine-tune the multimodal pre-trained model to obtain a large multimodal model.

[0048] After fine-tuning the multimodal large model, the validation set is used to evaluate the model performance to ensure that the model can accurately identify and locate elements in the UI renderings. After preprocessing the historical image data, the historical image data is divided into a training set and a validation set according to a preset ratio. The training set is used to fine-tune the multimodal pre-trained model, and the validation set is used to validate and evaluate the multimodal large model. The validation evaluation includes classification evaluation and location evaluation. The cross-entropy loss is used for classification evaluation. The input data is forward propagated through the model to obtain the predicted probability distribution of each category. The true label is converted into a one-hot encoding format (One-Hot Encoding), and the cross-entropy loss is calculated using the cross-entropy formula: ; in, is the true label, is the predicted probability output by the model.

[0049] Position loss is used for position evaluation. Position loss is used to evaluate the performance of the task in the chart positioning prediction position. The model outputs the bounding box coordinates of each target (center point coordinates, width and height), obtains the bounding box coordinates of the real target, and uses smooth L1 loss or MSE to calculate the difference between the predicted bounding box and the real bounding box, that is, the position loss: ; in, are the true bounding box coordinates, are the predicted bounding box coordinates.

[0050] The classification loss and position loss are weightedly combined to calculate the total loss of the model, and the recognition accuracy of the current multimodal large model is judged based on the total loss.

[0051] In other cases, quantitative metrics are used to evaluate the image recognition capabilities of the multimodal large model. High-level language models are used to measure the quality of the generated responses. Fine-tuning training is performed based on historical image data and multimodal pre-trained models. Obtaining a multimodal large model also includes: Construct triplets containing a diagram, its corresponding text description, and related questions; Compare the candidate models with the output results of the multimodal large model corresponding to the triplets based on the evaluation model, and output the evaluation results of the multimodal large model; Determine whether the multimodal large model requires iterative training based on the evaluation results.

[0052] Specifically, a triplet containing a chart, its corresponding text description, and a related question is constructed. The candidate model (e.g., LLM) generates an answer based on the question and chart content, thereby creating a reference prediction based on the question and text description, which is generated using a text generation model. After obtaining the answers from the candidate model and the multimodal large model, we provide the question, chart information (in the form of text description), and the answers generated by the two models to the evaluator (e.g., text generation model). The evaluator will score the generated answers based on the relevance, accuracy, clarity, and detail of the answers, using a scoring system of 1 to 10, where a higher score indicates better overall performance. As shown in Table 1, this is an evaluation score sheet for one case. According to the evaluation score sheet, the recognition performance of the multimodal large model is obtained, so that the parts with poor recognition performance can be adjusted in a targeted manner.

[0053] Table 1 Evaluation score sheet ; According to the verification results, adjust the training parameters and training set, and perform multiple iterations of training and verification until the multimodal large model reaches the expected performance indicators. The expected performance indicator can be a preset total loss threshold. When the total loss of the multimodal large model is less than the preset total loss threshold, the multimodal large model is considered to have reached the expected performance indicators. The learning rate, batch size and other training parameters can be adjusted according to the verification results to make the output value of the multimodal large model closer to the actual value. The training data can also be expanded through data enhancement techniques (such as rotation, scaling, cropping, etc.) to improve the generalization ability of the model. Re-enter the model with the adjusted parameters and enhanced data for iterative training. After each iteration of training, use the validation set for evaluation until the model performance reaches the expected target.

[0054] Building a vector database based on built-in component data structures includes: Obtain the relationship between diagram elements to build a knowledge graph; Obtain the built-in component data structure and build a vector database based on the knowledge graph and the built-in component data structure.

[0055] In the UI rendering, there are mutual logical relationships between chart elements. The elements of the chart and the relationships between them are represented in a structured manner according to the knowledge graph. Through nodes and edges, the knowledge graph can reflect the layout of the chart, the logical relationships between elements, and the overall semantic information.

[0056] As in this embodiment, the nodes of the knowledge graph represent elements: including the title, axis, data series, data points, labels, etc. of the chart, and each node contains the attributes of the element (type, coordinates, text content, etc.). The edges of the knowledge graph represent relationships: the relationship between elements, such as "axis-data series", "data series-data point", "data point-label", etc. The knowledge graph is converted into a fixed-length vector V_input that can capture global and local features through the graph convolutional network (GCN) embedding model, capturing the global and local features of the chart, thereby learning the representation of the chart structure and the relationship between elements. For example: node: -N1: Title ("Sales Trends") -N2: X axis ("Month") - N3: Y axis ("Sales") -N4: Data Series ("Product A") -N5: Data point 1 ("value, coordinate") -N6: Data point 2 ("value, coordinate") side: -N1 and N4: Title-related data series -N2 and N5 / N6: X-axis related data points -N3 and N5 / N6: Y-axis related data points -N4 and N5 / N6: The data series contains data points.

[0057] The built-in component data structure includes UI chart components and their data structure. UI chart components include at least Spc chart, dashboard, line chart, scatter chart, curve chart, bar chart, bubble chart, funnel chart, donut chart, Gantt chart, histogram, spider chart, progress bar, area chart, and pie chart. Add detailed property descriptions for each component, including component type, component ID, display name, label, image name, position (x, y coordinates), width and height, etc. Organize the built-in component data structure into JSON format and perform vectorization processing.

[0058] Building a vector database based on the knowledge graph and built-in component data structures also includes: Convert built-in component data structures into JSON object data; Extract key attributes from each JSON object data and convert them into feature vectors; Normalize the feature vector; The standardized feature vectors are relevance constructed according to the knowledge graph to obtain a vector database.

[0059] Extract key attributes (e.g. component type, location, size, etc.) from each JSON object and convert them into feature vectors. For example, vectorize component location information: ; in, is the feature vector of component position information, x is the component x coordinate, y is the component y coordinate, is the component width, Is the component height.

[0060] The feature vectors are normalized to ensure that all features are on the same scale. In this embodiment, the normalization is achieved by subtracting the mean and dividing by the standard deviation: ; in, is the normalized feature vector, is the mean, is the standard deviation.

[0061] The standardized feature vectors are stored in the vector database. Each vector and its corresponding JSON object are associated and stored based on the knowledge graph to facilitate subsequent retrieval and call.

[0062] In this embodiment, a basic vector database is constructed using database technologies for efficient vector retrieval such as FAISS (Facebook AI Similarity Search) or Milvus, all standardized feature vectors are imported into the basic vector database, and the correlation between them is established. The vector data is indexed using a suitable indexing method (such as IVF, HNSW, etc.) to obtain a vector database, thereby ensuring efficient retrieval performance on large-scale data sets.

[0063] Using a search algorithm to search a vector database for the first recognition data, and outputting generation information corresponding to the UI effect diagram to be recognized also includes: Using RAG to search the vector database for the first recognition data based on a preset prompt template, and obtaining the component format data with the highest matching similarity; The structural content data of the UI effect diagram to be identified is obtained, and the component format data and the structural content data are integrated to obtain the generation information of the corresponding UI effect diagram to be identified.

[0064] The UI effect diagram to be identified includes the image data to be identified and the structural content data. The first identification data includes at least the component type and the component position. The component type and the position of each component are identified according to the multimodal large model, and the relationship between each component is obtained based on the relationship information between the elements in the vector database, so as to call the component format data. It can be understood that although the description of historical image data is used in this embodiment, it is actually a set of several UI effect diagrams corresponding to the annotated component data. Taking the sketch export file as an example, the attribute information (such as color, size, etc.) of each point, line, and surface is extracted, and compared and integrated with the data identified by the fine-tuning model. The structural content data is pre-stored in the HTML file, and the type, size, and positioning of basic components such as lines, rectangles, and texts are identified by parsing HTML. It is converted into built-in basic components such as lines, rectangles, and texts through business code, and then the complex charts identified by LLM are used to realize the complete page.

[0065] In this embodiment, the generation information of the UI effect diagram to be identified includes at least component format data and structure content data. Since any UI effect diagram obtains corresponding standard component format data and structure content data after identification, the consistency and efficiency of regenerating any UI design diagram can be guaranteed.

[0066] In other cases, the generation information of the UI effect diagram to be identified includes at least the page code corresponding to the component format data and the structure content data. At this time, the code corresponding to each component format data is pre-stored in the vector database, and the code corresponding to the structure content data is pre-stored in the system. The code of the component format data and the code of the structure content data are integrated to obtain the page code, so as to facilitate the direct deployment of the required UI effect diagram to the page specified by the system, automatically complete the restoration and display of the UI effect diagram, significantly improve the generation quality and efficiency of the UI effect diagram, and ensure the consistency and efficiency of the design.

[0067] In this embodiment, a large amount of design drawing data is fine-tuned through multimodal large model technology, which effectively improves the generalization ability of the model. At the same time, the process of traditional deep learning solutions is simplified, reducing the dependence on large amounts of training data and complex pre-processing and post-processing steps, reducing development costs and improving development efficiency. A vector database is established, which describes in detail the JSON data structure and attribute descriptions of the system's built-in element components. The vector database enables the model to deeply understand and correctly apply these components, thereby ensuring the consistency and efficiency of the design when generating UI renderings. RAG (Retrieval-Augmented Generation) is adopted, so that the model can use the retrieval-augmented generation method when generating UI renderings, ensuring the consistency of the style and structure of the generated design drawings, and effectively improving the user experience and interface aesthetics.

[0068] As a second embodiment of the present application, performing fine-tuning training according to historical image data and a multimodal pre-training model further includes: Obtain key features of historical image data, and fine-tune the hierarchy based on key feature matching of historical image data; Use LoRA to perform fine-tuning training on the multimodal pre-trained model based on the fine-tuning level and historical image data.

[0069] Key features include basic visual features, complex structural features, and global semantic features. At this time, the key features of historical image data are obtained, and the fine-tuning level based on the key feature matching of historical image data includes: Acquire historical image data and identify key features of the historical image data; If the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a complex structural feature, the intermediate convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a global semantic feature, the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning level.

[0070] Basic visual features include color, texture, edge, etc. When there are components with unique appearance in historical image data, such as special color combinations or texture patterns, their basic visual features are more prominent and serve as key features. Complex structural features include shape, internal structure or layout, etc. When there are more complex relationships between components in historical image data, their complex structural features are more prominent and serve as key features. Global semantic features include component functions or domain-specific terms, etc. When there are component function descriptions in historical image data, their global semantic features are more prominent and serve as key features. The hierarchical structure of image encoding determines the degree of abstraction of its extracted features. The low layer (initial convolution layer): learns basic visual features such as edges, textures, colors, etc. The middle layer (intermediate convolution layer): captures more complex shapes and local structures. The high layer (top convolution layer or fully connected layer): learns global semantic information and advanced features. Different fine-tuning forms are selected for different historical image data, and the remaining layers are frozen at the same time, which reduces the amount of training calculations while being more targeted, achieving simultaneous improvement in efficiency and accuracy.

[0071] Fine-tuning the multimodal pre-trained model using LoRA based on the fine-tuning hierarchy and historical image data also includes: When the intermediate convolutional layer of the image encoding is used as the fine-tuning layer: Use LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level; Perform image encoding and text encoding on historical image data to obtain visual features and language features; Map visual features and language features based on the cross-attention module and calculate the attention weight; Obtain fused visual features based on attention weights and gated fusion mechanism; Based on the loss function and the fused visual features, the fine-tuning parameters are updated and fine-tuning training is performed.

[0072] In the multimodal projection layer, a cross-attention module is designed to map the visual features and language features to the same dimension through linear transformation, and then the attention weight is calculated to obtain the degree of correlation between the visual features and the language features. At the same time, the visual features and the language features are fused using the attention weight and the gating mechanism. The gating mechanism can dynamically adjust the importance of each feature in the fusion process according to the attention weight, thereby generating fused visual features containing rich cross-modal information. Define a suitable loss function to measure the difference between the model output and the true label. The gradient is calculated using the fused visual features through the back-propagation algorithm, and the parameters of the low-rank matrix after LoRA decomposition are updated. The entire model is fine-tuned using the updated parameters and other parts of the original model. During the training process, the performance of the model on the validation set is continuously monitored, and hyperparameters such as the learning rate and regularization parameter are adjusted as needed to optimize the performance of the model.

[0073] In this embodiment, the extracted visual features and language features are input into the cross attention module. The module maps the relationship between them by calculating the dot product or other similarity metrics between the two, and assigns an attention weight related to the language feature to each element in the visual feature according to the mapping result. The calculated attention weight and the gated fusion mechanism are used to fuse the visual features and the language features. The gating mechanism can dynamically adjust the importance of each feature in the fusion process according to the attention weight, so as to show the degree of training of the correlation between language and visual features and the nonlinear relationship between features based on the fused visual features, and use the loss function to judge whether the expected training requirements are met. If so, the fine-tuning training is terminated. If not, the fine-tuning parameters are updated based on the difference between the fused visual features and the actual visual features, and the fine-tuning training is continued.

[0074] In this embodiment, updating the fine-tuning parameters based on the loss function and the fused visual features, and performing the fine-tuning training further includes: When the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning layer: The prefix tuning technique is used to update the prefix vector parameters of the text encoding and perform fine-tuning training.

[0075] Using prefix tuning technology, only a small number of LLM parameters are fine-tuned to avoid overfitting and reduce the consumption of computing resources. Among them, the prefix key and value vectors are initialized for each Transformer layer, and the prefix length is set. In the multi-head attention, the extended key and value are updated according to the initialized prefix key and value vector and the current layer, while the query vector remains unchanged. Forward propagation is performed according to the attention score and the attention output, the cross entropy loss is calculated, and only the parameters of the prefix vector are updated, keeping most of the parameters of the pre-trained model unchanged.

[0076] The chart recognition task has its own unique visual features, such as chart type, text and symbols, color and style. By partially fine-tuning specific layers of image encoding, the model can be more focused on learning these features that are closely related to chart recognition. This targeted fine-tuning can significantly improve the model's ability to understand chart details, thereby improving recognition accuracy. At the same time, it avoids comprehensive fine-tuning of the entire image encoding, reduces the consumption of computing resources and training time, and achieves efficient fine-tuning. Further, by realizing the deep fusion of visual and language information in the projection layer, the model can simultaneously understand and process the visual features and text information in the chart. Through cross-modal attention and gating mechanisms, the model can capture the complex relationship between vision and language, thereby generating more accurate and rich chart descriptions, improving the model's ability to understand charts, and expanding the scope of application of the model, so that the model can handle more types of charts and more complex text descriptions, improving the generalization ability and practicality of the model. Furthermore, by fine-tuning a small number of prefix parameters to adapt to downstream tasks, the model can quickly adapt to different chart recognition tasks without large-scale adjustments to the entire model, avoiding model performance degradation or overfitting problems caused by excessive fine-tuning, and reducing the consumption of computing resources, making model fine-tuning more efficient.

[0077] In other embodiments, any fine-tuning method can also be combined, such as through the collaboration of image encoding fine-tuning and cross-modal fusion, the model can more accurately extract the visual features of the chart and the visual features can be combined with text information to generate a more accurate and rich chart description. Through the collaboration of cross-modal fusion and text encoding fine-tuning methods, the model can generate more accurate and fluent text descriptions while ensuring the stability and efficiency of the model during the fine-tuning process. By combining partial fine-tuning of image encoding, cross-modal fusion of enhanced projection layers, and adopting parameter-efficient text encoding fine-tuning methods, the model training achieves a balance between efficiency and accuracy, which can not only quickly adapt to different chart recognition tasks, but also generate accurate and rich chart descriptions, meeting the needs of practical applications.

[0078] As a third embodiment of the present application, a UI effect diagram recognition system based on a multimodal large model at least includes: Question and answer annotation module, which obtains historical image data based on user annotations and questions and answers; The fine-tuning training module performs fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model; Vector database, pre-stored built-in component data structures; The recognition generation module obtains the UI effect diagram to be recognized, outputs the first recognition data based on the multimodal large model and the UI effect diagram to be recognized, uses the retrieval algorithm to search the vector database for the first recognition data, and outputs the generation information corresponding to the UI effect diagram to be recognized.

[0079] In this embodiment, the question-answer annotation module is connected to the fine-tuning training module, and the fine-tuning training module and the vector database are connected to the recognition generation module. The recognition generation module at least includes an interaction unit for obtaining a UI effect diagram to be recognized.

[0080] The fine-tuning training module includes at least a visual encoder corresponding to image encoding, a multimodal projector corresponding to multimodal projection, and a language model corresponding to text encoding. In this embodiment, the language model is an LLM model. By partially fine-tuning the visual encoder, a specific layer of the visual encoder is fine-tuned to capture task-related visual features, and an advanced cross-modal mechanism is introduced to enable a deeper fusion of visual and language information at the projection layer, thereby improving the model's understanding and generation capabilities. Using prefix tuning technology, only a small number of parameters of the LLM are fine-tuned to avoid overfitting and reduce the consumption of computing resources. While improving the efficiency of model fine-tuning, the accuracy of model fine-tuning is improved, thereby improving the accuracy of subsequent models for UI renderings.

[0081] As a fourth embodiment of the present application, a computer-readable storage medium is used to store a computer program or instruction. When the computer program or instruction is executed by a processing device, the above-mentioned UI effect diagram recognition method based on a multimodal large model is implemented. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0082] The specific implementation methods described above are the preferred implementation methods of the UI effect diagram recognition method, system and medium based on the multimodal large model of the present application, and are not intended to limit the specific implementation scope of the present application. The scope of the present application includes but is not limited to the specific implementation methods. All equivalent changes made in accordance with the shape and structure of the present application are within the protection scope of the present application.

Claims

1. A UI effect diagram recognition method based on a multimodal large model, characterized by: The steps include: Construct a multimodal pre-training model based on image encoding, multimodal projection, and text encoding; Perform fine-tuning training based on historical image data and multimodal pre-trained models to obtain a multimodal large model; Build a vector database based on built-in component data structures; Obtain a UI effect diagram to be identified, and output first identification data based on the multimodal large model and the UI effect diagram to be identified; The vector database is searched for the first recognition data using a retrieval algorithm, and generation information corresponding to the UI effect diagram to be recognized is output.

2. The UI effect diagram recognition method based on multimodal large model as claimed in claim 1, characterized in that: The performing fine-tuning training according to the historical image data and the multimodal pre-trained model to obtain the multimodal large model includes: Acquire historical image data, and perform data preprocessing on the historical image data; Use LoRA to adjust the weight matrix of the multimodal pre-trained model; The historical image data after data preprocessing is input into the adjusted multimodal pre-trained model, and fine-tuning training is performed to obtain a multimodal large model.

3. The UI effect diagram recognition method based on multimodal large model according to claim 1, characterized in that: The step of searching the vector database for the first recognition data using a search algorithm and outputting generation information corresponding to the UI effect diagram to be recognized further includes: Using RAG to search the vector database for the first recognition data based on a preset prompt template, to obtain the component data with the highest matching similarity; The structural content data of the UI effect diagram to be identified is obtained, and the component data and the structural content data are integrated to obtain the generation information of the corresponding UI effect diagram to be identified.

4. The UI effect diagram recognition method based on multimodal large model according to claim 1, characterized in that: The performing fine-tuning training according to the historical image data and the multimodal pre-trained model to obtain the multimodal large model also includes: Construct triplets containing a diagram, its corresponding text description, and related questions; Compare the candidate models with the output results of the multimodal large model corresponding to the triplets based on the evaluation model, and output the evaluation results of the multimodal large model; Determine whether the multimodal large model requires iterative training based on the evaluation results.

5. The UI effect diagram recognition method based on multimodal large model according to claim 1, characterized in that: The constructing of a vector database based on the built-in component data structure includes: Obtain the relationship between diagram elements to build a knowledge graph; Obtain the built-in component data structure and build a vector database based on the knowledge graph and the built-in component data structure.

6. The UI effect diagram recognition method based on multimodal large model as claimed in claim 5, characterized in that: The constructing of a vector database according to the knowledge graph and the built-in component data structure also includes: Convert built-in component data structures into JSON object data; Extract key attributes from each JSON object data and convert them into feature vectors; Normalize the feature vector; The standardized feature vectors are relevance constructed according to the knowledge graph to obtain a vector database.

7. The UI effect diagram recognition method based on multimodal large model according to claim 1, characterized in that: The performing fine-tuning training according to the historical image data and the multimodal pre-training model also includes: Obtain key features of historical image data, and fine-tune the hierarchy based on key feature matching of historical image data; Use LoRA to perform fine-tuning training on the multimodal pre-trained model based on the fine-tuning level and historical image data.

8. The UI effect diagram recognition method based on multimodal large model as claimed in claim 7, characterized in that: The obtaining of key features of historical image data and fine-tuning of the level based on matching of key features of historical image data include: Acquire historical image data and identify key features of the historical image data; If the key features are basic visual features, the initial convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a complex structural feature, the intermediate convolutional layer of the image encoding is used as the fine-tuning layer; If the key feature is a global semantic feature, the top convolutional layer or fully connected layer of the image encoding is used as the fine-tuning level.

9. The UI effect diagram recognition method based on multimodal large model as claimed in claim 8, characterized in that: The method of using LoRA to perform fine-tuning training on the multimodal pre-trained model according to the fine-tuning level and historical image data also includes: Use LoRA to adjust the weight matrix of the multimodal pre-trained model according to the fine-tuning level; Perform image encoding and text encoding on historical image data to obtain visual features and language features; Map visual features and language features based on the cross-attention module and calculate the attention weight; Obtain fused visual features based on attention weights and gated fusion mechanism; Based on the loss function and the fused visual features, the fine-tuning parameters are updated and fine-tuning training is performed.

10. The UI effect diagram recognition method based on multimodal large model according to claim 2, characterized in that: The obtaining of historical image data and preprocessing of the historical image data include: Performing image normalization and size adjustment on the picture data in the historical image data; Perform information parsing and format conversion on the structural content data in the historical image data.

11. The UI effect diagram recognition method based on multimodal large model according to claim 10, characterized in that: The resizing of the image data in the historical image data comprises: Obtain the resolution of the multimodal pre-trained model, and obtain an adjustment value according to the resolution of the multimodal pre-trained model and the original resolution of the image data; When the adjustment value is negative, the nearest neighbor interpolation is called to perform size adjustment on the image data; When the adjustment value is positive, the image volume is calculated and the corresponding order interpolation method is called according to the image volume gradient.

12. A UI effect diagram recognition system based on a multimodal large model, used to implement the UI effect diagram recognition method based on a multimodal large model as claimed in any one of claims 1 to 11, characterized in that: include: Question and answer annotation module, which obtains historical image data based on user annotations and questions and answers; The fine-tuning training module performs fine-tuning training based on historical image data and multimodal pre-trained models to obtain a large multimodal model; Vector database, pre-stored built-in component data structures; The recognition generation module obtains the UI effect diagram to be recognized, outputs the first recognition data based on the multimodal large model and the UI effect diagram to be recognized, uses the retrieval algorithm to search the vector database for the first recognition data, and outputs the generation information corresponding to the UI effect diagram to be recognized.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or instruction. When the computer program or instruction is executed by a processing device, the method for recognizing UI effect diagrams based on a multimodal large model as described in any one of claims 1 to 11 is implemented.

Images