A graphic design element importance detection method, system, device and medium based on weak supervision training

Abstract

A graphic design element importance detection method, system, device and medium based on weakly supervised training, the method comprising: manually labeling a global grid sequence dataset; dividing the graphic design into global and local grids through weakly supervised training; obtaining a predicted sequence and a relative weight using a local sequence prediction model and a global sequence prediction model; obtaining the relative weight of the text and visual features V t through a weight adaptive model; calculating an importance indicator of the graphic design element; the system, device and medium are used to implement the method; by introducing the division of the global grid and the local grid of the graphic design and the strategy of combining the multi-scale sequence prediction model, the training process is optimized and efficient sequence prediction is realized; the importance of the design element can be detected in an efficient, accurate and low-cost manner, providing a more feasible solution for the field of graphic design.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and creative media design, and in particular to a method, system, device and medium for detecting the importance of graphic design elements based on weakly supervised training. Background Technology

[0002] In today's digital age, the integration of graphic design and artificial intelligence has become a significant driving force for continuous innovation in the field of creative media design. With the rapid development of AI technology, especially the emergence of generative models, designers have gained more inspiration and creative possibilities in graphic design. AI technology can not only assist designers in complex graphic processing but also provide real-time feedback and suggestions needed for creative media design by learning from numerous design cases and trends. Against this backdrop, the importance detection of design elements has become a core technology of AI in creative media design. The main goal of this technology is to analyze and understand the importance of different design elements through deep learning algorithms, enabling more intelligent use of these elements in the graphic design process. Design elements are diverse, including color, shape, and text, and their interactions directly affect the overall effect of the design. Through importance detection, designers can more accurately grasp the position of each element in the overall layout, thereby improving the expressiveness and appeal of the design. The importance detection of design elements has broad value in practical applications. First, it can improve design efficiency, allowing designers to quickly find and emphasize key design elements, thus creating more profound and attractive works within a limited time. Secondly, it helps improve the user experience of designs. By rationally allocating the importance of various elements, it makes it easier for viewers to understand the design's intent and improves communication effectiveness. Most importantly, the importance detection of design elements drives creative media design towards a smarter direction that better meets human aesthetic needs, providing designers with more creative possibilities and space. Therefore, the importance detection of design elements is not only a hot topic but also one of the key technologies driving the continuous progress of creative media design.

[0003] In recent years, saliency detection in images and graphics has been extensively studied, resulting in a variety of detection methods. Common saliency detection methods include convolutional neural network methods based on deep learning, graph theory-based methods, and frequency domain analysis. However, these methods generally have some shortcomings. Some employ supervised learning, leading to high costs for labeling training datasets and limiting their feasibility in practical applications. Furthermore, some methods are limited by specific data domains, making them difficult to adapt to the broad field of graphic design, resulting in relatively low generalization ability. There is a clear distinction between importance detection and saliency detection. Saliency detection focuses on identifying salient regions in an image, typically achieved by highlighting features such as color, texture, or edges. However, importance detection emphasizes the hierarchical relationships and relative importance of graphic design elements, involving the understanding and analysis of the overall design structure. Traditional methods have failed to adequately meet this need because they tend to focus on saliency while neglecting the relative importance between elements, leading to unsatisfactory results in graphic design applications. Therefore, there is an urgent need for a low-cost, high-generalization method for detecting the importance of graphic design elements to overcome the shortcomings of existing methods. Such a method should be able to better capture the relationships between design elements, reduce reliance on large amounts of labeled data, and have sufficient generalization ability to adapt to the requirements of different fields and design styles, thus providing more effective support for the field of graphic design.

[0004] Patent application CN109741293A discloses an image saliency detection method and apparatus. This patent application uses supervised learning techniques to detect and label salient regions in an image. However, due to high training costs and limitations imposed by a specific data domain, its generalization ability is relatively low, making it difficult to adapt to a wide range of graphic design fields, thus resulting in inconsistent performance in practical applications.

[0005] Patent application CN111008558A discloses an image importance detection method based on relationships between people. This patent application infers the importance of people by learning and constructing relationships between people and the relationships between people and events in an image. Although this method can be used for importance detection in natural images, its practical application in the design field is significantly limited due to technical reasons such as high training costs, low generalization ability, and limited coverage. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for detecting the importance of graphic design elements based on weakly supervised training. By introducing a strategy that combines global and local grid partitioning of graphic design with a multi-scale sequence prediction model, the training process is optimized and efficient sequence prediction is achieved. This enables the detection of the importance of design elements in an efficient, accurate, and low-cost manner, providing a more feasible solution for a wide range of graphic design fields.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting the importance of graphic design elements based on weakly supervised training includes the following steps:

[0009] Step 1: Divide the acquired multiple planar designs into a K×K grid and manually annotate the design order of each cell in the grid at the grid level to obtain a dataset with annotations based on the global grid order, including the planar designs, K×K global grids, and the global grid order.

[0010] Step 2: Utilize the connectivity of components to extract elements of the graphic design from the dataset with global grid order annotations obtained in Step 1, resulting in text elements and visual elements.

[0011] Step 3: Construct a multi-scale sequence prediction model, including a local sequence prediction model and a global sequence prediction model. The input to the multi-scale sequence prediction model is the planar design in the dataset with global grid-based sequential annotations obtained in Step 1, and the output is the predicted sequence of a K×K global grid. S t It is a one-hot vector representing the prediction result at time t;

[0012] Step 4: Train the multi-scale sequence prediction model constructed in Step 3 using the dataset with global grid order annotations obtained in Step 1. During training, divide the K×K global grids of the dataset with global grid order annotations obtained in Step 1 into fixed-size M×M local grids. Input each local grid into the local sequence prediction model to obtain the prediction sequence for each local grid. Input the prediction sequences of the local grids and the planar design into the global sequence prediction model to obtain the prediction sequence of the global grids. Use the prediction sequences of the global grids and the global grid order in the dataset with global grid order annotations obtained in Step 1 to calculate the cross-quotient loss and perform backpropagation to update the parameters of the multi-scale sequence prediction model in Step 3. During this process, obtain the visual features V at each location. t and text features T t The relative weights;

[0013] Step 5: Utilize the text elements and visual elements obtained in Step 2, and the visual features V of each position in the global grid prediction sequence obtained in Step 4. t and text features T t The relative weights are used to calculate the importance index of the elements in the graphic design, and the final saliency mapping is obtained.

[0014] The specific method for step 2 includes:

[0015] Step 2-1: Extract initial elements from the planar design in the dataset with global grid order annotations obtained in Step 1 using the connectivity of the components;

[0016] Step 2-2: Determine the initial elements extracted in Step 2-1. If the overlap ratio of the bounding boxes of the elements exceeds the preset value α, then merge the adjacent elements to obtain a new set of elements.

[0017] Step 2-3: Use Optical Character Recognition (OCR) to detect all elements in the new element set from Step 2-2. If text is present, mark it as a text element; otherwise, mark it as a visual element.

[0018] The specific method for step 3 includes:

[0019] Step 3-1: Construct a local sequence prediction model. The local sequence prediction model includes an encoder and a Long Short-Term Memory (LSTM) network. The input is K×K global grids from the dataset obtained in Step 1 with global grid-based sequential annotations, and the output is the predicted sequence of the local grids. Where 0≤i≤K×K; O it It is a one-hot vector representing the prediction result of the i-th global grid at time t;

[0020] Step 3-2: Construct a global sequence prediction model. This model includes an encoder, a Long Short-Term Memory (LSTM) network, a weighted adaptive model, a text representation model, and a visual representation model. The input consists of K×K global grids from the dataset obtained in Step 1, labeled with global grid sequence annotations, and the predicted sequences of the local grids output in Step 3-1. The output is the predicted sequence of the K×K global grids. S t It is a one-hot vector representing the prediction result at time t.

[0021] The specific method for step 4 includes:

[0022] Step 4-1: Input each global grid of the dataset with global grid order annotation obtained in Step 1 into the encoder of the local sequence prediction model constructed in Step 3-1 to obtain the feature representation. Input the feature representation and the zero vector as START into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in Step 3-1 to obtain the probability O of each local grid being the first in the sequence. i1 The one with the highest probability is ranked as 1;

[0023] Step 4-2, take the probability O output in step 4-1. i1 The feature representation of the local grid cell with the highest probability of being ranked first is input into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in step 3-1 to obtain the probability of each small grid cell being ranked second, with the one with the highest probability being ranked second; this step is repeated to obtain the predicted sequences of all local grid cells. Where 0≤i≤K×K, O it It is a one-hot vector representing the prediction result of the i-th global grid at time t;

[0024] Step 4-3: Input the planar designs from the dataset with global grid order annotations obtained in Step 1 into the encoder of the global sequence prediction model to obtain the feature representation of the planar designs. Then, input the feature representation of the planar designs and the all-zero vector as START into the Long Short-Term Memory (LSTM) network of the global sequence prediction model to obtain the probability S1 of each global grid being ranked first; the one with the highest probability is ranked 1. The hidden state output by the LSTM network is h. t-1 ;

[0025] Step 4-4: Calculate the feature representations of the top-ranked global grid text and visual elements obtained in Step 4-3, and calculate the text feature T using the text representation model. t Use word2vec to map each word to a... The word embedding vectors are summed, and the sum is fed into a multilayer perceptron to obtain an ω-dimensional element vector. Finally, an h×w×ω text feature T is constructed by assigning element-wise vectors to all pixels within each text element and setting all remaining elements to zero. t h and w are the height and width of the global grid;

[0026] Visual features V are calculated using a visual representation model. tFirst, text elements are removed from the image, and the text pixels are filled with the background color of the graphic. Then, an image encoder based on a pre-trained classification network VGG16 is used to extract an image feature from the generated image. A global average pooling layer and two fully connected layers are added above the last convolutional layer of the VGG16 classification network to output an ω-dimensional image vector. Finally, an h×w×ω visual representation V is constructed using the same method as the text representation. t And set the other pixels to zero;

[0027] Step 4-5, based on the text features T obtained in step 4-4 t and visual features V t And the hidden state h output in step 4-3 t-1 The text features T at each position are obtained through a weighted adaptive model. t Relative weights M t As shown in the following formula:

[0028] M t =f(h) t-1 V t ,T t )

[0029] Among them, h t-1 M is the hidden layer state at time t-1. t It has the effect of weighted visual and textual representation contributing to content representation, where M t ∈[0,1] h×w f is a fully connected layer network;

[0030] Steps 4-6 involve processing the text features T obtained in step 4-4. t and visual features V t By utilizing the relative weight M obtained in steps 4-5 t By performing weighted summation, we obtain the feature representation C of the global grid. t ; as shown in the following formula:

[0031] C t =D c (M t )⊙T t +(1-D c (M t ))⊙V t

[0032] Where ⊙ represents element-wise multiplication, and D c The (·) function repeats M along the feature channel. t c times;

[0033] Step 4-7, convert the feature representation C of the global mesh obtained in step 4-6 into... tThe predicted sequences of all local grids output in step 4-2 and the probability S1 of each global grid being ranked first, output in step 4-3, are input into the global sequence prediction model to obtain the global grid ranked second. Steps 4-4 to 4-6 are repeated to obtain the predicted sequences of all global grids. S t It is a one-hot vector representing the prediction result at time t. The cross-quotient loss is calculated using the prediction sequence of the global grid and the global grid order in the dataset with global grid order annotation obtained in step 1, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model in step 3.

[0034] The specific method for step 5 includes:

[0035] Step 5-1: By calculating the overlap ratio, map the text elements or visual elements obtained in Step 2 onto the global grid. The overlap ratio is the intersection area between the global grid and the element divided by the minimum area between the global grid and the element. When the overlap ratio is greater than β, one global grid belongs to one element. For elements that have not been assigned any global grid, if the center position of the element is within the global grid, the element is set to belong to this global grid. Each element is associated with multiple global grids, thus obtaining the mapping relationship between each text element and visual element and the global grid.

[0036] Step 5-2: Based on the visual or text elements obtained in Step 2, and according to the mapping relationship obtained in Step 5-1, the text features T at each position obtained in Step 4-5 are processed. t Relative weights M t The significance values ​​are approximated by summing them, and the significance values ​​of all elements are normalized to [0,1], which means dividing each significance value by the maximum significance value, and convolving the result with a Gaussian filter to obtain the final significance mapping.

[0037] The global grid refers to each image patch that is divided into a K×K grid from the planar design in the dataset with global grid-based sequential annotations obtained in step 1.

[0038] Local grid refers to each image block that is an M×M grid divided from the global grid.

[0039] This invention also provides a planar design element importance detection system based on weakly supervised training, comprising:

[0040] A dataset acquisition module with global grid order annotation is used to divide multiple acquired planar designs into K×K grids and manually annotate the design order of each cell in the grid at the grid level, resulting in a dataset with global grid order annotation, including planar designs, K×K global grids, and global grid order.

[0041] The element extraction module is used to extract elements of a graphic design from a dataset with global grid order annotations by leveraging the connectivity of components, resulting in text and visual elements.

[0042] The multi-scale sequence prediction model building module is used to construct multi-scale sequence prediction models, including local and global sequence prediction models. The input of the multi-scale sequence prediction model is a planar design in the dataset with global grid order annotations, and the output is the predicted sequence of a K×K global grid. S t It is a one-hot vector representing the prediction result at time t;

[0043] The multi-scale sequence prediction model training module is used to train the multi-scale sequence prediction model using a dataset with global grid order annotations. During training, the K×K global grids of the dataset with global grid order annotations are divided into fixed-size M×M local grids. Each local grid is input into the local sequence prediction model to obtain the prediction sequence for each local grid. The prediction sequences of the local grids and the planar design are input into the global sequence prediction model to obtain the prediction sequence of the global grid. The cross-quotient loss is calculated using the prediction sequences of the global grids and the global grid order from the dataset with global grid order annotations, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model. During this process, the visual features V at each location are obtained. t and text features T t The relative weights;

[0044] The element importance detection module is used to predict the visual features V at each position in the sequence results using text elements, visual elements, and the global grid. t and text features T t The relative weights are used to calculate the importance index of the elements in the graphic design, and the final saliency mapping is obtained.

[0045] This invention also provides a planar design element importance detection device based on weakly supervised training, comprising:

[0046] Memory: A computer program that stores the above-mentioned method for detecting the importance of planar design elements based on weakly supervised training, and is a computer-readable device;

[0047] Processor: Used to implement the aforementioned method for detecting the importance of planar design elements based on weakly supervised training when executing the computer program.

[0048] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for detecting the importance of planar design elements based on weakly supervised training.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] 1. This invention introduces a weakly supervised training-based method for detecting the importance of graphic design elements by constructing a dataset with global grid order annotations in step 1 and training methods in step 4. This reduces the dependence on a large amount of labeled data, lowers training costs, improves generalization ability, and enhances the feasibility of applications in the field of graphic design.

[0051] 2. By combining local sequence prediction and global sequence prediction in step 3, this invention employs a step-by-step processing strategy of local and global grids, which enables the hierarchical relationships and relative importance of design elements to be captured more accurately, thereby enhancing the generalization ability and design adaptability of the model.

[0052] In summary, this invention reduces annotation costs and improves training efficiency by introducing a planar design element importance detection method based on weakly supervised training. Furthermore, by combining local and global sequence prediction, it enhances the model's generalization ability and can more accurately capture the hierarchical relationships and relative importance of design elements, demonstrating high adaptability and feasibility.

[0053] This invention significantly improves the efficiency and accuracy of graphic design analysis by introducing a weakly supervised training-based method for detecting the importance of graphic design elements. Compared to traditional methods, this technique eliminates the need for detailed element-level annotations when constructing training data, greatly reducing training costs. Furthermore, by employing a step-by-step processing strategy using local and global grids and a multi-scale sequence prediction model, it achieves efficient sequence prediction of graphic design elements. This not only gives the model a faster computation speed and shorter training time but also, by combining visual features V... t and text features T t The relative weight calculation improves the accuracy of importance detection, making it more generalizable in practical applications and providing a new, efficient, and low-cost solution for automated analysis in the field of graphic design. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the workflow of the present invention.

[0055] Figure 2 This is a schematic diagram of the overall structure of the multi-scale sequence prediction model of this invention.

[0056] Figure 3This is a schematic diagram of the structure of the local sequence prediction model of the present invention.

[0057] Figure 4 This is a schematic diagram of the structure of the global sequence prediction model of this invention.

[0058] Figure 5 This is a schematic diagram illustrating the relative importance scores of grid-level images and text predicted using a multi-scale sequence prediction model in an embodiment of the present invention.

[0059] Figure 6 This is a schematic diagram illustrating the relative importance of element-level images and text predicted using a multi-scale sequence prediction model in an embodiment of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Existing techniques for detecting the importance of graphic design elements are often hampered by expensive and cumbersome supervised training methods, which require large amounts of labeled datasets for model learning. This process not only incurs significant human and time costs but is also susceptible to issues such as labeling errors and inconsistencies, limiting the model's adaptability to real-world scenarios. Furthermore, traditional supervised training methods typically require substantial computational resources, resulting in significant energy and computational costs. These issues collectively lead to increased training costs and decreased model generalization ability, posing a series of challenges to the practical application of graphic design element importance detection. To address these problems, this invention proposes a graphic design element importance detection method based on weakly supervised training to overcome the shortcomings of existing technologies. By utilizing weakly supervised training, this invention can more effectively utilize unlabeled data, reducing data labeling requirements and significantly decreasing training costs. This method not only improves the model's generalization ability but also accelerates training speed, making graphic design element importance detection more efficient. Compared to traditional methods, the multi-scale sequence prediction model of this invention can achieve higher accuracy in a shorter time, while possessing lower costs and stronger generalization ability, bringing significant advantages to applications in the field of graphic design.

[0062] like Figure 1 As shown, a method for detecting the importance of planar design elements based on weakly supervised training includes the following steps:

[0063] Step 1: Divide the acquired multiple planar designs into a K×K grid and manually annotate the design order of each cell in the grid at the grid level to obtain a dataset with annotations based on the global grid order, including the planar designs, K×K global grids, and the global grid order.

[0064] Step 2: Utilize the connectivity of components to extract elements of the graphic design from the dataset with global grid order annotations obtained in Step 1, resulting in text elements and visual elements.

[0065] The specific method for step 2 includes:

[0066] Step 2-1: Extract initial elements from the planar design in the dataset with global grid order annotations obtained in Step 1 using the connectivity of the components;

[0067] Step 2-2: Determine the initial elements extracted in Step 2-1. If the overlap ratio of the bounding boxes of the elements exceeds the preset value of 0.3, then merge the adjacent elements to obtain a new set of elements.

[0068] Step 2-3: Use Optical Character Recognition (OCR) to detect all elements in the new element set from Step 2-2. If text is present, mark it as a text element; otherwise, mark it as a visual element.

[0069] Step 3, as follows Figure 2 As shown, a multi-scale sequence prediction model is constructed, including a local sequence prediction model and a global sequence prediction model. The input of the multi-scale sequence prediction model is the planar design in the dataset with global grid-based sequential annotations obtained in step 1, and the output is the predicted sequence of the K×K global grid. S t It is a one-hot vector representing the prediction result at time t;

[0070] The specific method for step 3 includes:

[0071] Step 3-1, as follows Figure 3 As shown, a local sequence prediction model is constructed. The local sequence prediction model includes an encoder and a Long Short-Term Memory (LSTM) network. The input is K×K global grids in the dataset with global grid-based sequential annotations obtained in step 1, and the output is the predicted sequence of the local grids. Where 0≤i≤K×K; O it It is a one-hot vector representing the prediction result of the i-th global grid at time t; the local sequence prediction model is used to predict the order of local grids.

[0072] The global grid refers to each image patch that is divided into a K×K grid from the planar design in the dataset with global grid-based sequential annotations obtained in step 1.

[0073] Local grid refers to each image block that is an M×M grid divided from the global grid.

[0074] like Figure 4As shown, step 3-2 involves constructing a global sequence prediction model. This model includes an encoder, a Long Short-Term Memory (LSTM) network, a weighted adaptive model, a text representation model, and a visual representation model. The input consists of K×K global grids from the dataset with global grid-based sequential annotations obtained in step 1, and the predicted sequences of the local grids output in step 3-1. The output is the predicted sequence of the K×K global grids. S t It is a one-hot vector representing the prediction result at time t; the global sequence prediction model is used to predict the order of the global grid.

[0075] Step 4: Train the multi-scale sequence prediction model constructed in Step 3 using the dataset with global grid order annotations obtained in Step 1. During training, divide the K×K global grids of the dataset with global grid order annotations obtained in Step 1 into fixed-size M×M local grids. Input each local grid into the local sequence prediction model constructed in Step 3-1 to obtain the prediction sequence for each local grid. Input the prediction sequences of the local grids and the planar design into the global sequence prediction model constructed in Step 3-2 to obtain the prediction sequence of the global grids. Use the prediction sequences of the global grids and the global grid order in the dataset with global grid order annotations obtained in Step 1 to calculate the cross-quotient loss and perform backpropagation to update the parameters of the multi-scale sequence prediction model in Step 3. During this process, the visual features V at each location will be obtained. t and text features T t The relative weights;

[0076] The specific method for step 4 includes:

[0077] Step 4-1: Input each global grid of the dataset with global grid order annotation obtained in Step 1 into the encoder of the local sequence prediction model constructed in Step 3-1 to obtain the feature representation. Input the feature representation and the zero vector as START into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in Step 3-1 to obtain the probability O of each local grid being the first in the sequence. i1 The one with the highest probability is ranked as 1;

[0078] Step 4-2, take the probability O output in step 4-1. i1 The feature representation of the local grid cell with the highest probability of being ranked first is input into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in step 3-1 to obtain the probability of each small grid cell being ranked second, with the one with the highest probability being ranked second; this step is repeated to obtain the predicted sequences of all local grid cells. Where 0≤i≤K×K, O itIt is a one-hot vector representing the prediction result of the i-th global grid at time t; such as Figure 3 As shown.

[0079] Step 4-3: Input the planar designs from the dataset with global grid order annotations obtained in Step 1 into the encoder of the global sequence prediction model to obtain the feature representation of the planar designs. Then, input the feature representation of the planar designs and the all-zero vector as START into the Long Short-Term Memory (LSTM) network of the global sequence prediction model to obtain the probability S1 of each global grid being ranked first; the one with the highest probability is ranked 1. The hidden state output by the LSTM network is h. t-1 ;

[0080] Step 4-4: Calculate the feature representations of the top-ranked global grid text and visual elements obtained in Step 4-3, and calculate the text feature T using the text representation model. t Use word2vec to map each word to a... The word embedding vectors are summed, and the sum is fed into a multilayer perceptron to obtain a 50-dimensional element vector. Finally, an h×w×50 text feature T is constructed by assigning element-level vectors to all pixels within each text element and setting all remaining elements to zero. t h and w are the height and width of the global grid;

[0081] Visual features V are calculated using a visual representation model. t First, text elements are removed from the image, and the text pixels are filled with the background color of the graphic. Then, an image encoder based on a pre-trained classification network VGG16 is used to extract an image feature from the generated image. A global average pooling layer and two fully connected layers are added above the last convolutional layer of the VGG16 classification network to output an ω-dimensional image vector. Finally, an h×w×50 visual representation V is constructed using the same method as the text representation. t And set the other pixels to zero;

[0082] Step 4-5, based on the text features T obtained in step 4-4 t and visual features V t And the hidden state h output in step 4-3 t-1 The text features T at each position are obtained through a weighted adaptive model. t Relative weights M t As shown in the following formula:

[0083] M t =f(h) t-1 V t ,T t )

[0084] Among them, h t-1 M is the hidden layer state at time t-1. t It has the effect of weighted visual and textual representation contributing to content representation, where M t ∈[0,1] h×w f is a fully connected layer network;

[0085] Steps 4-6 involve processing the text features T obtained in step 4-4. t and visual features V t By utilizing the relative weight M obtained in steps 4-5 t By performing weighted summation, we obtain the feature representation C of the global grid. t ; as shown in the following formula:

[0086] C t =D c (M t )⊙T t +(1-D c (M t ))⊙V t

[0087] Where ⊙ represents element-wise multiplication, and D c The (·) function repeats M along the feature channel. t c times;

[0088] Step 4-7, convert the feature representation C of the global mesh obtained in step 4-6 into... t The predicted sequences of all local grids output in step 4-2 and the probability S1 of each global grid being ranked first, output in step 4-3, are input into the global sequence prediction model to obtain the global grid ranked second. Steps 4-4 to 4-6 are repeated to obtain the predicted sequences of all global grids. S t It is a one-hot vector representing the prediction result at time t. The cross-quotient loss is calculated using the prediction sequence of the global grid and the global grid order in the dataset with global grid order annotation obtained in step 1, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model in step 3.

[0089] Step 5: Utilize the text elements and visual elements obtained in Step 2, and the visual features V of each position in the global grid prediction sequence obtained in Step 4. t and text features T t The relative weights are used to calculate the importance index of the elements in the graphic design, and the final saliency mapping is obtained.

[0090] The specific method for step 5 includes:

[0091] Step 5-1: By calculating the overlap ratio, map the text elements or visual elements obtained in Step 2 onto the global grid. The overlap ratio is the intersection area between the global grid and the element divided by the minimum area of ​​the global grid and the element. When the overlap ratio is greater than 0.5, one global grid belongs to one element. For elements that have not been assigned any global grid, if the center position of the element is within the global grid, the element is set to belong to this global grid. Each element is associated with multiple global grids, thus obtaining the mapping relationship between each text element and visual element and the global grid.

[0092] Step 5-2: Based on the visual or text elements obtained in Step 2, and according to the mapping relationship obtained in Step 5-1, the text features T at each position obtained in Step 4-5 are processed. t Relative weights M t The significance values ​​are approximated by summing them, and the significance values ​​of all elements are normalized to [0,1], which means dividing each significance value by the maximum significance value, and convolving the result with a Gaussian filter to obtain the final significance mapping.

[0093] Example

[0094] This invention discloses a method for detecting the importance of graphic design elements based on weakly supervised training. This method is applied to the problem of element importance detection in the graphic design process, and provides a method for detecting the importance of design elements that minimizes model training costs, avoids dataset limitations, and can generalize to other datasets.

[0095] To evaluate the importance detection method for design elements based on weakly supervised training, this invention conducted experiments on the algorithm from two aspects: the element perspective and the grid perspective. Each aspect is described below.

[0096] From a grid perspective, specifically, given a cell, this invention accumulates the weighted graphs of its visual and textual representations to obtain visual and textual weights, then normalizes them and sums them to 1 to obtain the importance scores of the visual and textual elements in the cell.

[0097] Experimental results show that, Figure 5 As shown, predicting the image and text in the first cell is almost equally important in understanding the cell's content. However, in the sixth cell, the text plays a more significant role due to the ambiguity of the visual elements.

[0098] From an element perspective, given a graphic design, calculate a heatmap of element design importance. This is because the model in this invention predicts a text feature T for each location. t Relative weights M tFor each cell in a global grid of a design, all local mappings need to be aggregated to generate a global saliency map for that design. Specifically, for a visual / text element, it is first associated with many cells based on the amount of overlap. The detailed association process is achieved by calculating the overlap ratio, which is the intersection area between a cell and an element divided by the minimum area of ​​the cell and the element. When the overlap ratio is greater than 0.5, a cell belongs to an element. For elements without any assigned cells, a cell is assigned to the element when the element's center is inside a cell. Thus, each element is associated with many cells with a prediction sequence. Then, its saliency value is approximated by summing the visual / text weights of the relevant cells in the element. Finally, all element-level saliency values ​​are normalized to [0,1] (i.e., each saliency value is divided by the maximum saliency value), and the result is convolved with a Gaussian filter to obtain the final saliency map.

[0099] Experimental results show that, Figure 6 As shown, the present invention can pinpoint the parts of a design that are important for the viewer to understand its information. For example, as shown in the second column, the method of the present invention demonstrates that handwritten text (left) and illustrations of people on a pedestrian crossing (middle) are equally important for understanding the message the design aims to convey: safe crossing.

[0100] The core of this invention is a proposed method for detecting the importance of graphic design elements based on weakly supervised training, aiming to solve the technical problems of low training cost and high generalization ability. The method includes a dataset with manually labeled global grid order, dividing the graphic design into global and local grids through weakly supervised training, and using local and global sequence prediction models to obtain predicted sequences and relative weights. Visual feature V t and text features T t The relative weights are obtained through a weighted adaptive model, and the importance index of the graphic design elements is finally calculated. Specific steps include a multi-scale sequence prediction model based on a Long Short-Term Memory (LSTM) network, and visual features V... t and text features T t The method involves extraction, weighted adaptive modeling, and saliency mapping generation. Compared to traditional methods, this invention employs a weakly supervised training approach, offering advantages such as low training cost and strong generalization ability. It also demonstrates significant superiority in terms of computational speed, training time, and accuracy. This method provides an efficient, accurate, and economical solution for importance detection of graphic design elements, and is expected to achieve widespread application in the field of graphic design.

[0101] The inventive points that need to be protected in this invention are: the network model of the solution design and the structure of each sub-module in the model, as well as the implementation process, method and steps.

[0102] This invention also provides a planar design element importance detection system based on weakly supervised training, comprising:

[0103] The dataset acquisition module with global grid order annotation is used to perform K×K grid division on the acquired multiple planar designs in step 1, and to manually annotate the design order of the content of each cell in the grid at the grid level, so as to obtain a dataset with global grid order annotation, including planar designs, K×K global grids and global grid order.

[0104] The element extraction module is used to extract the elements of the graphic design in the dataset with global grid order annotation obtained in step 1 by utilizing the connectivity of the components in step 2, so as to obtain text elements and visual elements.

[0105] The multi-scale sequence prediction model construction module is used to construct the multi-scale sequence prediction model in step 3, including a local sequence prediction model and a global sequence prediction model. The input of the multi-scale sequence prediction model is the planar design in the dataset with global grid order annotation obtained in step 1, and the output is the predicted sequence of the K×K global grid. S t It is a one-hot vector representing the prediction result at time t;

[0106] The multi-scale sequence prediction model training module is used in step 4 to train the multi-scale sequence prediction model constructed in step 3 using the dataset with global grid order annotations obtained in step 1. During training, the K×K global grids of the dataset with global grid order annotations obtained in step 1 are divided into fixed-size M×M local grids. Each local grid is input into the local sequence prediction model to obtain the prediction sequence for each local grid. The prediction sequences of the local grids and the planar design are input into the global sequence prediction model to obtain the prediction sequence of the global grids. The cross-quotient loss is calculated using the prediction sequences of the global grids and the global grid order in the dataset with global grid order annotations obtained in step 1, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model in step 3. During this process, the visual features V at each location are obtained. t and text features T t The relative weights;

[0107] The element importance detection module is used to perform step 5 by using the text elements and visual elements obtained in step 2, as well as the visual features V of each position in the global grid prediction sequence obtained in step 4. t and text features T t The relative weights are used to calculate the importance index of the elements in the graphic design, and the final saliency mapping is obtained.

[0108] This invention also provides a planar design element importance detection device based on weakly supervised training, comprising:

[0109] Memory: A computer program that stores the above-mentioned method for detecting the importance of planar design elements based on weakly supervised training, and is a computer-readable device;

[0110] Processor: Used to implement the aforementioned method for detecting the importance of planar design elements based on weakly supervised training when executing the computer program.

[0111] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for detecting the importance of planar design elements based on weakly supervised training.

[0112] The same or similar parts among the various embodiments in this specification can be referred to interchangeably. The above-described embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A method for detecting the importance of planar design elements based on weakly supervised training, characterized in that, Includes the following steps: Step 1: Divide the acquired multiple planar designs into a K×K grid and manually annotate the design order of each cell in the grid at the grid level to obtain a dataset with annotations based on the global grid order, including the planar designs, K×K global grids, and the global grid order. Step 2: Utilize the connectivity of components to extract elements of the graphic design from the dataset with global grid order annotations obtained in Step 1, resulting in text elements and visual elements. Step 3: Construct a multi-scale sequence prediction model, including a local sequence prediction model and a global sequence prediction model. The input to the multi-scale sequence prediction model is the planar design in the dataset with global grid-based sequential annotations obtained in Step 1, and the output is the predicted sequence of a K×K global grid. S t It is a one-hot vector representing the prediction result at time t; Step 4: Train the multi-scale sequence prediction model constructed in Step 3 using the dataset with global grid order annotations obtained in Step 1. During training, divide the K×K global grids of the dataset with global grid order annotations obtained in Step 1 into fixed-size M×M local grids. Input each local grid into the local sequence prediction model to obtain the prediction sequence for each local grid. Input the prediction sequences of the local grids and the planar design into the global sequence prediction model to obtain the prediction sequence of the global grids. Use the prediction sequences of the global grids and the global grid order in the dataset with global grid order annotations obtained in Step 1 to calculate the cross-quotient loss and perform backpropagation to update the parameters of the multi-scale sequence prediction model in Step 3. During this process, obtain the visual features V at each location. t and text features T t The relative weights; Step 5: Utilize the text elements and visual elements obtained in Step 2, and the visual features V of each position in the global grid prediction sequence obtained in Step 4. t and text features T t The relative weights are used to calculate the importance index of the graphic design elements, and the final saliency mapping is obtained.

2. The method for detecting the importance of planar design elements based on weakly supervised training according to claim 1, characterized in that, The specific method for step 2 includes: Step 2-1: Extract initial elements from the planar design in the dataset with global grid order annotations obtained in Step 1 using the connectivity of the components; Step 2-2: Determine the initial elements extracted in Step 2-1. If the overlap ratio of the bounding boxes of the elements exceeds the preset value α, then merge the adjacent elements to obtain a new set of elements. Step 2-3: Use Optical Character Recognition (OCR) to detect all elements in the new element set from Step 2-2. If text is present, mark it as a text element; otherwise, mark it as a visual element.

3. The method for detecting the importance of planar design elements based on weakly supervised training according to claim 1, characterized in that, The specific method for step 3 includes: Step 3-1: Construct a local sequence prediction model. The local sequence prediction model includes an encoder and a Long Short-Term Memory (LSTM) network. The input is K×K global grids from the dataset obtained in Step 1 with global grid-based sequential annotations, and the output is the predicted sequence of the local grids. Where 0≤i≤K×K; O it It is a one-hot vector representing the prediction result of the i-th global grid at time t; Step 3-2: Construct a global sequence prediction model. This model includes an encoder, a Long Short-Term Memory (LSTM) network, a weighted adaptive model, a text representation model, and a visual representation model. The input consists of K×K global grids from the dataset obtained in Step 1, labeled with global grid sequence annotations, and the predicted sequences of the local grids output in Step 3-1. The output is the predicted sequence of the K×K global grids. S t It is a one-hot vector representing the prediction result at time t.

4. The method for detecting the importance of planar design elements based on weakly supervised training according to claim 1, characterized in that, The specific method for step 4 includes: Step 4-1: Input each global grid of the dataset with global grid order annotation obtained in Step 1 into the encoder of the local sequence prediction model constructed in Step 3-1 to obtain the feature representation. Input the feature representation and the zero vector as START into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in Step 3-1 to obtain the probability O of each local grid being the first in the sequence. i1 The one with the highest probability is ranked as 1; Step 4-2, take the probability O output in step 4-1. i1 The feature representation of the local grid cell with the highest probability of being ranked first is input into the Long Short-Term Memory (LSTM) network of the local sequence prediction model constructed in step 3-1 to obtain the probability of each small grid cell being ranked second, with the one with the highest probability being ranked second; this step is repeated to obtain the predicted sequences of all local grid cells. Where 0≤i≤K×K, O it It is a one-hot vector representing the prediction result of the i-th global grid at time t; Step 4-3: Input the planar designs from the dataset with global grid order annotations obtained in Step 1 into the encoder of the global sequence prediction model to obtain the feature representation of the planar designs. Then, input the feature representation of the planar designs and the all-zero vector as START into the Long Short-Term Memory (LSTM) network of the global sequence prediction model to obtain the probability S1 of each global grid being ranked first; the one with the highest probability is ranked 1. The hidden state output by the LSTM network is h. t-1 ; Step 4-4: Calculate the feature representations of the top-ranked global grid text and visual elements obtained in Step 4-3, and calculate the text feature T using the text representation model. t Use word2vec to map each word to a... The word embedding vectors are summed, and the sum is fed into a multilayer perceptron to obtain an ω-dimensional element vector. Finally, an h×w×ω text feature T is constructed by assigning element-wise vectors to all pixels within each text element and setting all remaining elements to zero. t h and w are the height and width of the global grid; Visual features V are calculated using a visual representation model. t First, text elements are removed from the image, and the text pixels are filled with the background color of the graphic. Then, an image encoder based on a pre-trained classification network VGG16 is used to extract an image feature from the generated image. A global average pooling layer and two fully connected layers are added above the last convolutional layer of the VGG16 classification network to output an ω-dimensional image vector. Finally, an h×w×ω visual representation V is constructed using the same method as the text representation. t And set the other pixels to zero; Step 4-5, based on the text features T obtained in step 4-4 t and visual features V t And the hidden state h output in step 4-3 t-1 The text features T at each position are obtained through a weighted adaptive model. t Relative weights M t As shown in the following formula: M t =f(h t-1 ,V t ,T t ) Among them, h t-1 M is the hidden layer state at time t-1. t It has the effect of weighted visual and textual representation contributing to content representation, where M t ∈[0,1] h×w f is a fully connected layer network; Steps 4-6 involve processing the text features T obtained in step 4-4. t and visual features V t By utilizing the relative weight M obtained in steps 4-5 t By performing weighted summation, we obtain the feature representation C of the global grid. t ; as shown in the following formula: C t =D c (M t )⊙T t +(1-D c (M t ))⊙V t Where ⊙ represents element-wise multiplication, and D c The (·) function repeats M along the feature channel. t c times; Step 4-7, convert the feature representation C of the global mesh obtained in step 4-6 into... t The predicted sequences of all local grids output in step 4-2 and the probability S1 of each global grid being ranked first, output in step 4-3, are input into the global sequence prediction model to obtain the global grid ranked second. Steps 4-4 to 4-6 are repeated to obtain the predicted sequences of all global grids. S t It is a one-hot vector representing the prediction result at time t. The cross-quotient loss is calculated using the prediction sequence of the global grid and the global grid order in the dataset with global grid order annotation obtained in step 1, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model in step 3.

5. The method for detecting the importance of planar design elements based on weakly supervised training according to claim 1, characterized in that, The specific method for step 5 includes: Step 5-1: By calculating the overlap ratio, map the text elements or visual elements obtained in Step 2 onto the global grid. The overlap ratio is the intersection area between the global grid and the element divided by the minimum area between the global grid and the element. When the overlap ratio is greater than β, one global grid belongs to one element. For elements that have not been assigned any global grid, if the center position of the element is within the global grid, the element is set to belong to this global grid. Each element is associated with multiple global grids, thus obtaining the mapping relationship between each text element and visual element and the global grid. Step 5-2: Based on the visual or text elements obtained in Step 2, and according to the mapping relationship obtained in Step 5-1, the text features T at each position obtained in Step 4-5 are processed. t Relative weights M t The significance values ​​are approximated by summing them, and the significance values ​​of all elements are normalized to [0,1], which means dividing each significance value by the maximum significance value, and convolving the result with a Gaussian filter to obtain the final significance mapping.

6. The method for detecting the importance of planar design elements based on weakly supervised training according to claim 3, characterized in that, In step 3-1, the global grid refers to each image patch that is divided into a K×K grid based on the planar design in the dataset with global grid order annotation obtained in step 1. Local grid refers to each image block that is an M×M grid divided from the global grid.

7. A planar design element importance detection system based on weakly supervised training, based on the method of any one of claims 1 to 6, characterized in that, include: A dataset acquisition module with global grid order annotation is used to divide multiple acquired planar designs into K×K grids and manually annotate the design order of each cell in the grid at the grid level, resulting in a dataset with global grid order annotation, including planar designs, K×K global grids, and global grid order. The element extraction module is used to extract elements of a graphic design from a dataset with global grid order annotations by leveraging the connectivity of components, resulting in text and visual elements. The multi-scale sequence prediction model building module is used to construct multi-scale sequence prediction models, including local and global sequence prediction models. The input of the multi-scale sequence prediction model is a planar design in the dataset with global grid order annotations, and the output is the predicted sequence of a K×K global grid. S t It is a one-hot vector representing the prediction result at time t; The multi-scale sequence prediction model training module is used to train the multi-scale sequence prediction model using a dataset with global grid order annotations. During training, the K×K global grids of the dataset with global grid order annotations are divided into fixed-size M×M local grids. Each local grid is input into the local sequence prediction model to obtain the prediction sequence for each local grid. The prediction sequences of the local grids and the planar design are input into the global sequence prediction model to obtain the prediction sequence of the global grid. The cross-quotient loss is calculated using the prediction sequences of the global grids and the global grid order from the dataset with global grid order annotations, and backpropagation is performed to update the parameters of the multi-scale sequence prediction model. During this process, the visual features V at each location are obtained. t and text features T t The relative weights; The element importance detection module is used to predict the visual features V at each position in the sequence results using text elements, visual elements, and the global grid. t and text features T t The relative weights are used to calculate the importance index of the graphic design elements, and the final saliency mapping is obtained.

8. A device for detecting the importance of planar design elements based on weakly supervised training, characterized in that, include: Memory: A computer program for detecting the importance of planar design elements based on weakly supervised training as described in any one of claims 1-6, which is a computer-readable device; Processor: Used to implement the method for detecting the importance of planar design elements based on weakly supervised training as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the method for detecting the importance of planar design elements based on weakly supervised training as described in any one of claims 1-6.

Images