AI-based adaptive layout optimization method and system for visual elements

Abstract

This invention relates to the field of image processing technology, and more particularly to an AI-based method and system for adaptive layout optimization of visual elements. The method involves acquiring mixed image and text content, constructing a multimodal semantic association model to parse the semantic feature vectors of image and text elements, calculating semantic relevance in a unified semantic space, and constructing a semantic dependency graph to identify associated image-text pairs. The visual subject boundaries and visual safety regions of image elements are extracted as spatial repulsion and attraction constraints, respectively. A layout energy function based on the semantic dependency graph is established, and the target alignment position of text elements is solved by minimizing this function to generate the layout result of the mixed image and text. This invention achieves layout optimization through semantic perception and spatial constraints, improving the aesthetics and semantic consistency of the layout.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an AI-based method and system for adaptive layout optimization of visual elements. Background Technology

[0002] In graphic design and digital media, where text and images are mixed, existing layout methods primarily rely on human experience or automated layout algorithms based on geometric constraints. The conventional approach typically begins with basic image analysis, such as extracting the visual subject's outline through edge detection or salient region identification. Then, based on the rigid rule of non-overlapping text and images, the text is placed in blank areas or fixed, pre-defined positions within the image. Some systems employ a pre-set template library, matching the most similar layout template based on image size and text length, and adjusting the arrangement through scaling and displacement. The core idea behind these solutions is to simplify the layout problem into spatial collision detection and geometric filling of white space.

[0003] The aforementioned conventional approach has two significant drawbacks. First, geometrically constrained layout methods lack the ability to automatically perceive the semantic relationship between text and images. Text elements are often mechanically assigned to corners or edges not occupied by the main image subject, ignoring the matching relationship between the text content and the core semantics of the image. This results in key text being obscured behind the visual subject or placed in semantically irrelevant visual areas, compromising the integrity of information delivery. Second, the universality of layout templates is insufficient to cover diverse image content and text styles. When the main image subject has an irregular shape or the visual focus is off-center, the preset template cannot adaptively adjust, often resulting in text intruding into the visual interference area of ​​the main subject or being excessively far from related objects, reducing the professionalism and readability of the page.

[0004] Due to the lack of precise definition of visually safe areas, existing technologies generally adopt a global exclusion rule, which prohibits text from entering any image area. This is essentially a coarse-grained spatial constraint. This approach not only wastes areas in the image with simple background textures and low visual interference, but also fails to dynamically distinguish which areas should be absolutely avoided from being covered and which areas can accept appropriate overlap. Users often need to manually proofread and repeatedly adjust the coordinates of individual text blocks, which is time-consuming and cannot guarantee consistency. Summary of the Invention

[0005] This invention provides an AI-based method and system for adaptive layout optimization of visual elements, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides an AI-based adaptive layout optimization method for visual elements, comprising:

[0007] Obtain the mixed text and image content to be laid out, wherein the mixed text and image content includes image elements and text elements;

[0008] A multimodal semantic association model is constructed. The image elements are subjected to visual semantic analysis to obtain image semantic feature vectors, and the text elements are subjected to text semantic analysis to obtain text semantic feature vectors.

[0009] In a unified semantic space, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated. Based on the semantic relevance, a semantic dependency graph between image and text elements is constructed. Based on the edge weight distribution in the semantic dependency graph, associated image and text pairs are identified.

[0010] For the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted. The visual subject boundary is used as the spatial exclusion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element.

[0011] A layout energy function based on the semantic dependency graph is established. The target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image is generated according to the target alignment position.

[0012] Constructing a multimodal semantic association model, performing visual semantic analysis on the image elements using the multimodal semantic association model to obtain image semantic feature vectors, and performing text semantic analysis on the text elements to obtain text semantic feature vectors include:

[0013] Visual content decomposition is performed on image elements to extract the visual subject object, visual scene attributes, and visual style features of the image elements. The visual subject object, visual scene attributes, and visual style features are then mapped to a semantic description space to obtain the image semantic description of the image elements.

[0014] Semantic structure parsing is performed on text elements to extract their semantic topics, semantic entities, and semantic relationships. These semantic topics, semantic entities, and semantic relationships are then mapped to the semantic description space to obtain the text semantic description of the text elements.

[0015] By using a multimodal semantic association model to perform cross-modal semantic alignment of the image semantic description and the text semantic description in the semantic description space, a correspondence is established between the visual subject object and the semantic entity, and a correspondence is established between the visual scene attributes and the semantic topic.

[0016] Based on the aforementioned correspondences, the image semantic description is encoded into an image semantic feature vector, and the text semantic description is encoded into a text semantic feature vector.

[0017] In a unified semantic space, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated. Based on the semantic relevance, a semantic dependency graph between image and text elements is constructed. Then, based on the edge weight distribution in the semantic dependency graph, associated image-text pairs are identified, including:

[0018] In a unified semantic space, the semantic similarity and semantic complementarity between image semantic feature vectors and text semantic feature vectors are calculated, and the semantic similarity and semantic complementarity are nonlinearly fused to obtain the semantic relevance.

[0019] Using each image element and text element in the mixed image and text content as a node, a directed edge is established for each pair of image elements and text elements, and the semantic relevance is used as the edge weight of the directed edge to construct a semantic dependency graph.

[0020] Statistical analysis is performed on the edge weight distribution of the semantic dependency graph, and the mean and variance features of the edge weight distribution are extracted. The association strength partitioning threshold is adaptively calculated based on the mean and variance features, and the image elements and text elements connected by directed edges with edge weights greater than the association strength partitioning threshold are identified as associated graph-text pairs.

[0021] For the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted. The visual subject boundary is used as the spatial repulsion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element.

[0022] Visual attention prediction is performed on image elements in associated image-text pairs to generate a visual attention distribution map. Based on the gradient changes of the visual attention distribution map, the boundary lines of attention regions are identified, and the boundary lines are used as the boundaries of visual subjects.

[0023] Based on the visual subject boundary, the image elements are spatially segmented to obtain the core visual region within the visual subject boundary and the edge visual region outside the visual subject boundary.

[0024] For each candidate subregion in the edge visual region, a content substitutability assessment is performed. The content substitutability assessment is determined based on the ratio of the visual complexity of the candidate subregion to the attention value of the candidate subregion in the visual attention distribution map. Candidate subregions whose content substitutability assessment values ​​meet the coverage admission criteria are marked as visually safe regions.

[0025] The visual subject boundary is mapped to a spatial repulsion constraint of the text element. The spatial repulsion constraint applies a repulsive potential energy to the text element in the layout calculation. The visual safe area is mapped to a spatial attraction constraint of the text element.

[0026] Visual attention prediction is performed on image elements in associated image-text pairs to generate a visual attention distribution map. Based on the gradient changes in the visual attention distribution map, the boundaries of attention regions are identified, and these boundaries are used as the visual subject boundaries.

[0027] Calculate the color contrast and edge density of image elements, and non-linearly fuse the color contrast and edge density to obtain an initial attention intensity value;

[0028] Semantic topic words are extracted from text elements in associated image-text pairs. The semantic matching degree between the visual content corresponding to each pixel position in the image element and the semantic topic words is calculated. The initial attention intensity value is weighted and adjusted based on the semantic matching degree to obtain a visual attention distribution map.

[0029] Perform bidirectional gradient calculation on the visual attention distribution map to obtain horizontal gradient components and vertical gradient components, and calculate the gradient magnitude based on the horizontal gradient components and the vertical gradient components;

[0030] The local peak positions of the gradient magnitude are extracted as boundary feature points. The boundary feature points are connected according to spatial proximity to form a dividing line. The dividing line is then smoothed to obtain the visual subject boundary.

[0031] A layout energy function based on the semantic dependency graph is established. The target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image is generated based on the target alignment position, including:

[0032] Extract the set of associated edges from the semantic dependency graph. For each strongly associated edge in the set of associated edges, calculate the spatial distance between the text node and the image node it connects in the layout space. Define the product of the edge weight of the associated edge and the spatial distance as the one-sided semantic cost. Accumulate all the one-sided semantic costs to obtain the semantic association energy term.

[0033] The visual subject boundary is divided into multiple boundary segments. For each boundary segment, the minimum distance between the text element and the boundary segment is calculated. When the minimum distance is less than the safety interval threshold, the boundary proximity penalty value is calculated. All boundary proximity penalty values ​​are accumulated to obtain the boundary constraint energy term.

[0034] A layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is optimized by gradient descent. The layout energy function is minimized by iteratively adjusting the position coordinates of the text elements. The position coordinates corresponding to the minimum value are determined as the target alignment position. The text elements and the image elements are placed according to the target alignment position to generate a layout result of mixed text and image.

[0035] A layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is then optimized using gradient descent. Iterative adjustment of the text element position coordinates to minimize the layout energy function includes:

[0036] The first weight coefficient and the second weight coefficient are dynamically determined based on the ratio of the number of associated edges in the semantic dependency graph to the number of boundary segments of the visual subject boundary. The layout energy function is obtained by summing the product of the semantic association energy term and the first weight coefficient and the product of the boundary constraint energy term and the second weight coefficient.

[0037] The gradient vector is obtained by calculating the partial derivative of the layout energy function with respect to the position coordinates of the text element;

[0038] Calculate the position update amount along the opposite direction of the gradient vector, and then add the position update amount to the current position coordinates to obtain the updated position coordinates;

[0039] Determine whether the difference between the function value of the layout energy function at the updated position coordinates and the function value before the update is less than the convergence threshold. If it is less than the convergence threshold, terminate the iteration; if it is not less than the convergence threshold, continue the iteration. Adjust the position coordinates of the text elements iteratively to make the layout energy function reach the minimum value.

[0040] A second aspect of this invention provides an AI-based visual element adaptive layout optimization system, comprising:

[0041] The content acquisition unit is used to acquire the mixed text and image content to be laid out, which includes image elements and text elements;

[0042] The semantic parsing unit is used to construct a multimodal semantic association model, and to perform visual semantic parsing on the image elements to obtain image semantic feature vectors and to perform text semantic parsing on the text elements to obtain text semantic feature vectors.

[0043] The association recognition unit is used to calculate the semantic relevance between the image semantic feature vector and the text semantic feature vector in a unified semantic space, construct a semantic dependency graph between image and text elements based on the semantic relevance, and identify associated image and text pairs based on the edge weight distribution in the semantic dependency graph.

[0044] The constraint construction unit is used to extract the visual subject boundary and visual safe region of the image element for the associated image-text pair, and use the visual subject boundary as the spatial exclusion constraint of the text element and the visual safe region as the spatial attraction constraint of the text element.

[0045] The layout optimization unit is used to establish a layout energy function based on the semantic dependency graph, solve the target alignment position of the text element relative to the image element by minimizing the layout energy function, and generate a layout result of mixed text and image based on the target alignment position.

[0046] A third aspect of the present invention provides an electronic device, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0051] The semantic relevance of the mixed text and image layout is significantly improved. Text and image elements are automatically aligned based on high-level semantic features, avoiding the text-image separation problem caused by random or fixed arrangement in traditional layouts. The layout result can accurately reflect the mapping relationship between the main content of the image and the expressive intent of the text. For example, the visual subject boundary in the image acts as a repulsion constraint to prevent text from intruding into the main subject area, while the visual safe area acts as an attraction constraint to guide the text to automatically fit into the safe area, forming a visual effect of text and image. Users do not need to manually adjust the position, greatly reducing the design threshold and operation time.

[0052] The construction and minimization of the layout energy function ensures global optimality for text position optimization. Edge weight distribution based on the semantic dependency graph automatically identifies key related text-image pairs, ensuring that text-image elements with high semantic relevance are prioritized for close arrangement. The constraints of visual subject boundaries and visual safety zones work synergistically to avoid occlusion distortion caused by text overlapping with image subjects, and to prevent layout imbalance caused by text deviating from the visual center of gravity. This adaptive optimization method eliminates the need to redesign templates when dynamic content changes, significantly improving layout flexibility and reusability across multiple scenarios.

[0053] Multimodal semantic association models unify the semantic spaces of images and text. By calculating the semantic relevance of feature vectors, they can accurately capture the correspondence between implicit concepts in images and words in text, thereby generating highly matching layout schemes. Compared with traditional layout methods based on coordinates or dimensions, this method does not require manually labeled training data and can directly derive layout strategies from the content itself. It is particularly suitable for batch processing scenarios of mixed text and images, such as e-books, advertising posters, and social media covers. The layout results achieve a balance between visual consistency, information delivery efficiency, and artistic aesthetics, presenting an overall professional-grade typography quality. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the AI-based adaptive layout optimization method for visual elements according to an embodiment of the present invention.

[0055] Figure 2 This is a flowchart of the layout energy function optimization based on semantic dependency graph in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0058] Figure 1 This is a flowchart illustrating the AI-based adaptive layout optimization method for visual elements according to an embodiment of the present invention.

[0059] AI-based adaptive layout optimization methods for visual elements include:

[0060] Obtain the mixed text and image content to be laid out, wherein the mixed text and image content includes image elements and text elements;

[0061] A multimodal semantic association model is constructed. The image elements are subjected to visual semantic analysis to obtain image semantic feature vectors, and the text elements are subjected to text semantic analysis to obtain text semantic feature vectors.

[0062] In a unified semantic space, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated. Based on the semantic relevance, a semantic dependency graph between image and text elements is constructed. Based on the edge weight distribution in the semantic dependency graph, associated image and text pairs are identified.

[0063] For the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted. The visual subject boundary is used as the spatial exclusion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element.

[0064] A layout energy function based on the semantic dependency graph is established. The target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image is generated according to the target alignment position.

[0065] In one optional implementation, a multimodal semantic association model is constructed. The image elements are then subjected to visual semantic parsing using the multimodal semantic association model to obtain image semantic feature vectors. The text elements are then subjected to text semantic parsing to obtain text semantic feature vectors, including:

[0066] Visual content decomposition is performed on image elements to extract the visual subject object, visual scene attributes, and visual style features of the image elements. The visual subject object, visual scene attributes, and visual style features are then mapped to a semantic description space to obtain the image semantic description of the image elements.

[0067] Semantic structure parsing is performed on text elements to extract their semantic topics, semantic entities, and semantic relationships. These semantic topics, semantic entities, and semantic relationships are then mapped to the semantic description space to obtain the text semantic description of the text elements.

[0068] By using a multimodal semantic association model to perform cross-modal semantic alignment of the image semantic description and the text semantic description in the semantic description space, a correspondence is established between the visual subject object and the semantic entity, and a correspondence is established between the visual scene attributes and the semantic topic.

[0069] Based on the aforementioned correspondences, the image semantic description is encoded into an image semantic feature vector, and the text semantic description is encoded into a text semantic feature vector.

[0070] When decomposing image elements into visual content, a hierarchical feature extraction strategy is employed, deconstructing the image content into three dimensions: visual subject objects, visual scene attributes, and visual style features. The extraction of visual subject objects relies on object detection and instance segmentation techniques to identify foreground entities with independent semantic meaning in the image, such as people, products, animals, or buildings. Simultaneously, the category label, confidence score, and spatial position of each subject object in the image coordinate system are recorded. The extraction of visual scene attributes focuses on the overall environmental semantics of the image, including scene category, lighting conditions, spatial depth relationships, and background semantic labels. These attributes collectively describe the macro-contextual information presented by the image. The extraction of visual style features focuses on the visual expression of the image, covering color style, texture style, and composition style. These features play a crucial guiding role in subsequent matching with text layout styles.

[0071] The process of mapping the three types of visual information to the semantic description space is accomplished through a pre-trained visual-language alignment model. This model transforms the category labels and semantic attributes of visual subjects into natural language description fragments, visual scene attributes into scene semantic label sequences, and visual style features into style description word vectors. These three types of descriptive information are fused and concatenated in the semantic description space to form an image semantic description of the image elements. This description is stored in the form of a structured semantic vector sequence, preserving rich information about the image content at different semantic granularities.

[0072] When performing semantic structure analysis on text elements, a semantic analysis process based on deep language models is adopted, decomposing the text content into three levels: semantic topic, semantic entity, and semantic relation. Semantic topic extraction uses topic modeling methods to obtain the core issues of the text, reflecting the main information direction conveyed by the text as a whole, such as "product promotion," "natural landscape introduction," or "character story narration." Semantic entity extraction uses named entity recognition technology to locate words or phrases with clear referential meanings in the text, including names of people, places, organizations, products, and other proper nouns, and labels each entity with its semantic category and importance weight in the text. Semantic relation extraction uses dependency parsing and relation extraction techniques to identify semantic associations between entities and between entities and scenes in the text, such as "subject-action" relationships and "attribute-description" relationships. These relationships constitute the structural framework of the text's semantics.

[0073] Mapping semantic topics, semantic entities, and semantic relations to the same semantic description space as the image semantic description is a key prerequisite for achieving cross-modal alignment. This mapping process is accomplished through a shared semantic embedding layer, trained through contrastive learning on a large-scale image-text alignment dataset, capable of projecting semantic information from different modalities onto the same metric space. Semantic topics are encoded as topic semantic vectors, semantic entities as sets of entity semantic vectors, and semantic relations as relation semantic matrices; together, these three constitute the textual semantic description of the text elements.

[0074] When performing cross-modal semantic alignment in the semantic description space, the core task of multimodal semantic association models is to establish fine-grained correspondences between image semantic descriptions and text semantic descriptions. The correspondence between visual subject objects and semantic entities is established by calculating the cosine similarity between the entity semantic vector and the subject object's semantic description vector. Image subject objects and text entities with similarity exceeding a preset threshold are considered to have a semantic correspondence; for example, the subject object "laptop" in an image corresponds to the semantic entity "computer" in the text. The correspondence between visual scene attributes and semantic topics is established through bidirectional matching between scene semantic vectors and topic semantic vectors. The matching process considers both semantic similarity and semantic complementarity to ensure semantic consistency and coordination between the image scene and the text topic.

[0075] The training objective of cross-modal semantic alignment is to maximize the similarity of positive sample pairs in the semantic description space while minimizing the similarity of negative sample pairs, optimized using a contrastive loss function. During the inference phase, the alignment process not only outputs the existence judgment of correspondences but also the alignment confidence score for each pair of correspondences. This score will subsequently serve as the weight coefficient of the semantic constraint term in the layout energy function, ensuring that the strength of semantic association can quantitatively influence the final layout decision.

[0076] When encoding image semantic descriptions into image semantic feature vectors based on correspondences, an attention-weighted aggregation mechanism is employed. The alignment confidence scores between each visual subject and text entity are used as attention weights to weight and fuse the subject object semantic description vectors, scene attribute semantic vectors, and style feature semantic vectors, generating image semantic feature vectors with fixed dimensions. This vector not only preserves the global semantic information of the image, but also highlights visual components that are semantically relevant to the current text content through an attention mechanism, so that the position of the image semantic feature vector in the semantic description space can reflect the degree of semantic response of the image content to the specific text.

[0077] Encode the text semantic description into a text semantic feature vector. At the same time, a hierarchical semantic aggregation strategy is adopted. First, at the entity level, the semantic entity vectors are aggregated to obtain entity aggregate vectors. Then, at the topic level, the entity aggregate vectors are fused with the topic semantic vectors. Finally, the semantic relation matrix is ​​encoded into relation semantic vectors through a graph neural network, concatenated with the aforementioned fusion result, and compressed through a linear projection layer to the same size as the previous result. Using the same dimensional space, the final text semantic feature vector is obtained. The hierarchical aggregation strategy ensures that text semantic information at different granular levels is effectively preserved, avoiding the loss of fine-grained semantic information when directly encoding the entire text as a whole.

[0078] and The relative positional relationships within a unified semantic space directly determine the accuracy of subsequent semantic relevance calculations and semantic dependency graph construction. Through the complete processing flow of the aforementioned multimodal semantic association model, the semantic content of image and text elements is effectively transformed into computable and comparable vector representations, providing a solid semantic foundation for subsequent layout optimization.

[0079] In one optional implementation, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated in a unified semantic space; a semantic dependency graph between image and text elements is constructed based on the semantic relevance; and associated image-text pairs are identified based on the edge weight distribution in the semantic dependency graph, including:

[0080] In a unified semantic space, the semantic similarity and semantic complementarity between image semantic feature vectors and text semantic feature vectors are calculated, and the semantic similarity and semantic complementarity are nonlinearly fused to obtain the semantic relevance.

[0081] Using each image element and text element in the mixed image and text content as a node, a directed edge is established for each pair of image elements and text elements, and the semantic relevance is used as the edge weight of the directed edge to construct a semantic dependency graph.

[0082] Statistical analysis is performed on the edge weight distribution of the semantic dependency graph, and the mean and variance features of the edge weight distribution are extracted. The association strength partitioning threshold is adaptively calculated based on the mean and variance features, and the image elements and text elements connected by directed edges with edge weights greater than the association strength partitioning threshold are identified as associated graph-text pairs.

[0083] In a unified semantic space, the relationship between image semantic feature vectors and text semantic feature vectors cannot be described by a single dimension. The semantic association between the two modalities includes both content-level similarity and functional-level complementarity. Semantic similarity measures the degree of overlap in semantic expression between image content and text description. For example, an image showing flowers and text describing the colors of flowers have high semantic similarity. Semantic complementarity, on the other hand, measures the degree of synergistic gain between image and text at the information level. For example, an image presents a visual scene while text provides background explanation; although their content does not overlap, they have a strong information complementarity relationship. Relying solely on semantic similarity will miss a large number of image-text pairs with actual typographical relevance, therefore, it is necessary to fuse the two measures.

[0084] Semantic similarity is calculated using a normalized inner product approach. This involves performing a dot product between the image semantic feature vector and the text semantic feature vector in a unified semantic space, then dividing the result by the product of their magnitudes to obtain the semantic similarity. The semantic complementarity is calculated based on the principle of information difference, measuring the amount of independent information carried by each feature vector by calculating the magnitude of its orthogonal components in the semantic space. Specifically, after removing the projection components of the text semantic feature vector onto the image semantic feature vector, the magnitude of the remaining orthogonal components reflects the independent information contribution of the text relative to the image. Normalizing this magnitude yields the semantic complementarity. .

[0085] semantic similarity semantic complementarity When performing nonlinear fusion, a weighted exponential fusion strategy is adopted instead of simple linear weighting to avoid the saturation distortion problem of linear combinations near extreme values. Semantic relevance after fusion. The calculation method is as follows: ;

[0086] in and These are the nonlinear exponential parameters that contribute weights to controlling similarity and complementarity, respectively. The value range is typically set between 0.5 and 2.0. The value range is set between 0.5 and 1.5. When When the similarity is small, its contribution tends to saturate, meaning even low similarity can contribute significantly to the overall relevance; when... When the complementarity is large, a high level of complementarity is required to significantly improve the overall correlation. The above nonlinear fusion form ensures that when any one metric is close to 1, the overall correlation also approaches 1, and when both metrics are low, the overall correlation remains at a low level, exhibiting good numerical stability.

[0087] Using each image and text element in the mixed image and text content as a node, a directed edge is established for each pair of image and text elements, with the direction of the directed edge pointing from the image node to the text node, representing the semantic constraint relationship between the image and the text. The semantic relevance calculated above is then used to... Assign values ​​to the corresponding directed edges to construct a complete semantic dependency graph. Semantic dependency graph It is a bipartite directed graph, where the set of directed edges between the set of image nodes and the set of text nodes completely describes the distribution of semantic association strengths among all graph and text elements. For those containing Image elements and The semantic dependency graph contains a total of [number] text elements with mixed text and image layouts. There are 1 directed edge, and each edge carries an edge weight value between 0 and 1.

[0088] After constructing the semantic dependency graph, statistical analysis is performed on its edge weight distribution to identify graph-text pairs with substantial correlation. The edge weight values ​​of all directed edges in the semantic dependency graph are collected to form an edge weight sample set. The mean feature of this sample set is calculated. It reflects the average level of semantic association between the overall text and image; it calculates the variance feature. This reflects the dispersion of the semantic association strength between text and images. The mean and variance features together describe the overall distribution of semantic associations in the current mixed text and image content.

[0089] Association strength threshold The adaptive calculation method is as follows:

[0090] ;

[0091] in Let be the standard deviation of the edge weight distribution. For adaptive adjustment coefficients, The value is dynamically adjusted based on the complexity of the mixed text and image content: when the mixed text and image content has a large number of elements and a dense semantic distribution, A larger value is used to increase the threshold and avoid identifying too many weakly related text-image pairs; when the number of elements is small and the semantic distribution is sparse. A smaller value is used to lower the threshold, ensuring that a sufficient number of related image-text pairs can be identified for subsequent layout optimization. In practical applications, The typical value range is from 0.3 to 1.2.

[0092] The adaptive thresholding mechanism described above has significant advantages over the fixed thresholding method. Fixed thresholding often exhibits oversensitivity or undersensitivity when dealing with mixed text and image content of varying styles and complexities. For example, for content with generally low semantic relevance, a fixed threshold may fail to identify any valid text-image pairs; conversely, for content with generally high semantic relevance, a fixed threshold may identify almost all text-image pairs as related, rendering the filtering process meaningless. The adaptive thresholding mechanism, by anchoring the statistical characteristics of the current distribution through both mean and variance features, ensures that the threshold remains near a reasonable quantile of the edge weight distribution, guaranteeing the stability and effectiveness of the recognition results.

[0093] The edge weight is greater than the association strength threshold. Image elements and text elements connected by directed edges are identified as associated image-text pairs. These identified pairs serve as the basic units for subsequent spatial constraint extraction and layout energy function establishment. For cases where the same image element forms associated image-text pairs with multiple text elements, all relationships satisfying the threshold conditions are retained and arranged in descending order of edge weight, providing a priority reference for subsequent layout optimization. Similarly, for cases where the same text element satisfies the threshold conditions with multiple image elements, all relationships are retained and uniformly processed in the layout energy function through multi-constraint superposition, ensuring that the final position of the text element simultaneously satisfies the spatial relationship requirements with multiple associated image elements.

[0094] In one optional implementation, for the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted, and the visual subject boundary is used as the spatial repulsion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element, including:

[0095] Visual attention prediction is performed on image elements in associated image-text pairs to generate a visual attention distribution map. Based on the gradient changes of the visual attention distribution map, the boundary lines of attention regions are identified, and the boundary lines are used as the boundaries of visual subjects.

[0096] Based on the visual subject boundary, the image elements are spatially segmented to obtain the core visual region within the visual subject boundary and the edge visual region outside the visual subject boundary.

[0097] For each candidate subregion in the edge visual region, a content substitutability assessment is performed. The content substitutability assessment is determined based on the ratio of the visual complexity of the candidate subregion to the attention value of the candidate subregion in the visual attention distribution map. Candidate subregions whose content substitutability assessment values ​​meet the coverage admission criteria are marked as visually safe regions.

[0098] The visual subject boundary is mapped to a spatial repulsion constraint of the text element. The spatial repulsion constraint applies a repulsive potential energy to the text element in the layout calculation. The visual safe area is mapped to a spatial attraction constraint of the text element.

[0099] For image elements in a related image-text pair, a visual attention prediction process is first executed. The image elements are input into a saliency prediction network based on convolutional neural networks and multi-scale feature fusion. This network comprehensively utilizes shallow texture features and deep semantic features, outputting a visual attention distribution map with the same size as the original image. The value of each pixel in the attention distribution map represents the attractiveness of that location to the human visual system; a higher value indicates that the area is more likely to be noticed preferentially. The visual attention distribution map is essentially a continuous probability density heatmap, with pixel values ​​normalized to... Interval.

[0100] Based on the obtained visual attention distribution map, its spatial gradient is calculated to identify the boundaries of attention regions. (The text then abruptly shifts to a seemingly unrelated topic: attention distribution map.) Partial derivatives are calculated in both the horizontal and vertical directions to obtain the gradient magnitude map. Locations with high gradient magnitudes correspond to regions where attention values ​​change drastically, i.e., the transition zone between the core visual subject and the background. Non-maximum suppression is applied to the gradient magnitude map to suppress non-edge responses and retain pixels with local maxima gradient responses. Then, a double-threshold hysteresis connection algorithm is used to connect the discrete pixels with strong edge responses into a complete closed contour. This closed contour is the visual subject boundary, which physically represents the spatial boundary of the core object that attracts the most visual attention in the image. The accuracy of visual subject boundary extraction directly affects the accuracy of subsequent spatial constraints. Therefore, a Gaussian smoothing filter is applied to the attention distribution map before gradient calculation to eliminate the interference of local noise on gradient calculation. The standard deviation of the smoothing kernel is... The settings are adaptively adjusted based on image resolution; a larger value is used for higher resolution images. To avoid over-segmentation.

[0101] Image elements are spatially segmented based on visual subject boundaries, dividing the image into two non-overlapping regions. The region inside the visual subject boundary is defined as the core visual region, which carries the main semantic information and visual focus of the image; the region outside the visual subject boundary is defined as the edge visual region, which is usually background, white space, or auxiliary content with relatively low visual attention. Edge visual regions are potential coverage spaces for text elements, but not all edge visual regions are suitable for placing text, requiring further evaluation of the substitutability of content in each candidate sub-region.

[0102] A set of candidate sub-regions is generated by dividing the visual edge region using a regular grid. The size of each candidate sub-region is consistent with the bounding box size of the text element to be laid out, to ensure that the evaluation results match the actual text coverage scenario. For each candidate sub-region... Content substitutability is assessed, with evaluation metrics considering both visual complexity and attention scores. Visual complexity of candidate sub-regions is also considered. The visual complexity is calculated by weighting the local standard deviation of pixel grayscale values ​​within the sub-region with the edge response density. Higher visual complexity indicates that the region contains rich texture details, and text coverage will cause a significant loss of visual information. The attention value of the candidate sub-region in the attention distribution map is also considered. The average attention value of all pixels within this sub-region is taken. Content substitutability assessment value. Defined as the ratio of attention value to visual complexity, i.e. .

[0103] when When the value is small, it indicates that the visual complexity of this sub-region is relatively high relative to its attention value. Covering this region will destroy a lot of effective visual information, and it is not suitable to place text there. A larger value indicates that the sub-region has low visual complexity and low attention value, meaning that being covered by text has little impact on the overall visual experience, making it suitable as a text placement area. Set the coverage threshold to... ,in The content substitutability threshold is obtained by statistically analyzing manually labeled acceptable text coverage areas in the training dataset. Candidate sub-regions that meet the coverage admission criteria are marked as visually safe regions. All visually safe regions together constitute the spatial set in which text elements can be legally covered. In the case of multiple adjacent visually safe regions, spatially adjacent and connected sub-regions are merged into a complete visually safe region block to reduce the number of constraints in subsequent layout calculations and improve computational efficiency.

[0104] Mapping the visual subject boundary to the spatial repulsion constraint of text elements physically means that during layout calculations, text elements are subjected to repulsive potential energy from the visual subject boundary, preventing them from moving towards the core visual area. Repulsive potential energy Defined as the shortest distance between the bounding box of a text element and the boundary of the visual subject. The inverse proportional function of the repulsive potential energy means that the smaller the distance between the text element and the visual subject boundary, the greater the repulsive potential energy, thus driving the text element away from the visual subject area during the energy minimization process. Specifically, ,in The repulsion strength coefficient, To prevent extremely small positive numbers with a denominator of zero, when the text element is completely outside the visual subject boundary, Take a positive value; when the text element overlaps with the visual subject boundary, Taking a negative value causes the repulsive potential energy to increase sharply, forming a strong penalty term in the energy function, ensuring that the text in the final layout will not obscure the core visual area.

[0105] Mapping the visual safety zone to spatial attraction constraints for text elements means that, physically, the text elements are subject to the attractive potential energy from the visual safety zone during layout calculations, guiding them to move towards and ultimately settle within the visual safety zone. Attractive potential energy Defined as the Euclidean distance between the center point of the bounding box of a text element and the center point of the nearest visually safe region. a quadratic function, i.e. ,in The attraction strength coefficient is denoted as . When the center of a text element coincides with the center of a visually safe area, the attraction potential energy reaches its minimum value of zero, indicating that this position is the ideal landing point for the text element. Repulsion and attraction constraints work together in the layout energy function. Through iterative solution by minimizing energy, text elements are encouraged to preferentially converge towards the visually safe area while avoiding occlusion of the core visual area. This achieves automatic optimization of the spatial relationship between text and images in mixed text and image layouts, ensuring that the final layout balances visual hierarchy clarity and readability.

[0106] In one optional implementation, visual attention prediction is performed on image elements in the associated image-text pair to generate a visual attention distribution map. Based on the gradient changes in the visual attention distribution map, the boundary lines of attention regions are identified, and these boundary lines are used as the visual subject boundaries.

[0107] Calculate the color contrast and edge density of image elements, and non-linearly fuse the color contrast and edge density to obtain an initial attention intensity value;

[0108] Semantic topic words are extracted from text elements in associated image-text pairs. The semantic matching degree between the visual content corresponding to each pixel position in the image element and the semantic topic words is calculated. The initial attention intensity value is weighted and adjusted based on the semantic matching degree to obtain a visual attention distribution map.

[0109] Perform bidirectional gradient calculation on the visual attention distribution map to obtain horizontal gradient components and vertical gradient components, and calculate the gradient magnitude based on the horizontal gradient components and the vertical gradient components;

[0110] The local peak positions of the gradient magnitude are extracted as boundary feature points. The boundary feature points are connected according to spatial proximity to form a dividing line. The dividing line is then smoothed to obtain the visual subject boundary.

[0111] like Figure 2 As shown, the method includes:

[0112] When predicting visual attention for image elements in a linked image-text pair, it is necessary to comprehensively consider the low-level visual features of the image itself and the high-level guiding information of the text semantics, thereby generating a visual attention distribution map that accurately reflects the area of ​​human eye focus. The extraction of low-level visual features is carried out from two dimensions: color contrast and edge density. Color contrast is obtained by calculating the mean Euclidean color difference between each pixel and its neighboring pixels in the LAB color space. The LAB color space has a good linear approximation of color differences perceived by the human eye and can more accurately reflect visually significant differences. Edge density is measured by applying the Sobel or Canny operator to the image and statistically analyzing the proportion of edge response points within the neighborhood window of each pixel. Dense edge regions often correspond to structurally complex and visually attractive areas in the image.

[0113] The initial attention intensity value is obtained by non-linearly fusing color contrast and edge density. Let the pixel position... The color contrast at that location is edge density is Then the initial attention intensity value at that location Calculated using the following nonlinear fusion method: ,in and These are nonlinear exponential parameters for color contrast and edge density, respectively, used to adjust the contribution ratio of each to the initial attention intensity. The power-law form of nonlinear fusion effectively suppresses noise interference in low-saliency regions while enhancing the response amplitude in high-saliency regions, avoiding the dilution problem between the two types of features encountered in linear weighted fusion.

[0114] After obtaining the initial attention intensity value, textual semantic information is introduced to weight and adjust it. For text elements in the associated image-text pair, semantic topic words are extracted using the TF-IDF method or a keyword extraction method based on a pre-trained language model. These semantic topic words represent the core semantic focus of the text content. For the local visual content corresponding to each pixel position in the image, its visual semantic description vector is extracted using a pre-trained visual-language alignment model, and the cosine similarity between this vector and the semantic topic word embedding vector is calculated as the semantic matching degree at that position. Regions with high semantic matching indicate that the visual content at that location in the image is highly relevant to the text topic and should therefore receive higher weight in the attention distribution. The initial attention intensity value is then weighted and adjusted based on the semantic matching degree to obtain the adjusted attention intensity value. The calculation method is as follows ,in This is the semantic guidance gain coefficient, which controls the degree to which the semantic matching score affects the attention intensity. It applies to all pixel locations across the entire image. After normalization, the final visual attention distribution map can be obtained.

[0115] Bidirectional gradient calculation of the visual attention distribution map is a crucial step in identifying the boundaries of attention regions. The Sobel operator is used to convolve the attention distribution map along both the horizontal and vertical directions to obtain the horizontal gradient component. and vertical gradient components Horizontal gradient components The vertical gradient component reflects the rate of change of attention values ​​in the left-right direction. This reflects the rate of change of the attention value in the vertical direction. The gradient magnitude at each pixel location is calculated based on the gradient components in both directions. The calculation formula is: Locations with larger gradient magnitudes correspond to regions of drastic intensity changes in the attention distribution map. These regions are located precisely at the boundary between high-attention and low-attention regions, and therefore can serve as candidate locations for the visual subject boundary.

[0116] Local peak locations are extracted from the gradient magnitude map as boundary feature points. The determination of local peaks uses a non-maximum suppression method: for each pixel location, two adjacent locations are selected along its gradient direction, one on each side. If the gradient magnitude at this location is strictly greater than the gradient magnitudes of its two adjacent locations, it is marked as a candidate boundary feature point. To further filter out false boundary feature points introduced by noise, dual thresholds for high and low gradient magnitudes are set. and Gradient magnitude exceeds The location was directly identified as a strong boundary feature point, and the gradient magnitude was between and Positions between and spatially adjacent to strong boundary feature points are identified as weak boundary feature points, with gradient magnitudes lower than 1. The positions that are not specified are then removed. The dual threshold mechanism can effectively suppress noise interference while ensuring the integrity of the boundary.

[0117] The selected boundary feature points are connected according to spatial proximity to form continuous boundary lines. A connection strategy based on minimum spanning tree or greedy search is adopted: starting from any unvisited strong boundary feature point, the nearest unconnected boundary feature point is searched within its 8-neighborhood, and this process is extended sequentially until no more points satisfying the distance threshold can be found. Up to the adjacent feature points, a boundary line segment is formed. The above process is repeated for all unvisited strong boundary feature points to obtain several boundary line segments. For endpoints with a distance less than the spacing tolerance threshold... Adjacent boundary segments are connected by interpolation to bridge the boundary breakage problem caused by insufficient local gradient response.

[0118] The completed boundary lines are smoothed to eliminate jagged noise introduced by pixel-by-pixel connections. The smoothing process employs a B-spline-based curve fitting method: the coordinate sequence of feature points on the boundary line is used as control points to fit a third-order B-spline curve. The smoothness is controlled by adjusting the spline node spacing, eliminating local jitter while preserving the overall direction of the boundary line. The smoothed boundary line serves as the visual subject boundary, spatially defining the transition zone between the visual subject area and the background area in the image. Using this visual subject boundary as a spatial exclusion constraint for text elements ensures that text content does not cover the core visual subject of the image, thereby achieving visual hierarchy separation of text and image content during layout optimization and improving the overall readability and aesthetics of the layout.

[0119] In one optional implementation, a layout energy function based on the semantic dependency graph is established, and the target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image elements is then generated based on the target alignment position, including:

[0120] Extract the set of associated edges from the semantic dependency graph. For each strongly associated edge in the set of associated edges, calculate the spatial distance between the text node and the image node it connects in the layout space. Define the product of the edge weight of the associated edge and the spatial distance as the one-sided semantic cost. Accumulate all the one-sided semantic costs to obtain the semantic association energy term.

[0121] The visual subject boundary is divided into multiple boundary segments. For each boundary segment, the minimum distance between the text element and the boundary segment is calculated. When the minimum distance is less than the safety interval threshold, the boundary proximity penalty value is calculated. All boundary proximity penalty values ​​are accumulated to obtain the boundary constraint energy term.

[0122] A layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is optimized by gradient descent. The layout energy function is minimized by iteratively adjusting the position coordinates of the text elements. The position coordinates corresponding to the minimum value are determined as the target alignment position. The text elements and the image elements are placed according to the target alignment position to generate a layout result of mixed text and image.

[0123] After constructing the semantic dependency graph and extracting the visual subject boundaries and visual safety regions, the structured information needs to be transformed into a quantifiable and optimizable mathematical objective to drive the text elements to find the optimal placement position in the image space. The establishment of the layout energy function is the core of this process. It encodes semantic association constraints and spatial geometric constraints into a differentiable scalar function. By minimizing this function, the target alignment position satisfying multiple constraints can be obtained.

[0124] From semantic dependency graph Extract all strongly correlated edges to form a set of correlated edges. For sets Each strongly correlated edge in Its connection image nodes With text nodes Let the image nodes be... The anchor point coordinates in the layout space are Text node The current position coordinates are The Euclidean distance between two nodes in the layout space is denoted as . The calculation method is as follows Set the edges The edge weight is Then the one-sided semantic cost of that edge is defined as Its physical meaning is: the stronger the semantic association ( The larger the value, the lower the cost when text and image elements are spatially close, thus driving the spatial proximity of strongly semantically related text-image pairs in layout. The semantic association energy term is obtained by summing the one-sided semantic costs of all strongly related edges. The expression is This energy factor ensures that semantically related text and images are not placed too far apart in the final layout, thus guaranteeing the semantic coherence of the visual presentation of text and images.

[0125] After establishing the semantic association energy term, it is also necessary to encode the spatial exclusion constraint of the visual subject boundary into a boundary constraint energy term. The visual subject boundary is then divided according to spatial continuity. Each boundary segment ( The bounding box of a text element is defined by a set of ordered boundary points. For the text element to be laid out, its bounding box is denoted as a rectangular region. For each boundary segment Calculate the bounding box of text elements With boundary segment Minimum geometric distance between This distance is the shortest distance from all points on the four edges of the text bounding box to the polyline of the boundary segment. Let the safety interval threshold be... ,when When this occurs, it indicates that the text element is too close to the visual subject boundary, and a penalty needs to be applied. The penalty value is calculated based on the boundary proximity. Defined as A squared form is used to make the penalty increase sharply as the distance decreases, thus forming a smooth and differentiable repulsive barrier. The penalty values ​​of all boundary segments satisfying the proximity condition are accumulated to obtain the boundary constraint energy term. The expression is ,in This is an indicator function. Its function is to prevent text elements from encroaching on the main visual area, protect the core content of the image from being obscured by text, and maintain the necessary visual spacing between the text and the image to ensure that the layout has good readability.

[0126] semantic association energy term With boundary constraint energy terms By performing weighted linear combinations, a complete layout energy function is established. Its expression is ,in These are the weighting coefficients for the semantic association energy term. These are the weighting coefficients for the boundary constraint energy term. The values ​​of the two weighting coefficients reflect the relative importance between semantic-driven and geometric constraints, and can be adjusted according to the layout scenario in practical applications: when the layout design emphasizes the close semantic relationship between text and images, the weighting coefficients should be increased appropriately. When the visual subject of an image needs strict protection and text must not intrude, the size should be appropriately increased. Layout energy function Regarding the position coordinates of text elements It is continuously differentiable, which provides a theoretical basis for using gradient descent for optimization.

[0127] Gradient descent method is used to analyze the layout energy function. Perform iterative optimization, and in each iteration, calculate... text element position coordinates and The partial derivatives are used to obtain the gradient vector. Then, update the position coordinates of the text elements in the negative direction of the gradient. Let the iteration step size (learning rate) be... The update rule for position coordinates is as follows: superscript This indicates the current iteration. To avoid gradient descent getting stuck in local minima, a momentum mechanism is introduced during the optimization process. The position update from the previous iteration is added to the current update step in a certain proportion, thus giving the optimization trajectory a certain inertia and enabling it to cross shallow local minima.

[0128] The iteration termination condition is set to one of the following two cases: when the change in the layout energy function between two adjacent iterations... Less than the preset convergence threshold When the optimization is considered to have converged, or when the number of iterations reaches the maximum limit for the number of iterations. When the iteration terminates, forcibly terminate the iteration. After the iteration terminates, record the current position coordinates of the text element. The target alignment position is determined. If there are multiple text elements in the scene, the position coordinates of all text elements are jointly optimized. The target alignment position of all text elements is solved simultaneously under the same layout energy function framework to ensure that multiple text elements do not overlap with each other, and the overall layout achieves an optimal balance in both semantic association and spatial constraints.

[0129] After obtaining the target alignment positions of all text elements, the text elements are placed on the page according to their respective target alignment positions, while the image elements remain in their original positions. Together, they constitute the final text-image layout. When generating the layout, the font size, line spacing, and other typography parameters of the text elements need to be fine-tuned to ensure that the actual area occupied by the text elements at their target alignment positions does not exceed the boundary of the visual safety zone. If the text elements still exceed the boundary after fine-tuning, the font size of the text elements is scaled proportionally according to the boundary of the visual safety zone until the text bounding box completely falls within the visual safety zone. The final output layout ensures that semantically related text and image elements remain spatially adjacent while ensuring that the text content does not obscure the visual subject of the image, achieving a synergistic optimization of semantic-driven and visual protection.

[0130] In one optional implementation, a layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is then optimized using gradient descent. Iterative adjustment of the text element's position coordinates to minimize the layout energy function includes:

[0131] The first weight coefficient and the second weight coefficient are dynamically determined based on the ratio of the number of associated edges in the semantic dependency graph to the number of boundary segments of the visual subject boundary. The layout energy function is obtained by summing the product of the semantic association energy term and the first weight coefficient and the product of the boundary constraint energy term and the second weight coefficient.

[0132] The gradient vector is obtained by calculating the partial derivative of the layout energy function with respect to the position coordinates of the text element;

[0133] Calculate the position update amount along the opposite direction of the gradient vector, and then add the position update amount to the current position coordinates to obtain the updated position coordinates;

[0134] Determine whether the difference between the function value of the layout energy function at the updated position coordinates and the function value before the update is less than the convergence threshold. If it is less than the convergence threshold, terminate the iteration; if it is not less than the convergence threshold, continue the iteration. Adjust the position coordinates of the text elements iteratively to make the layout energy function reach the minimum value.

[0135] When establishing the layout energy function, it is necessary to reasonably configure the relative weights of the semantic association energy term and the boundary constraint energy term so that the two types of constraints play a balanced regulatory role in the optimization process. Fixed weight coefficient schemes often exhibit significant limitations in different mixed text and image layout scenarios: when the visual subject boundaries in the image are complex and the number of boundary segments is large, the gradient contribution of the boundary constraint energy term will be relatively large, causing text elements to shift excessively to areas far from the image; conversely, when the number of semantic association edges is large and the number of boundary segments is small, the semantic association energy term will dominate the optimization direction, causing text elements to cluster near semantic anchor points while ignoring boundary safety constraints. Therefore, a dynamic weight determination strategy is adopted, adaptively calculating the first and second weight coefficients based on the ratio of the number of association edges in the semantic dependency graph to the number of boundary segments of the visual subject boundaries.

[0136] Let the total number of strongly related edges in the semantic dependency graph be... The total number of boundary segments used to define the visual subject boundary follows the aforementioned definition. Then the ratio of the number of boundary segments to the number of associated edges is defined as: ,in To prevent extremely small positive numbers with a denominator of zero, based on the ratio. First weighting coefficient With the second weighting coefficient The dynamic determination is based on the following relationship: when A larger value indicates that the number of boundary segments is greater than the number of associated edges, and the boundary constraint structure is more complex. In this case, the value should be increased appropriately. To strengthen the constraint of boundary constraint energy terms on the layout; when When the number of edges is smaller, the number of associated edges is relatively larger, and the semantic association structure is richer. In this case, the number of edges should be increased appropriately. To ensure semantic alignment. Specifically, let , Thus ensuring The sum of the two is normalized to avoid drastic fluctuations in the overall magnitude of the energy function as the scene changes. Based on this, the energy function is then deployed. Defined as semantic association energy term With boundary constraint energy terms Weighted linear combination: This dynamic weighting mechanism enables the layout optimization process to automatically adjust the relative contributions of the two types of constraints based on the structural characteristics of the specific text and image content, thereby improving the method's adaptability to different layout complexities.

[0137] After constructing the layout energy function, gradient descent is used to iteratively optimize it. The core operation of the optimization is to calculate... About text nodes Current position coordinates The partial derivative of is used to obtain the gradient vector at that location. .because Depend on and The gradient vector is obtained by linearly superimposing the terms and then summing them according to their weights. .in, about The gradient reflects the direction in which a text node is pulled toward its semantic anchor. about The gradient reflects the repulsive direction of the text node away from the visual subject boundary. The gradient vector resulting from the superposition of the two gradients comprehensively expresses the direction of the resultant force of semantic attraction and boundary repulsion.

[0138] The position update is calculated along the opposite direction of the gradient vector, i.e., the text node. In the The position update amount in the round of iteration is ,in This represents the iteration step size (learning rate) for gradient descent. This update is then added to the current position coordinates to obtain the updated position coordinates. Iteration step size The choice of step size has a significant impact on the convergence speed and stability of the optimization process. If the step size is too large, the position update is too aggressive, causing the energy function to oscillate around the minimum or even diverge; if the step size is too small, the convergence speed is slow, the number of iterations increases, and the computational cost increases. In practical implementation, a fixed step size strategy can be adopted. Set an empirical value within the range of 0.5% to 2% of the layout canvas size to strike a balance between convergence speed and stability.

[0139] After each round of position update, calculate the layout energy function value at the updated position coordinates. Function value at the original position coordinates The absolute value of the difference and compare it with the convergence threshold Compare. If If the layout energy function has converged sufficiently, the iteration process is terminated, and the current position coordinates are set. As a text node Target alignment position coordinates ;like If the current state is not met, the next iteration will continue, synchronously updating the position coordinates of all text nodes to be laid out. To prevent the energy function from failing to converge for an extended period in certain extreme scenarios, an upper limit on the maximum number of iterations is set. When the iteration round achieve The iteration is forcibly terminated regardless of whether the convergence condition is met, and the current position coordinates are used as the final output result.

[0140] Convergence threshold The settings need to be balanced with the accuracy requirements of the specific application scenario. In printing and typesetting scenarios where high layout accuracy is required, It can be set to a smaller value so that the optimization process stops closer to the true minimum; in interactive typesetting scenarios with high real-time requirements, The constraints can be appropriately relaxed to reduce the number of iterations and lower response latency. Through the complete iterative process of dynamic weight configuration, gradient calculation, position update, and convergence judgment, the target alignment position of all text elements relative to image elements can be solved under the dual conditions of satisfying semantic alignment constraints and boundary safety constraints, thereby generating a mixed text and image layout result that takes into account both visual aesthetics and semantic consistency.

[0141] A second aspect of this invention provides an AI-based visual element adaptive layout optimization system, comprising:

[0142] The content acquisition unit is used to acquire the mixed text and image content to be laid out, which includes image elements and text elements;

[0143] The semantic parsing unit is used to construct a multimodal semantic association model, and to perform visual semantic parsing on the image elements to obtain image semantic feature vectors and to perform text semantic parsing on the text elements to obtain text semantic feature vectors.

[0144] The association recognition unit is used to calculate the semantic relevance between the image semantic feature vector and the text semantic feature vector in a unified semantic space, construct a semantic dependency graph between image and text elements based on the semantic relevance, and identify associated image and text pairs based on the edge weight distribution in the semantic dependency graph.

[0145] The constraint construction unit is used to extract the visual subject boundary and visual safe region of the image element for the associated image-text pair, and use the visual subject boundary as the spatial exclusion constraint of the text element and the visual safe region as the spatial attraction constraint of the text element.

[0146] The layout optimization unit is used to establish a layout energy function based on the semantic dependency graph, solve the target alignment position of the text element relative to the image element by minimizing the layout energy function, and generate a layout result of mixed text and image based on the target alignment position.

[0147] A third aspect of the present invention provides an electronic device, comprising:

[0148] processor;

[0149] Memory used to store processor-executable instructions;

[0150] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0151] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0152] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based adaptive layout optimization method for visual elements, characterized in that, include: Obtain the mixed text and image content to be laid out, wherein the mixed text and image content includes image elements and text elements; A multimodal semantic association model is constructed. The image elements are subjected to visual semantic analysis to obtain image semantic feature vectors, and the text elements are subjected to text semantic analysis to obtain text semantic feature vectors. In a unified semantic space, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated. Based on the semantic relevance, a semantic dependency graph between image and text elements is constructed. Based on the edge weight distribution in the semantic dependency graph, associated image and text pairs are identified. For the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted. The visual subject boundary is used as the spatial exclusion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element. A layout energy function based on the semantic dependency graph is established. The target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image is generated according to the target alignment position.

2. The method according to claim 1, characterized in that, Constructing a multimodal semantic association model, performing visual semantic analysis on the image elements using the multimodal semantic association model to obtain image semantic feature vectors, and performing text semantic analysis on the text elements to obtain text semantic feature vectors include: Visual content decomposition is performed on image elements to extract the visual subject object, visual scene attributes, and visual style features of the image elements. The visual subject object, visual scene attributes, and visual style features are then mapped to a semantic description space to obtain the image semantic description of the image elements. Semantic structure parsing is performed on text elements to extract their semantic topics, semantic entities, and semantic relationships. These semantic topics, semantic entities, and semantic relationships are then mapped to the semantic description space to obtain the text semantic description of the text elements. By using a multimodal semantic association model to perform cross-modal semantic alignment of the image semantic description and the text semantic description in the semantic description space, a correspondence is established between the visual subject object and the semantic entity, and a correspondence is established between the visual scene attribute and the semantic topic. Based on the aforementioned correspondences, the image semantic description is encoded into an image semantic feature vector, and the text semantic description is encoded into a text semantic feature vector.

3. The method according to claim 1, characterized in that, In a unified semantic space, the semantic relevance between the image semantic feature vector and the text semantic feature vector is calculated. Based on the semantic relevance, a semantic dependency graph between image and text elements is constructed. Then, based on the edge weight distribution in the semantic dependency graph, associated image-text pairs are identified, including: In a unified semantic space, the semantic similarity and semantic complementarity between image semantic feature vectors and text semantic feature vectors are calculated, and the semantic similarity and semantic complementarity are nonlinearly fused to obtain the semantic relevance. Using each image element and text element in the mixed image and text content as a node, a directed edge is established for each pair of image elements and text elements, and the semantic relevance is used as the edge weight of the directed edge to construct a semantic dependency graph. Statistical analysis is performed on the edge weight distribution of the semantic dependency graph, and the mean and variance features of the edge weight distribution are extracted. The association strength partitioning threshold is adaptively calculated based on the mean and variance features, and the image elements and text elements connected by directed edges with edge weights greater than the association strength partitioning threshold are identified as associated graph-text pairs.

4. The method according to claim 1, characterized in that, For the associated image-text pair, the visual subject boundary and visual safe region of the image element are extracted. The visual subject boundary is used as the spatial repulsion constraint of the text element, and the visual safe region is used as the spatial attraction constraint of the text element. Visual attention prediction is performed on image elements in associated image-text pairs to generate a visual attention distribution map. Based on the gradient changes of the visual attention distribution map, the boundary lines of attention regions are identified, and the boundary lines are used as the boundaries of visual subjects. Based on the visual subject boundary, the image elements are spatially segmented to obtain the core visual region within the visual subject boundary and the edge visual region outside the visual subject boundary. For each candidate subregion in the edge visual region, a content substitutability assessment is performed. The content substitutability assessment is determined based on the ratio of the visual complexity of the candidate subregion to the attention value of the candidate subregion in the visual attention distribution map. Candidate subregions whose content substitutability assessment values ​​meet the coverage admission criteria are marked as visually safe regions. The visual subject boundary is mapped to a spatial repulsion constraint of the text element. The spatial repulsion constraint applies a repulsive potential energy to the text element in the layout calculation. The visual safe area is mapped to a spatial attraction constraint of the text element.

5. The method according to claim 4, characterized in that, Visual attention prediction is performed on image elements in associated image-text pairs to generate a visual attention distribution map. Based on the gradient changes in the visual attention distribution map, the boundaries of attention regions are identified, and these boundaries are used as the visual subject boundaries. Calculate the color contrast and edge density of image elements, and non-linearly fuse the color contrast and edge density to obtain an initial attention intensity value; Semantic topic words are extracted from text elements in associated image-text pairs. The semantic matching degree between the visual content corresponding to each pixel position in the image element and the semantic topic words is calculated. The initial attention intensity value is weighted and adjusted based on the semantic matching degree to obtain a visual attention distribution map. Perform bidirectional gradient calculation on the visual attention distribution map to obtain horizontal gradient components and vertical gradient components, and calculate the gradient magnitude based on the horizontal gradient components and the vertical gradient components; The local peak positions of the gradient magnitude are extracted as boundary feature points. The boundary feature points are connected according to spatial proximity to form a dividing line. The dividing line is then smoothed to obtain the visual subject boundary.

6. The method according to claim 1, characterized in that, A layout energy function based on the semantic dependency graph is established. The target alignment position of the text element relative to the image element is solved by minimizing the layout energy function. The layout result of the mixed text and image is generated based on the target alignment position, including: Extract the set of associated edges from the semantic dependency graph. For each strongly associated edge in the set of associated edges, calculate the spatial distance between the text node and the image node it connects in the layout space. Define the product of the edge weight of the associated edge and the spatial distance as the one-sided semantic cost. Accumulate all the one-sided semantic costs to obtain the semantic association energy term. The visual subject boundary is divided into multiple boundary segments. For each boundary segment, the minimum distance between the text element and the boundary segment is calculated. When the minimum distance is less than the safety interval threshold, the boundary proximity penalty value is calculated. All boundary proximity penalty values ​​are accumulated to obtain the boundary constraint energy term. A layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is optimized by gradient descent. The layout energy function is minimized by iteratively adjusting the position coordinates of the text elements. The position coordinates corresponding to the minimum value are determined as the target alignment position. The text elements and the image elements are placed according to the target alignment position to generate a layout result of mixed text and image.

7. The method according to claim 6, characterized in that, A layout energy function is established by weighted linear combination of the semantic association energy term and the boundary constraint energy term. The layout energy function is then optimized using gradient descent. Iterative adjustment of the text element position coordinates to minimize the layout energy function includes: The first weight coefficient and the second weight coefficient are dynamically determined based on the ratio of the number of associated edges in the semantic dependency graph to the number of boundary segments of the visual subject boundary. The layout energy function is obtained by summing the product of the semantic association energy term and the first weight coefficient and the product of the boundary constraint energy term and the second weight coefficient. The gradient vector is obtained by calculating the partial derivative of the layout energy function with respect to the position coordinates of the text element; Calculate the position update amount along the opposite direction of the gradient vector, and then add the position update amount to the current position coordinates to obtain the updated position coordinates; Determine whether the difference between the function value of the layout energy function at the updated position coordinates and the function value before the update is less than the convergence threshold. If it is less than the convergence threshold, terminate the iteration; if it is not less than the convergence threshold, continue the iteration. Adjust the position coordinates of the text elements iteratively to make the layout energy function reach the minimum value.

8. An AI-based visual element adaptive layout optimization system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The content acquisition unit is used to acquire the mixed text and image content to be laid out, which includes image elements and text elements; The semantic parsing unit is used to construct a multimodal semantic association model, and to perform visual semantic parsing on the image elements to obtain image semantic feature vectors and to perform text semantic parsing on the text elements to obtain text semantic feature vectors. The association recognition unit is used to calculate the semantic relevance between the image semantic feature vector and the text semantic feature vector in a unified semantic space, construct a semantic dependency graph between image and text elements based on the semantic relevance, and identify associated image and text pairs based on the edge weight distribution in the semantic dependency graph. The constraint construction unit is used to extract the visual subject boundary and visual safe region of the image element for the associated image-text pair, and use the visual subject boundary as the spatial exclusion constraint of the text element and the visual safe region as the spatial attraction constraint of the text element. The layout optimization unit is used to establish a layout energy function based on the semantic dependency graph, solve the target alignment position of the text element relative to the image element by minimizing the layout energy function, and generate a layout result of mixed text and image based on the target alignment position.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images