Image processing method and image processing device

Abstract

The invention discloses an image processing method and an image processing device, and belongs to the field of image processing. The method comprises the following steps: segmenting a first image into at least two local images with independent semantics through an image segmentation model, and determining description information of each local image; based on the description information of each local image, determining a target typesetting template matched with the semantics of the first image from a preset typesetting template library; and performing typesetting processing on the local image based on the target typesetting template to obtain a second image after typesetting processing.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method and an image processing apparatus. Background Technology

[0002] Currently, the artistic quality of images is usually enhanced by cropping and rearranging them into blocks.

[0003] In related technologies, image processing mainly employs manual typesetting or automatic typesetting using fixed templates. Manual typesetting is cumbersome and inefficient, while template-based typesetting, which only processes images according to preset templates, results in images lacking logic and aesthetic appeal, leading to unsatisfactory effects. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method and an image processing apparatus that can improve the efficiency and effectiveness of image processing.

[0005] In a first aspect, embodiments of this application provide an image processing method, including: The first image is segmented into at least two local images with independent semantics using an image segmentation model, and the descriptive information of each local image is determined. Based on the descriptive information of each local image, a target layout template that semantically matches the first image is determined from a preset layout template library; The local image is processed based on the target layout template to obtain the second image after layout processing.

[0006] Secondly, embodiments of this application provide an image processing apparatus, including: The image segmentation module is used to segment a first image into at least two local images with independent semantics using an image segmentation model, and to determine the descriptive information of each local image; The template matching module is used to determine the target layout template that semantically matches the first image from a preset layout template library based on the description information of each local image. The layout processing module is used to perform layout processing on a local image based on a target layout template to obtain a second image after layout processing.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, the first image is segmented according to its semantics to obtain at least two local images and descriptive information for each local image. Based on the descriptive information of each local image, a target layout template that semantically matches the first image is determined from a layout template library. Finally, the first image is processed for layout based on the local images and the target layout template to obtain a second image after layout processing. According to this embodiment, automatic image layout can be achieved, improving layout efficiency compared to manual layout. Furthermore, according to this embodiment, determining the layout template based on the image's semantics makes the layout image more semantically consistent with the original image, thereby improving the logic and artistry of the layout result and enhancing the overall effect of image processing. Attached Figure Description

[0012] Figure 1 This is one of the schematic flowcharts of an image processing method provided in some embodiments of this application; Figure 2 This is a second schematic flowchart of an image processing method provided in some embodiments of this application; Figure 3 This is the third of a flowchart illustrating an image processing method provided in some embodiments of this application; Figure 4 These are schematic diagrams of first images provided in some embodiments of this application; Figure 5 These are schematic diagrams of second images provided in some embodiments of this application; Figure 6 These are schematic diagrams of image processing apparatuses provided in some embodiments of this application; Figure 7 These are block diagrams of electronic devices provided in some embodiments of this application; Figure 8 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] Before providing a more detailed description of the embodiments of the present invention, the image processing methods in related technologies will be introduced. As mentioned earlier, the methods for segmenting, cropping, and rearranging images in related technologies often rely on manual operation or simple fixed templates. For example, users manually select different areas in a photo for cutting, and then adjust the position, size, and angle of each area based on experience. This processing method is tedious and requires a high level of aesthetic ability from the user. Some automatic layout tools can only segment images according to preset fixed templates (such as cinematic three-part composites, film-style two-part composites, etc.), using uniform cropping and scaling methods for all image areas. They cannot dynamically adjust the cutting and layout methods according to the image content, resulting in a mismatch between the segmented areas and the semantic content, a lack of logic and artistry in the layout results, and poor splicing effects.

[0016] The image processing method, apparatus, electronic device, storage medium, and program product provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0017] The image processing method provided in this application can be applied to image layout scenarios. The following section combines... Figures 1-5 The image processing method provided in the embodiments of this application will be described in detail. It should be noted that the image processing method provided in the embodiments of this application can be executed by an electronic device, such as a mobile phone, tablet computer, laptop computer, desktop computer, or other electronic device. This application embodiment uses an electronic device executing the image processing method as an example to illustrate the image processing method provided in the embodiments of this application.

[0018] See Figure 1The image processing method provided in some embodiments of this application includes the following steps 110-140, which will be described in detail below.

[0019] Step 110. Using an image segmentation model, the first image is segmented into at least two local images with independent semantics, and the descriptive information of each local image is determined.

[0020] In some embodiments of this application, the first image can be any image to be formatted in an image storage application on an electronic device. This image storage application can be, for example, a photo album app, a gallery app, a camera app, image processing software, etc. For instance, the first image can be an image captured by a user using a camera app on an electronic device.

[0021] In some embodiments of this application, the user inputs the first image to be typed into a pre-trained image segmentation model, thereby segmenting the first image into at least two local images with independent semantics and parsing out the descriptive information of each local image.

[0022] The descriptive information of a local image is information used to describe the local image. For example, the descriptive information of a local image may include, but is not limited to: the region identifier, semantic category, location information, and region association information between the local image and other local images.

[0023] The region identifier of a local image is a unique identifier for that local image; for example, the region identifier can be a region number. Each local image obtained by segmenting the first image corresponds to a region identifier, and different local images correspond to different region identifiers. Different local images can be distinguished based on the region identifiers.

[0024] The semantic category of a local image is used to indicate the semantics of the image elements contained within that local image. For example, semantic categories can be divided into portraits, grasslands, buildings, skies, and oceans. Different local images can correspond to different semantic categories.

[0025] The location information of the local image is used to indicate the position of the local image within the first image. For example, the location information of the local image may include the coordinates of the smallest bounding rectangle of the local image within the first image and the centroid coordinates of that smallest bounding rectangle. Wherein, the coordinates of the smallest bounding rectangle of the local image... Used to represent the boundary range of a local image. This represents the minimum x-coordinate of the smallest bounding rectangle. This represents the minimum y-coordinate of the smallest bounding rectangle. This represents the maximum x-coordinate of the smallest bounding rectangle. This represents the maximum y-coordinate of the smallest bounding rectangle. (After obtaining...) Then, the arithmetic mean of the coordinates of all pixels within the smallest bounding rectangle can be calculated, thus obtaining the centroid coordinates of the local image. Using the coordinates of the smallest bounding rectangle and the execution coordinates as the positional information of the local image ensures the accuracy of spatial positioning.

[0026] The region association information between a local image and other local images is information used to indicate the association relationship between a local image and other local images of the first image. For example, the region association information between a local image and other local images may include at least one of the following: an association score between the local image and adjacent local images, and a subjectivity score of the local image. The association score between the local image and adjacent local images indicates the association strength between the local image and its corresponding adjacent local images; a higher association score indicates a stronger association. The subjectivity score of the local image indicates the likelihood that the local image is the visual subject of the first image; a higher subjectivity score indicates a greater likelihood that the local image is the visual subject of the first image.

[0027] In some embodiments of this application, the image segmentation model is a pre-trained deep neural network. Before step 110 above, the deep neural network can be trained to obtain an image segmentation model capable of pixel-level semantic segmentation.

[0028] The training of deep neural networks can rely on tens of thousands of images with annotations covering multiple scenes and scales, including natural landscapes, urban architecture, and portrait photography. All images in the image set are annotated locally, and the annotations can cover various semantic categories such as sky, vegetation, buildings, human torso, and human parts, and include annotations on the relationships between different local images. The relationship types can include, but are not limited to, adjacent and overlapping. Semantic association labels can also be assigned to each pair of local images to identify semantic connections such as subject and background, subject and decoration, causal relationship, etc., so that the network can not only learn the semantic discrimination ability of single local images during training, but also perceive the structural and semantic relationships between different local images.

[0029] In some embodiments of this application, the image segmentation model may adopt a dual-branch structure, specifically including a segmentation branch corresponding to the segmentation module and a relation parsing branch corresponding to the relation parsing module. The segmentation module is mainly used to perform semantic segmentation on the input original image, obtaining local images with independent semantics and information about each local image itself, such as location information, region identifiers, and semantic categories. The relation parsing module is mainly used to parse the relationships between the local images obtained by the segmentation module, thereby obtaining the association information between the local images.

[0030] In some embodiments of this application, the segmentation module can be built based on the DeepLab-v3+ model, a semantic segmentation model that exhibits excellent performance in semantic segmentation by introducing an encoding / decoding structure and depthwise separable convolution. During the model training phase, a progressively dilated convolution strategy can be introduced on top of the ResNet-101 backbone network in the DeepLab-v3+ model, gradually increasing the dilation rate to an optimal state. This allows the segmentation module to maintain feature coherence while expanding the receptive field. The resolution of the feature map output by the segmentation module is 1 / 8 of the original input image size, thereby improving large-scale semantic consistency while preserving spatial details.

[0031] In some embodiments of this application, the key parsing module may include a graph attention network. The graph attention network can construct a region relationship graph, wherein the region relationship graph uses the feature vector of each local image as nodes, the relationship between local images as edges, and the relationship score between local images as the weight of the edges.

[0032] In some embodiments of this application, when the image segmentation model adopts a dual-branch structure including a segmentation module and a relation parsing module, the above step 110 may specifically include the following steps 1101-1103.

[0033] Step 1101. Using the segmentation module, perform semantic segmentation on the first image to obtain at least two local images with independent semantics and the region identifier, semantic category and location information of each local image.

[0034] Step 1102. Through the relationship parsing module, determine the region association information of each local image. The region association information is used to indicate the region association relationship between the local image and other local images.

[0035] In some embodiments of this application, where the region association information includes the association score between a local image and adjacent local images, see [link to relevant documentation]. Figure 2 In step 1102 above, the association score between the local image and the adjacent local image can be determined through the following steps a1-a3.

[0036] Step a1. Determine the adjacent local images for each local image.

[0037] Adjacent local images refer to local images that are adjacent to or intersect with the local image.

[0038] In some embodiments of this application, in step a1 above, for each local image, its adjacent local images can be determined by the following steps a111-a112.

[0039] Step a111. Determine the cross-union ratio between each local image and each other local image.

[0040] Here, "other local images" refers to all other local images in the first image besides the current local image. The current local image refers to the local image whose adjacent local images need to be determined in this instance. For example, if semantic segmentation of the first image yields five local images, numbered 1, 2, 3, 4, and 5, then when determining the adjacent local images of local image 1, local image 1 is the current local image, and local images 2, 3, 4, and 5 are the other local images.

[0041] In some embodiments of this application, the cross-union ratio between two local images can be calculated according to the following formula (1):

[0042] In the above formula (1), Representing a local image and local images The crossover ratio between them Representing a local image and local images The area of ​​intersection Representing a local image and local images The area of ​​the union of the sets.

[0043] Step a112. Other local images with a cross-union ratio greater than the cross-union ratio threshold are identified as adjacent local images of the local image.

[0044] In some embodiments of this application, the intersection-over-union (IoU) threshold is a threshold used to determine whether two local images are adjacent. The specific value of the IoU threshold can be set according to actual needs; for example, the IoU threshold is a value greater than or equal to 0.05.

[0045] For each other local image, after obtaining the intersection-over-union (IoU) ratio between the other local image and the current local image, the IoU ratio is compared with an IoU threshold. If the IoU ratio is greater than the threshold, the other local image is determined to be an adjacent local image of the current local image. For example, if the IoU threshold is set to 0.05, for each other local image, if its IoU ratio with the current local image is greater than 0.05, then the other local image is determined to be an adjacent local image of the current local image.

[0046] In some embodiments of this application, in step a1 above, for each local image, its adjacent local images can be determined by the following steps a121-a122.

[0047] Step a121. For each other local image, determine the shortest boundary distance between the boundary of the local image and the boundary of the other local images.

[0048] In some embodiments of this application, for each other local image, the shortest Euclidean distance between the boundary of the other local image and the current local image can be calculated, and the shortest Euclidean distance can be used as the shortest boundary distance between the other local image and the current local image.

[0049] Specifically, the shortest boundary distance between two local images can be calculated according to the following formula (2):

[0050] In the above formula (2), Representing a local image and local images The shortest boundary distance between them Representing a local image The set of boundary pixels, Representing a local image The set of boundary pixels, Representing a local image Pixels on the boundary Representing a local image Pixels on the boundary The Euclidean norm is used to describe direct adjacency or slight overlap in space.

[0051] Step a122. Other local images whose shortest boundary distance is less than the distance threshold are identified as adjacent local images of the local image.

[0052] In some embodiments of this application, the distance threshold is a threshold used to determine whether two local images are adjacent. The specific value of the distance threshold can be set according to actual needs; for example, the distance threshold can be a value less than or equal to 10 pixels.

[0053] For each other local image, after obtaining the shortest boundary distance between the other local image and the current local image, the shortest boundary distance is compared with a distance threshold. If the shortest boundary distance is less than the distance threshold, the other local image is determined to be a neighboring local image of the current local image. For example, if the distance threshold is set to 10 pixels, for each other local image, if its shortest boundary distance with the current local image is less than 10 pixels, then the other local image is determined to be a neighboring local image of the current local image.

[0054] In some embodiments of this application, adjacent local images can also be determined by combining the intersection-union ratio (IU) and the shortest boundary distance between a local image and other local images. For example, for each other local image, its IU with the current local image can be calculated first. If the IU is less than or equal to an IU threshold, the shortest boundary distance between it and the current local image can be further calculated. If the shortest boundary distance is less than a distance threshold, it is determined to be an adjacent local image of the current local image; otherwise, it is determined not to be an adjacent local image of the current local image.

[0055] Step a2. For each local image, calculate the spatial distance and feature similarity between the local image and its neighboring local images.

[0056] In some embodiments of this application, the association score between a local image and its corresponding adjacent local images is determined by combining the spatial distance and feature similarity. The spatial distance between two local images indicates their relative positional relationship in spatial arrangement. The feature similarity between two local images indicates the degree of similarity between them at the semantic content, visual feature, or abstract representation level.

[0057] In some embodiments of this application, the spatial distance between two local images can be represented by the distance between the centroid coordinates of the two local images. Specifically, the spatial distance between two local images can be calculated according to the following formula (3):

[0058] In the above formula (3), Representing a local image and local images Spatial distance between them Indicates the standard deviation, used to control the air decay rate. Representing a local image The coordinates of the centroid, Representing a local image The coordinates of the centroid, Represents an exponential function. It represents the square of the modulus.

[0059] In some embodiments of this application, when determining the feature similarity between any two local images, the feature vectors of each local image in the two local images can be extracted respectively, and then the cosine similarity of the feature vectors of the two local images can be calculated. This cosine similarity is used as the feature similarity between the two local images. Specifically, the feature similarity between two local images can be calculated according to the following formula (4):

[0060] In the above formula (4), Representing a local image and local images Feature similarity between Representing a local image eigenvectors, Representing a local image eigenvectors.

[0061] Step a3. Based on spatial distance and feature similarity, determine the association score between neighboring local images and local images of the local image.

[0062] In some embodiments of this application, for each local image pair consisting of the current local image and a neighboring local image corresponding to the current local image, after obtaining the spatial distance and feature similarity of the local image pair, the association score of the local image pair can be determined by weighted summation of the spatial distance and feature similarity. Specifically, the association score can be calculated according to the following formula (5):

[0063] In the above formula (5), Representing a local image and local images The association score of the local image pairs. This represents the first weight corresponding to spatial distance. This represents the second weight corresponding to feature similarity. The contributions of spatial distance and feature similarity can be balanced based on the first and second weights. The values ​​of the first and second weights can be set empirically.

[0064] In some embodiments of this application, in order to improve the accuracy of the obtained association score, for each local image pair consisting of the current local image and the adjacent local image, the association score of the local image pair can be determined by the following steps a31-a35.

[0065] Step a31. Based on the first weight corresponding to spatial distance and the second weight corresponding to feature similarity, perform a weighted sum of spatial distance and feature similarity to obtain the initial association score.

[0066] In some embodiments of this application, the initial association score can be calculated based on the above formula (5).

[0067] Step a32. Perform edge detection on the local image and the adjacent local image respectively to obtain the edge contour segments of the local image and the edge contour segments of the adjacent local image.

[0068] In some embodiments of this application, for each local image, its edge contour segments can be extracted using the Canny edge detection algorithm. The Canny edge detection algorithm has advantages such as low error rate, accurate localization, and single edge response, which can ensure the accuracy of the extracted edge contour segments.

[0069] Step a33. Based on the edge contour segments of the local image and the edge contour segments of adjacent local images, determine the collinear edge lengths of the local image and adjacent local images.

[0070] In some embodiments of this application, the collinear edge length of two local images refers to the length of the overlapping portion of the edges of the two local images. It can be determined based on the coordinates of the edge contour segments of the two local images. Specifically, the line segment composed of pixels with the same coordinates in the edge contour segments of the two local images is the collinear edge, and the length of this line segment is the collinear edge length.

[0071] Step a34. When the length of the collinear edge is greater than the length threshold, the sum of the initial association score and the preset score increment is determined as the association score between the adjacent local images of the local image.

[0072] In some embodiments of this application, the length threshold is a threshold used to determine whether the length of collinear edges of local image pairs meets a preset requirement. The value of the length threshold can be set according to actual needs and experience, and is not specifically limited thereto.

[0073] In some embodiments of this application, the score increment is a preset fixed increment, which is an integer greater than 0. If the length of the collinear edges of a local image pair is greater than a length threshold, it indicates that the edge continuity of the local image pair is strong. In this case, by increasing the score increment based on the initial association score of the local image pair, the semantic coherence of the local image pair can be strengthened, and the information transmission weight of local image pairs with strong edge continuity in the graph attention network can be significantly improved.

[0074] Step a35. When the length of collinear edges is less than or equal to the length threshold, the initial association score is determined as the association score between adjacent local images and local images of the local image.

[0075] In some embodiments of this application, if the collinear edge length of a local image pair is less than or equal to a length threshold, it indicates that the edge continuity of the local image pair is weak. In this case, the initial association score calculated earlier can be directly determined as the association score of the local image pair.

[0076] In the above scheme, edge consistency optimization can improve the correlation accuracy between adjacent local images.

[0077] In some embodiments of this application, where the region association information includes the subjectivity score of the local image, see [link to relevant documentation]. Figure 3 In step 1102 above, the region association information of each local image is determined through the following steps b1-b3.

[0078] Step b1. Determine the area proportion of each local image in the first image.

[0079] In some embodiments of this application, for each local image, the ratio of the number of pixels in the local image to the number of pixels in the first image can be determined as the area proportion of the local image in the first image. Specifically, the area proportion of the local image in the first image can be calculated according to the following formula (6):

[0080] In the above formula (6), Representing a local image First image The area proportion in Representing a local image The total number of pixels contained. Represents the first image The total number of pixels contained.

[0081] Step b2. Determine the visual saliency score for each local image.

[0082] In some embodiments of this application, the visual displayability score of a local image is used to indicate the attractiveness of the local image to human visual attention; the higher the visual displayability score, the stronger the attractiveness to human visual attention.

[0083] In some embodiments of this application, for each local image, the visual saliency score of that local image can be determined by a visual saliency model. Exemplarily, the visual saliency model can be such as the Itti-Koch model, the spectral residual model, GBVS, deep learning models (such as DSCL, RFCN, U²-Net), etc., and is not specifically limited thereto.

[0084] Step b3. For each local image, based on the third weight corresponding to the area proportion and the fourth weight corresponding to the visual saliency score, the area proportion and visual saliency score of the local image are weighted and summed to obtain the subjectivity score of the local image.

[0085] In some embodiments of this application, for each local image, the subjectivity score of the local image is determined by weighted summation of its area proportion and saliency score. Specifically, the subjectivity score of the local image can be determined according to the following formula (7):

[0086] In the above formula (7), Representing a local image Subjectivity score, This indicates the third weight corresponding to the area percentage. This indicates the fourth weight corresponding to the reveal score. Representing a local image The significance score is calculated. The values ​​of the third and fourth weights are consistent with the semantic primary and secondary determination strategy in the preceding training process.

[0087] In the above scheme, for each local image, the subjectivity score of the local image is determined by combining the area ratio of the local image in the first image and the saliency score of the local image. This balances saliency and entityness, avoids over-focusing on small but significant elements, ensures that the subject has a real presence, and achieves a subject recognition effect that is more in line with human cognition and more stable and reliable.

[0088] In some embodiments of this application, for each local image, after obtaining the subjectivity score of the local image, the decorative score of the local image can be further determined. Specifically, (1- The decorative score of the local image is used to ensure complementarity. The decorative score of the local image indicates the likelihood that the local image will serve as decorative content in the first image; a higher decorative score indicates a greater likelihood of it serving as decorative content.

[0089] Step 1103. For each local image, the region identifier, semantic category, location information, and association information between the local image and other local images are determined as the description information of the local image.

[0090] In some embodiments of this application, for the convenience of subsequent processing, the description information of the local images output by the image segmentation model can be structured data in JSON format, in which each entry fully records the region identifier, semantic category, location information, and association information with other local images of a local image, thereby providing an accurate and directly callable input basis for the subsequent generation of dynamic layout rules.

[0091] For example, the descriptive data of a certain local image is as follows: { "region_id":001, "category":"Human Trunk", "bbox":[120,205,380,720], "centroid":[250,462], "adjacent_regions":[002,003], "semantic_weight":{ "main_score": 0.88, "relational_strength":{"002":0.75,"003":0.92} } In this context, `region_id":001` indicates that the region identifier of the local image is 001; `bbox":[120,205,380,720]` indicates that the coordinates of the minimum bounding rectangle of the local image are [120,205,380,720]; `centroid":[250,462]` indicates that the centroid coordinates of the local image are [250,462]; `adjacent_regions":[002,003]` indicates that the adjacent local images of the local image include local images 002 and 003; `semantic_weight` indicates the association information of the local image, where `main_score":0.88` indicates that the main score of the local image is 0.88; and `relational_strength":{"002":0.75,"003":0.92}` indicates that the association score between the local image and local image 002 is 0.75, and the association score between the local image and local image 003 is 0.92.

[0092] Step 120. Based on the description information of each local image, determine the target layout template that semantically matches the first image from the preset layout template library.

[0093] In some embodiments of this application, a layout template library is pre-constructed, which includes multiple pre-constructed layout templates. Each layout template has a corresponding semantic scene. Based on this, after obtaining the description information of each local image in the first image, the semantic scene of the first image can be determined based on the obtained description information. Then, a layout template that matches the semantic scene of the first image is determined from the layout template library as the target layout template for typesetting the first image.

[0094] In some embodiments of this application, the typography template library can be constructed based on a systematic analysis of tens of thousands of art and design works and the experience of professional designers. First, computer vision technology can be used to automatically extract four basic typography templates from a large number of design works: triptych, central radial, diagonal symmetrical, and free montage. Each template defines the layout form and visual guidance path of a local area. Then, designers formulate a strict set of constraints for the templates, including constraints such as the position distribution matrix of the area on the output canvas, the allowable range of rotation angles, and the upper and lower thresholds of scaling ratios, to ensure the aesthetic standardization and visual balance of the typography.

[0095] In some embodiments of this application, after obtaining the layout template and constraints, a mapping relationship between semantic scenes and layout templates can be further constructed based on these templates and constraints. In the mapping relationship, nodes represent the types of predefined semantic scenes, such as grassland portraits or urban street scenes, and edges represent the mapping relationship between semantic scenes and layout templates. The weight of the edges can be determined through a designer voting and scoring mechanism to reflect the strength of the applicability of the template to a specific semantic scene. In this case, a semantic scene can correspond to at least one layout template.

[0096] In some embodiments of this application, step 120 may specifically include steps 1201-1203.

[0097] Step 1201. Determine the semantic scene of the first image based on the description information of each local image.

[0098] In some embodiments of this application, the description information of each local image includes information such as the semantic category of the local image. Based on this, the semantic scene of the first image can be determined by comprehensively analyzing the description information of all local images.

[0099] In some embodiments of this application, a classifier can be used to determine the semantic scene of the first image based on the descriptive information of each local image. Specifically, step 1202 above may include steps 12021-12023.

[0100] Step 12021. Generate input features based on the descriptive information of each local image.

[0101] In some embodiments of this application, the descriptive information of each local image can be processed based on the format of the input features supported by the classifier used, thereby obtaining input features suitable for the classifier.

[0102] In some embodiments of this application, a multilayer perceptron classifier can be used to determine the semantic scene of the first image. Based on this, the descriptive information of the acquired local images can be concatenated to obtain input features.

[0103] In some embodiments of this application, to adapt to the computational efficiency of electronic devices, a lightweight classifier can be used. For example, a multilayer perceptron classifier can be replaced with a random forest classifier. The random forest classifier is built based on multiple decision trees, and the depth of the decision trees can be set according to actual needs. When using a random forest classifier, the dimensions of the input features can be simplified to three core features: "total amount of local images, subjectivity score of each local image, and semantic category." Experiments have shown that the random forest classifier predicts the semantic scene of the first image in less than 50ms, which is about 60% more efficient than the multilayer perceptron classifier, and the semantic scene type matching accuracy remains above 92%, almost on par with the accuracy of the multilayer perceptron classifier.

[0104] In some embodiments of this application, if the electronic device supports lightweight CNN (Convolutional Neural Network) inference, a small CNN classifier can also be used to determine the semantic scene of the first image. For example, a small CNN classifier with 3 convolutional layers (3×3 kernel size) and 1 fully connected layer can be used, which has only 8k parameters and can further improve the inference speed to <30ms.

[0105] Step 12022. Input the input features into the classifier to obtain the probability distribution of the predefined semantic scene output by the classifier.

[0106] In some embodiments of this application, the predefined semantic scene is a pre-set semantic scene. For example, the predefined semantic scene may include, but is not limited to, grassland portrait, city street scene, etc.

[0107] In some embodiments of this application, the input features are input into a classifier, which analyzes the input features to determine the probability that the first image belongs to each predefined semantic scene, and finally outputs the probability distribution of the predefined semantic scene, which includes the probability that the first image belongs to each predefined semantic scene.

[0108] In some embodiments of this application, the classifier is a multilayer perceptron classifier. Based on this, after the input features are input into the multilayer perceptron classifier, the classifier can output a probability distribution containing a predefined semantic scene through several fully connected layers and nonlinear activation functions.

[0109] Step 12023. Based on the probability distribution, determine the predefined semantic scenes with probabilities greater than the probability threshold as the semantic scenes of the first image.

[0110] In some embodiments of this application, the probability threshold is a threshold used to filter out semantic scenes that match the first image. The probability threshold can be set based on expert experience; for example, the probability threshold can be set to 0.7, without specific limitation.

[0111] In some embodiments of this application, after obtaining the probability distribution output by the classifier, each probability value can be compared with a probability threshold to obtain a probability value greater than the probability threshold, and the predefined semantic scene corresponding to the probability value greater than the probability threshold is determined as the semantic scene of the first image.

[0112] Step 1202. Based on the preset mapping relationship between semantic scenarios and typesetting templates, determine at least one candidate typesetting template that matches the semantic scenario.

[0113] In some embodiments of this application, after obtaining the semantic scene of the first image, a preset mapping relationship between semantic scenes and layout templates can be found to determine layout templates that have a mapping relationship with the semantic scene of the first image, and these layout templates are identified as candidate layout templates. As mentioned above, in the mapping relationship between semantic scenes and layout templates, one semantic scene can correspond to at least one layout template. Therefore, at least one candidate layout template can be determined based on the mapping relationship between semantic scenes and layout templates.

[0114] Step 1203. Based on at least one candidate layout template, determine the target layout template that semantically matches the first image.

[0115] In some embodiments of this application, when there is only one candidate layout template, the candidate layout template can be directly determined as the target layout template that semantically matches the first image.

[0116] In some embodiments of this application, when there are at least two candidate layout templates, one of the at least two candidate layout templates can be selected as the target layout template for semantic matching with the first image.

[0117] In some embodiments of this application, in order to improve the semantic scene fit between the target layout template and the first image, when there are at least two candidate layout templates, the target layout template can be determined through the following steps 12031-12034.

[0118] Step 12031. Determine the region number matching score between the first image and each candidate layout template based on the number of local images in the first image and the preset number of regions for each candidate layout template.

[0119] In some embodiments of this application, a corresponding number of regions is pre-set for each sorting template in the sorting template library. The number of regions is used to indicate the optimal number of local images in the image to which the sorting template is adapted, with one region representing one local image.

[0120] In some embodiments of this application, for each candidate layout template, its region number matching score with the first image can be calculated according to the following formula (8):

[0121] In the above formula (8), This represents the score indicating the degree of matching between the candidate layout template and the number of regions in the first image. This indicates the number of regions corresponding to the candidate layout templates. This represents the total number of local images obtained from the first image segmentation.

[0122] The region number matching score between the candidate layout template and the first image reflects the degree of fit between the first image and the candidate layout template in terms of spatial division. The closer the region number matching score is to 1, the better the region number matches, and the higher the degree of fit between the two in terms of spatial division.

[0123] Step 12032. Based on the preset semantic category of each candidate layout template and the semantic category of each local image in the first image, determine the semantic category coverage score of the first image relative to each candidate layout template.

[0124] In this embodiment of the application, the preset semantic category of the candidate layout template is used to indicate the semantic category of the image content required by the candidate layout template. For example, for a layout template whose corresponding semantic scene is a city street scene, the required image content may be buildings and roads. Based on this, the preset semantic category of the layout template may include the two semantic categories of buildings and roads.

[0125] In some embodiments of this application, a sorting template library can be constructed, and designers can set the semantic category corresponding to each layout template based on expert experience, so that the preset semantic category of the selected layout template can be obtained from the sorting template library.

[0126] In some embodiments of this application, for each candidate layout template, the semantic category coverage score of the first image relative to the candidate layout template can be calculated according to the following formula (9):

[0127] In the above formula (9), This represents the semantic category coverage score of the first image relative to the candidate layout template. This represents the total number of semantic categories corresponding to the candidate layout templates. The indicator function represents the semantic category required for each candidate ranking template. In the semantic category set Includes this semantic category hour, The value is 1, otherwise, The value is 0. Among them, the semantic category set... It is a set consisting of the semantic categories of all local images of the first image.

[0128] The semantic category coverage score of the first image relative to the candidate typography template is used to measure the coverage of the semantic elements required by the candidate typography template in the first image.

[0129] Step 12033. For each candidate layout template, determine the comprehensive matching score between the candidate layout template and the first image based on the region quantity matching score and semantic category coverage score corresponding to the candidate layout template.

[0130] In some embodiments of this application, for each candidate layout template, after obtaining the region quantity matching score and semantic category coverage score corresponding to the candidate layout template, the comprehensive matching score between the candidate layout template and the first image can be calculated by weighted summation of the region quantity matching score and the semantic category coverage score. Specifically, for each candidate layout template, its comprehensive matching score with the first image can be calculated according to the following formula (10):

[0131] In the above formula (10), This represents the overall matching score between the candidate layout template and the first image. This indicates the weight corresponding to the matching score of the number of regions. This represents the weight corresponding to the semantic category coverage score. Wherein, The value range is 0 to 1, and the specific value can be determined based on expert experience. For example, No specific limitations are set for this.

[0132] Step 12034. Among all candidate layout templates, the candidate layout templates with a comprehensive matching score greater than the score threshold are determined as the target layout templates that semantically match the first image.

[0133] In some embodiments of this application, the scoring threshold is a threshold used to determine whether the candidate layout template is suitable for the first image. The scoring threshold can be set according to actual needs and expert experience; for example, the scoring threshold can be set to 0.85, and there is no specific limitation thereto.

[0134] For each candidate layout template, after obtaining the comprehensive matching score between the candidate layout template and the first image, the comprehensive matching score is compared with the score threshold. If the comprehensive matching score is greater than the score threshold, the candidate layout template is determined to be compatible with the first image, and thus the candidate layout template is determined as the target layout template that semantically matches the first image.

[0135] The above method first performs preliminary screening of the sorting templates in the sorting template library based on the mapping relationship, and then performs fine screening of the candidate layout templates obtained from the preliminary screening based on the comprehensive matching score. Compared with screening based solely on the mapping relationship, this screening method improves the semantic scene fit between the target layout template and the first image. Compared with screening based solely on the comprehensive matching score, it improves the matching efficiency and avoids the calculation of the comprehensive matching score of all templates.

[0136] In some embodiments of this application, to further improve matching efficiency, the features of the layout templates in the layout template library can be pre-calculated and indexed, and the core features of each sorting template, such as the "range of regions, set of semantic categories, and position distribution matrix", can be stored in advance, and a hash index table can be constructed. Based on this, before executing the above step 12031, the hash index can be used to quickly filter out the sorting templates that match the current first image in terms of "difference in the number of regions ≤ 1 and core semantic category". Then, the above steps 12031-12034 are executed on the templates selected from the candidate sorting templates. In this way, the traversal of all candidate sorting templates can be avoided, and the overall matching process time is controlled within 100ms, which meets the real-time interaction requirements of electronic devices.

[0137] Step 130. Perform layout processing on the local image based on the target layout template to obtain the second image after layout processing.

[0138] In some embodiments of this application, after obtaining the target layout template, a partial image of the first image is processed based on the target layout template to obtain a second image after layout processing. The layout processing may include, but is not limited to, operations such as position mapping, rotation, and scaling.

[0139] In some embodiments of this application, in order to adapt local images to the target sorting template and improve the artistic effect of the processed second image, when processing the local images based on the target sorting template, a sorting rule information set for the local images can be dynamically generated based on the target sorting template. The local images are then processed based on the sorting rule information set to obtain the second image. The sorting rule information set may include sorting rule information corresponding to each local image, and the sorting rule information set can be used to indicate the following information: (1) Position information of each local image on the output canvas. The output canvas is the canvas used to draw the second image; (2) Combinations of processing operation types for each local image. For example: immutable, rotated, cropped and / or scaled, etc.; (3) Calculation logic of operation parameters. For example, the rotation angle is determined by the direction of the semantic axis, and the cropping ratio is dynamically adjusted by the entropy value of the region saliency map.

[0140] In some embodiments of this application, a corresponding processing rule function library can be pre-bound to each layout template in the layout template library. The functions in the processing rule function library are used to dynamically calculate layout rule information such as the position, rotation angle, cropping ratio, and scaling ratio of the local image in the output canvas based on the semantic category and subjectivity score of the local image. The processing rule function library corresponding to each layout template can be set by the designer based on expert experience. Layout processing of local images based on the layout rule information determined by the processing rule function library ensures efficient mapping from the descriptive information of the local image to the final layout scheme, balancing semantic rationality and artistic expression, and driving the generation of a dynamic layout rule set.

[0141] For example, the functions in the processing rule function library corresponding to the three-part template can be used as follows: for local images of the semantic category "sky", directly determine its position in the canvas as the upper 30% area of ​​the canvas, rotate the angle to zero, and crop the ratio to 0.6 to retain more spatial information; for local images with a subject score greater than 0.8, determine its position in the canvas as the middle 40% area, rotate the angle randomly selected from -90 degrees or 90 degrees to increase dynamism, and scale the ratio to 1.2 to highlight the subject; for other local images, determine its position in the canvas as the lower 30% area of ​​the canvas, crop the ratio to 1.0 to maintain the original image ratio, and increase the scaling ratio to 1.8 to form a local close-up. The above processing rule functions can be described by the following mathematical expressions (11), (12), and (13):

[0142] Of the three formulas above, This indicates the position information of a local image on the canvas. Represents the semantic category of a local image. Indicates the height of the canvas. This represents the rotation angle of a local image. Indicates the scaling ratio.

[0143] Based on this, step 130 above may specifically include the following steps 1301-1304.

[0144] Step 1301. Based on the processing rule function library bound to the target layout template, generate layout rule information corresponding to each local image. The layout rule information includes the position coordinates of the local image in the output canvas and the processing strategy corresponding to the local image. The processing strategy is used to process the local image into a sub-image that fits the canvas.

[0145] In some embodiments of this application, the processing strategy corresponding to a local image may include at least one of the following processing strategies: immutable, rotated, cropped, or scaled.

[0146] According to the layout rules, different local images can be processed in a targeted manner. For example, the prominent parts of the background area can be retained, the main body area can be rotated and corrected, and the details of the decorative area can be magnified, so that the processing results are more consistent with the image content and the final layout result is more consistent with the semantics of the first image and more aesthetically pleasing.

[0147] Step 1302. For each local image, process the local image based on the processing strategy corresponding to the local image to obtain the sub-image corresponding to the local image.

[0148] In some embodiments of this application, when the processing strategy corresponding to the local image includes cropping the local image according to a cropping ratio, step 1302 may include: taking the centroid of the local image as the cropping center, determining the cropping bounding box coordinates according to the cropping ratio in the processing strategy and the coordinates of the cropping center, and then cropping the local image based on the cropping bounding box coordinates, taking the image region of the cropped bounding box as the initial cropping region, and determining the cropping region to be retained after cropping based on the initial cropping region.

[0149] In some embodiments of this application, when performing semantic segmentation on the first image, the image segmentation model can use the GrabCut algorithm to accurately extract the mask of the segmented region. This algorithm is based on iterative graph cut optimization, and continuously updates the probability distribution model of foreground and background pixels through at least five iterations to finely adjust the segmentation boundary, ensuring the accuracy and coherence of the region contour. The cropping operation uses the centroid coordinates in the local image as the geometric reference point, and calculates the cropping bounding box according to the cropping ratio specified in the layout rules. The specific boundary adjustment depends on the cropping ratio parameter. Specifically, the cropping bounding box coordinates can be calculated according to the following formula (14):

[0150] In the above formula (14), Indicates the coordinates of the clipping bounding box. Represents the centroid coordinates of a local image. Indicates the width of a local image. Indicates the length of a local image. Indicates the cutting ratio.

[0151] The core elements of a local image are usually located at the centroid of the local image. By cropping in the manner described above, we can ensure that the center of the cropped area is located at the centroid, thus preserving as many core elements of the local image as possible.

[0152] In some embodiments of this application, the initial cropping region can be directly determined as the cropping region of the local image.

[0153] In some embodiments of this application, to avoid the loss of important details in the local image, after determining the initial cropping region, the ratio of the number of feature points in the initial cropping region to the number of feature points in the local image can be further calculated; if the ratio is greater than or equal to a ratio threshold, the initial cropping region is determined as the cropping region of the local image; if the ratio is less than the ratio threshold, the peak position of the feature point density in the local image is determined as the cropping center, and the process returns to the step of determining the cropping bounding box coordinates according to the cropping ratio and the coordinates of the cropping center in the processing strategy, until a cropping region with a feature point ratio of less than the ratio threshold of the local image is obtained.

[0154] The ratio threshold is used to determine whether important details of the local image are missing in the initial cropped area. The specific value of the ratio threshold can be set based on expert experience; for example, the ratio threshold can be set to 0.7, without any specific limitation.

[0155] For each local image, after obtaining the initial cropping region of that local image, feature points are extracted from the initial cropping region, and the number of feature points in the initial cropping region is determined. The feature points of the local image are extracted to determine the number of feature points in the local image. The ratio of the two Compared with the ratio threshold, if If the ratio is greater than or equal to a threshold, it is determined that the initial cropping region contains important details of the local image, thus the initial cropping region is determined as the cropping region of the local image and no cropping adjustment is needed. However, if... If the value is less than the proportional threshold, it is determined that important details are lost, and the cropping center is switched to the peak position of feature point density to retain key visual information to the greatest extent.

[0156] In some embodiments of this application, feature points of the initial cropping region and local images can be extracted using the Scale Invariant Feature Transform (SIFT) algorithm.

[0157] By using the cropping method described above, key visual information of the local image can be preserved to the greatest extent.

[0158] If the processing strategy includes other processing steps when the cropped area of ​​the local image is obtained, other processing steps can be performed on the basis of the cropped area until all processing is completed, and the sub-image corresponding to the local image is obtained. If the processing strategy does not include other processing steps, the cropped area can be determined as the sub-image corresponding to the local image.

[0159] In some embodiments of this application, when the processing strategy corresponding to the local image includes rotating the local image according to the rotation angle, step 1302 above may include: determining the rotation axis based on the semantic category of the local image; rotating the local image based on the rotation angle and the rotation axis to obtain the rotated local image; and generating a sub-image corresponding to the local image based on the rotated local image.

[0160] The rotation operation sets the rotation axis according to semantic category. For example, in the human figure area, the vertical direction determined by the midpoint of the line connecting the two shoulders and the midpoint of the line connecting the hips is used as the rotation axis to ensure the natural posture and visual aesthetics of the human body's main axis.

[0161] Rotation angle The calculation formula is shown in equation (15):

[0162] In the above formula (15), and The coordinates of the two endpoints of the rotation axis are given respectively. The rotation matrix performs a geometric transformation of the local image based on the rotation angle, which can ensure the rationality of the semantics and structure of the image content after rotation.

[0163] Step 1303. Map the sub-images corresponding to each local image to the output canvas according to the position coordinates of each local image on the canvas to obtain the initial layout image.

[0164] In some embodiments of this application, a sub-image can be mapped to an output canvas through an affine transformation. Specifically, step 1303 above may include steps 13031-13033.

[0165] Step 13031. Convert each sub-image to an image format that supports alpha channels to obtain the format-converted sub-image corresponding to each sub-image.

[0166] In some embodiments of this application, each processed sub-image may be converted to RGBA format to support the alpha channel.

[0167] Step 13032. For each format-converted sub-image, determine the affine transformation matrix corresponding to the format-converted sub-image based on the position coordinates of the local image corresponding to the format-converted sub-image in the output canvas.

[0168] In some embodiments of this application, for each sub-image, based on its position coordinates in the output canvas determined from the layout information of its corresponding local image, the affine transformation matrix shown in equation (16) is calculated. :

[0169] In the above formula (16), the parameter These are parameters used to control rotation, scaling, and shearing transformations. This is represented as a translation offset.

[0170] Step 13033. Based on the affine transformation matrix corresponding to each format-converted sub-image, map each format-converted sub-image to obtain the output canvas, and obtain the initial layout image.

[0171] For each format-transformed sub-image, applying its corresponding affine transformation matrix to the sub-image's coordinates enables accurate mapping and positioning of the sub-image to the output canvas.

[0172] After all sub-images have been mapped to the output canvas, the overall image on the output canvas is determined as the initial layout image.

[0173] Step 1304. Generate a second image based on the initial layout image.

[0174] In some embodiments of this application, the initial sorting image can be directly determined as the second image.

[0175] In some embodiments of this application, in order to enhance the coherence of color and texture of the final obtained second image, step 1304 may include the following steps 13041-13042.

[0176] Step 13041. Using the Poisson fusion algorithm, a transition band is generated at the junction of every two adjacent sub-images in the initial layout image to obtain an edge fusion image.

[0177] In some embodiments of this application, edge blending between sub-images in the initial layout image can be achieved by using a Poisson fusion algorithm to generate a width of [missing value] at the junction of adjacent regions. The transition zone, in which, It can be set according to actual needs, for example , This is the length of the short side of the canvas. By ensuring that the blending bandwidth is proportional to the canvas size, visual differences can be smoothly transitioned.

[0178] Step 13042. The color information in the edge-blended image is fused using the Laplacian pyramid algorithm to obtain the second image.

[0179] In some embodiments of this application, the color information fusion of the initial layout image can be achieved using the Laplacian pyramid method, which decomposes each sub-image in the initial layout image into multiple layers of detail and low-frequency components, fuses each layer separately within the transition zone, and finally reconstructs the fused image, thereby enhancing the coherence of color and texture.

[0180] The above method can achieve seamless stitching of sub-images, improving the overall visual coherence of the second image.

[0181] In this embodiment, the first image is segmented according to its semantics to obtain at least two local images and descriptive information for each local image. Based on the descriptive information of the local images, a target layout template that semantically matches the first image is determined from a layout template library. Finally, the first image is processed for layout based on the local images and the target layout template to obtain a second image after layout processing. According to this embodiment, automatic image layout can be achieved, improving layout efficiency compared to manual layout. Furthermore, this embodiment determines the layout template based on the image's semantics, making the layout image semantically more consistent with the original image, thereby improving the logic and artistry of the layout result and enhancing the overall effect of image processing.

[0182] In some embodiments of this application, to reduce the computational burden on electronic devices, the process of optimizing the initial layout image based on the Poisson fusion and Laplacian pyramid methods can be optimized by combining the hardware characteristics of the electronic devices with algorithm simplification. Specifically, optimization can be performed using any one or more of the following three methods: (1) Hardware acceleration adaptation: For electronic devices that support GPUs, parallel computing of Poisson fusion can be achieved using OpenGLES3.2 or Vulkan interfaces. Steps such as Laplacian pyramid decomposition and transition band color fusion are encapsulated into GPU shaders. By utilizing the parallel pixel processing capabilities of GPUs, the fusion calculation speed can be increased by 5-8 times (e.g., the fusion time of an 800×1000 pixel canvas is reduced from 500ms to less than 80ms). For terminals without independent GPUs, CPU multi-threaded parallelism (based on OpenMP) is used to divide the transition band into 4-8 parallel blocks for processing, which improves efficiency by 2-3 times.

[0183] (2) Algorithm simplification strategy: While ensuring visual effects, the fusion algorithm is moderately simplified—the number of decomposition layers of the Laplacian pyramid is reduced from 5 to 3 (only the base layer, detail layer 1, and detail layer 2 are retained). At the same time, "approximate gradients are used to replace the exact Poisson equation solution," and the fusion weights are directly generated by calculating the color gradient difference between adjacent pixels, avoiding the complex matrix inversion operation at the terminal, reducing the computational load by about 45%. In addition, the transition band width is dynamically adjusted (when the short side of the canvas is <500 pixels, t), when the shorter side of the canvas is ≥500 pixels, To balance the fusion effect with the computational cost.

[0184] (3) Real-time guarantee mechanism: If the terminal computing resources are tight (e.g., memory <2GB), the “fast fusion mode” can be triggered - temporarily storing the edge feature maps of each sub-image. During fusion, only the edge pixels in the transition zone are precisely gradient corrected, and the non-edge areas are simply linearly mixed. Under the premise of ensuring no obvious splicing traces, the fusion time is further compressed to within 50ms to meet the requirements of real-time preview and export.

[0185] In some embodiments of this application, in order to adapt to the limited memory space and computing resources of electronic devices, the image segmentation model can be optimized for terminal use. Specifically, the image segmentation model can be optimized for terminal use through one or more of the following three methods: (1) Model pruning optimization: For redundant channels and weights in the segmentation branch and relation parsing branch of the image segmentation model, a structured pruning strategy is adopted to remove convolutional channels with absolute weight values ​​less than a threshold (such as 1e-4). After pruning, the number of model parameters is reduced by about 50% and the amount of computation is reduced by 40%. At the same time, the accuracy loss is restored by fine-tuning to ensure that the decrease in semantic segmentation mIoU does not exceed 2%.

[0186] (2) Quantization compression processing: The weights of the image segmentation model are quantized from 32-bit floating-point number FP32 to 8-bit integer INT8. An asymmetric quantization strategy is used to dynamically calibrate the activation value range. The inference speed on the CPU of electronic devices is increased by 3-4 times, the memory usage is reduced by 75%, and the region semantic recognition accuracy is guaranteed to be less than 1.5% of the original model through quantization-aware training (QAT).

[0187] (3) Knowledge distillation adaptation: The original ResNet-101 backbone network model is used as the "teacher model" and the lightweight MobileNetV3-Large is used as the "student model". The semantic discrimination ability of the teacher model is transferred to the student model through distillation loss (including classification loss and feature distillation loss). The student model is then deployed as an image segmentation model in electronic devices. The final number of parameters in the student model is only 1 / 8 of that in the original model, and the single inference time of the electronic device can be controlled within 200ms, which meets the real-time processing requirements.

[0188] In some embodiments of this application, the processing rule function library bound to the typesetting template can be dynamically updated. For example, it can learn from user feedback by recording the adjustment parameters made by users to the automatic typesetting results. Examples of such adjustments include position offset, rotation angle change, and scaling correction. Cosine similarity is used to measure the similarity of adjustments made by different users in the same scene. When the cumulative number of adjustments reaches a preset number (e.g., 5 times) and the similarity exceeds a preset similarity (e.g., 80% similarity), the adjustment parameter can be automatically used as a candidate for a new rule version in the processing rule function library to enhance the adaptability of the processing rule function library to user preferences.

[0189] The cosine similarity function shown in formula (17) can be used to measure the similarity between different users in the same scene:

[0190] In the above formula (17), and Let be the adjustment parameter vectors for different adjustment instances, where ‖·‖ represents the vector norm, used to quantitatively determine the similarity of the adjustment parameters.

[0191] In some embodiments of this application, for layout templates with a matching degree within a preset matching degree range (e.g., [0.7, 0.85]), a cross-template fusion strategy can be used to merge the processing rule functions of different layout templates to form a new processing rule function. For example, the golden section point constraint in a center-radial template can be combined with the asymmetric scaling strategy of a free montage template to form a new composite rule to enhance the diversity and expressiveness of layout.

[0192] The following is an example of... Figure 4 Taking the layout processing of the grassland portrait photo 400 shown as an example, we will illustrate the flow steps S1-S3 of the image processing method provided in this application embodiment.

[0193] Step S1: Image semantic segmentation and relation parsing.

[0194] Semantic segmentation: The original image of the grassland portrait is input into a pre-trained deep neural network (segmentation branch based on DeepLabv3+, ResNet-101 backbone network), and three core regions are segmented through pixel-level recognition: Region 001: category = "Human Trunk" (main body), bbox = [120, 205, 380, 720], centroid = [250, 462] (centroid coordinates); Region 002: category="grassland vegetation" (background), bbox=[0,500,800,800], centroid=[400,650]; Area 003: category="sky" (background), bbox=[0,0,800,500], centroid=[400,250].

[0195] Relationship analysis: Analyzing region relationships using graph attention networks. Spatial topology: Region 001 (human body) is adjacent to 002 (grassland) (shortest distance between boundaries < 10), and adjacent to 003 (sky); Region 002 is adjacent to 003 (intersection over union ratio > 0.05).

[0196] Semantic association: Region 001 (human body) has a high association strength with 002 (grassland) (subject and background) and a medium association strength with 003 (sky). The subjectivity score of region 001 is 0.88 (high), while the subjectivity scores of 002 and 003 are <0.5 (background).

[0197] Generate structured data: Record the category, coordinates, adjacency relationship and association strength of the above regions in JSON format.

[0198] Step S2: Dynamic layout rules are generated.

[0199] Multi-dimensional matching: The art layout knowledge base is invoked to match the semantic scene of "grassland portrait" and determine that region 001 is the subject and regions 002 and 003 are the background.

[0200] Matching template rule set: Location: Area 003 (Sky) → 30% of the top of the canvas; Area 001 (Human Body) → 40% of the middle of the canvas; Area 002 (Grassland) → 30% of the bottom of the canvas.

[0201] Processing steps: Region 003 (Sky): Cropped, the scale is adjusted by the saliency map entropy value, retaining areas with dense clouds; Area 001 (Human Body): Rotation correction, using the line connecting the two shoulders as the axis, align the main axis with the horizontal direction of the canvas; Region 002 (Grassland): Cropping, retaining flower regions with entropy values ​​greater than the threshold T.

[0202] Step S3: Adaptive region processing and canvas composition.

[0203] Region processing includes the following operations: Sky: Cropped according to rules, retaining cloud sub-regions with high entropy values ​​and removing blurred edges; Human body: The angle is adjusted by rotating the matrix to make the torso aligned with the horizontal direction of the canvas, resulting in a more natural posture; Grassland: Trim and retain areas with dense flowers, removing monotonous grass sections.

[0204] Canvas compositing includes the following operations: The processed sub-image is mapped to the target canvas according to the coordinates of the rules, with a size of 800×1000 pixels.

[0205] Edge blending: A transition band with a width of 0.05×800=40 pixels is generated at the boundary between the sky and the grassland, and the colors are blended using the Laplacian pyramid; the edge direction is detected by the Sobel operator to ensure that the edge deviation between adjacent areas is <15° and there are no obvious splicing marks.

[0206] Output result: The final result is as follows Figure 5 The image shown is a stitched image of 500. The top of the image is a cropped sky highlighting the clouds, the middle is the main human figure, and the bottom is an enlarged grassland of flowers. The whole image is seamlessly integrated, which conforms to the artistic layout logic of grassland portraits.

[0207] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0208] See Figure 6 The image processing apparatus 600 provided in some embodiments of this application may include the following modules: The image segmentation module 601 is used to segment a first image into at least two local images with independent semantics using an image segmentation model, and to determine the descriptive information of each local image. The template matching module 602 is used to determine a target layout template that semantically matches the first image from a preset layout template library based on the description information of each local image. The layout processing module 603 is used to perform layout processing on a local image based on a target layout template to obtain a second image after layout processing.

[0209] In this embodiment, the first image is segmented according to its semantics to obtain at least two local images and descriptive information for each local image. Based on the descriptive information of the local images, a target layout template that semantically matches the first image is determined from a layout template library. Finally, the first image is processed for layout based on the local images and the target layout template to obtain a second image after layout processing. According to this embodiment, automatic image layout can be achieved, which improves layout efficiency compared to manual layout. Moreover, in this embodiment, the layout template is determined based on the semantics of the image, making the layout image more semantically consistent with the original image, thereby improving the logic and artistry of the layout result.

[0210] In some embodiments of this application, the image segmentation model includes a segmentation module and a relation parsing module; Image segmentation module 601 includes: The segmentation submodule is used to perform semantic segmentation on the first image through the segmentation module to obtain at least two local images with independent semantics and the region identifier, semantic category and location information of each local image; The relation parsing submodule is used to determine the region association information of each local image through the relation parsing module. The region association information is used to indicate the region association relationship between the local image and other local images. The output submodule is used to determine the descriptive information of each local image by identifying its region identifier, semantic category, location information, and association information with other local images.

[0211] In some embodiments of this application, the location information of the local image includes the coordinates of the minimum bounding rectangle of the local image in the first image and the centroid coordinates of the minimum bounding rectangle.

[0212] In some embodiments of this application, the region association information includes the association score between a local image and its neighboring local images, and the association score between the local image and its neighboring local images is used to indicate the association strength between the local image and its corresponding neighboring local image; The relation resolution submodule is specifically used for: Determine the adjacent local images for each local image; For each local image, calculate the spatial distance and feature similarity between the local image and its neighboring local images; Based on spatial distance and feature similarity, the association score between neighboring local images of a local image is determined.

[0213] In some embodiments of this application, the template matching module 602 is specifically used for: Based on the descriptive information of each local image, the semantic scene of the first image is determined; Based on the pre-defined mapping relationship between semantic scenarios and layout templates, at least one candidate layout template that matches the semantic scenario is determined. Based on at least one candidate layout template, a target layout template that semantically matches the first image is determined.

[0214] In some embodiments of this application, the typesetting processing module 603 is specifically used for: Based on the processing rule function library bound to the target layout template, layout rule information corresponding to each local image is generated. The layout rule information includes the position coordinates of the local image in the output canvas and the processing strategy corresponding to the local image. The processing strategy is used to process the local image into a sub-image that fits the canvas. For each local image, the local image is processed based on the processing strategy corresponding to the local image to obtain the sub-image corresponding to the local image; The sub-images corresponding to each local image are mapped to the output canvas according to the position coordinates of each local image in the output canvas to obtain the initial layout image; A second image is generated based on the initial layout image.

[0215] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0216] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0217] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0218] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0219] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0220] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0221] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0222] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0223] The processor 810 is used to segment the first image into at least two local images with independent semantics using an image segmentation model, and to determine the descriptive information of each local image. The processor 810 is also used to determine a target layout template that semantically matches the first image from a preset layout template library based on the description information of each local image. The processor 810 is also used to perform layout processing on a local image based on a target layout template to obtain a second image after layout processing.

[0224] In this embodiment, the first image is segmented according to its semantics to obtain at least two local images and descriptive information for each local image. Based on the descriptive information of the local images, a target layout template that semantically matches the first image is determined from a layout template library. Finally, the first image is processed for layout based on the local images and the target layout template to obtain a second image after layout processing. According to this embodiment, automatic image layout can be achieved, which improves layout efficiency compared to manual layout. Moreover, in this embodiment, the layout template is determined based on the semantics of the image, making the layout image more semantically consistent with the original image, thereby improving the logic and artistry of the layout result.

[0225] In some embodiments of this application, the image segmentation model includes a segmentation module and a relation parsing module; Processor 810, specifically used for: The segmentation module performs semantic segmentation on the first image to obtain at least two local images with independent semantics and the region identifier, semantic category and location information of each local image; The relation parsing module determines the region association information of each local image, which is used to indicate the region association relationship between the local image and other local images. The output submodule is used to determine the descriptive information of each local image by identifying its region identifier, semantic category, location information, and association information with other local images.

[0226] In some embodiments of this application, the location information of the local image includes the coordinates of the minimum bounding rectangle of the local image in the first image and the centroid coordinates of the minimum bounding rectangle.

[0227] In some embodiments of this application, the region association information includes the association score between a local image and its neighboring local images, and the association score between the local image and its neighboring local images is used to indicate the association strength between the local image and its corresponding neighboring local image; Processor 810, specifically used for: Determine the adjacent local images for each local image; For each local image, calculate the spatial distance and feature similarity between the local image and its neighboring local images; Based on spatial distance and feature similarity, the association score between neighboring local images of a local image is determined.

[0228] In some embodiments of this application, the processor 810 is specifically used for: Based on the descriptive information of each local image, the semantic scene of the first image is determined; Based on the pre-defined mapping relationship between semantic scenarios and layout templates, at least one candidate layout template that matches the semantic scenario is determined. Based on at least one candidate layout template, a target layout template that semantically matches the first image is determined.

[0229] In some embodiments of this application, the processor 810 is specifically used for: Based on the processing rule function library bound to the target layout template, layout rule information corresponding to each local image is generated. The layout rule information includes the position coordinates of the local image in the output canvas and the processing strategy corresponding to the local image. The processing strategy is used to process the local image into a sub-image that fits the canvas. For each local image, the local image is processed based on the processing strategy corresponding to the local image to obtain the sub-image corresponding to the local image; The sub-images corresponding to each local image are mapped to the output canvas according to the position coordinates of each local image in the output canvas to obtain the initial layout image; A second image is generated based on the initial layout image.

[0230] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0231] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0232] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0233] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0234] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0235] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0236] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0237] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0240] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized by, The method comprises: segmenting the first image into at least two local images with independent semantics by using an image segmentation model, and determining description information of each local image; determining a target layout template matching the semantics of the first image from a preset layout template library based on the description information of each local image; performing layout processing on the local images based on the target layout template to obtain a second image after layout processing.

2. The method of claim 1, wherein, The image segmentation model comprises a segmentation module and a relationship analysis module; The segmentation of the first image into at least two local images with independent semantics by using the image segmentation model and the determination of the description information of each local image comprise: performing semantic segmentation on the first image by using the segmentation module to obtain at least two local images with independent semantics and region identification, semantic category and position information of each local image; determining region association information of each local image by using the relationship analysis module, the region association information being used to indicate the region association relationship between the local image and other local images; for each local image, the region identification, semantic category, position information and association information with other local images of the local image are determined as the description information of the local image.

3. The method of claim 2, wherein, The position information of the local image comprises the coordinates of the minimum bounding rectangle of the local image in the first image and the centroid coordinates of the minimum bounding rectangle.

4. The method of claim 2, wherein, The region association information comprises an association score between the local image and an adjacent local image, the association score between the local image and the adjacent local image being used to indicate the association strength between the local image and the corresponding adjacent local image. The determination of the region association information of each local image comprises: determining the adjacent local images of each local image respectively; for each local image, calculating the spatial distance and feature similarity between the local image and the adjacent local images of the local image; determining the association score between the adjacent local images of the local image and the local image based on the spatial distance and the feature similarity.

5. The method of claim 1, wherein, The determination of the target layout template matching the semantics of the first image from the preset layout template library based on the description information of all local images comprises: determining a semantic scene of the first image based on the description information of each local image; determining at least one candidate layout template matching the semantic scene based on a preset mapping relationship between semantic scenes and layout templates; determining the target layout template matching the semantics of the first image based on the at least one candidate layout template.

6. The method of claim 1, wherein, The layout processing of the local images based on the target layout template to obtain the second image after layout processing comprises: generating layout rule information corresponding to each local image based on a processing rule function library bound to the target layout template, the layout rule information comprising position coordinates of the local image in an output canvas and a processing strategy corresponding to the local image, the processing strategy being used to process the local image into a sub-image adapted to the canvas; For each local image, processing the local image based on a processing strategy corresponding to the local image to obtain a sub-image corresponding to the local image; Mapping the sub-image corresponding to each local image to the output canvas according to the position coordinates of each local image in the output canvas to obtain an initial layout image; Generating a second image based on the initial layout image.

7. An image processing apparatus characterized by comprising: Comprise: An image segmentation module configured to segment a first image into at least two local images with independent semantics by using an image segmentation model, and determine description information of each local image; A template matching module configured to determine a target layout template matching the semantics of the first image from a preset layout template library based on the description information of each local image; A layout processing module configured to perform layout processing on the local images based on the target layout template to obtain a second image after layout processing.

8. The apparatus of claim 7, wherein, The image segmentation model comprises a segmentation module and a relationship analysis module; The image segmentation module comprises: A segmentation sub-module configured to perform semantic segmentation on the first image by using the segmentation module to obtain at least two local images with independent semantics, and region identification, semantic category and position information of each local image; A relationship analysis sub-module configured to determine region association information of each local image by using the relationship analysis module, wherein the region association information is used to indicate the region association relationship between the local image and other local images; An output sub-module configured to determine, for each local image, the region identification, semantic category, position information and association information with other local images of the local image as the description information of the local image.

9. The apparatus of claim 8, wherein, The position information of the local image comprises the coordinates of the minimum bounding rectangle of the local image in the first image and the centroid coordinates of the minimum bounding rectangle.

10. The apparatus of claim 8, wherein, The region association information comprises an association score between the local image and an adjacent local image, and the association score between the local image and the adjacent local image is used to indicate the association strength between the local image and the corresponding adjacent local image. The relationship analysis sub-module is specifically configured to: Determine the adjacent local images of each local image respectively; For each local image, calculate the spatial distance and feature similarity between the local image and the adjacent local images of the local image; Determine the association score between the adjacent local images of the local image and the local image based on the spatial distance and the feature similarity.

11. The apparatus of claim 7, wherein, The template matching module is specifically configured to: Determine the semantic scene of the first image based on the description information of each local image; Determine at least one candidate layout template matching the semantic scene based on a preset mapping relationship between the semantic scene and the layout template; Determine the target layout template matching the semantics of the first image based on the at least one candidate layout template.

12. The apparatus of claim 7, wherein, The layout processing module is specifically configured to: generate, based on the target layout template binding processing rule function library, layout rule information corresponding to each local image, the layout rule information including position coordinates of the local image in an output canvas and a processing strategy corresponding to the local image, the processing strategy being used to process the local image into a sub-image adapted to the canvas; for each local image, process the local image based on the processing strategy corresponding to the local image to obtain a sub-image corresponding to the local image; map the sub-image corresponding to each local image to the output canvas according to the position coordinates of each local image in the output canvas to obtain an initial layout image; generate a second image based on the initial layout image.

Images