Method and apparatus for generating image mosaic

Abstract

The application provides a method and device for generating an image mosaic, which can be used in the financial field or other fields. The method comprises: adaptively segmenting an input image to obtain a first image having a plurality of blocks with different sizes; performing feature description on the first image to establish a feature vector of the first image; retrieving matching feature vectors from a database for the plurality of blocks in the first image according to the feature vector of the first image; and filling the plurality of blocks in the first image with the retrieved matching images corresponding to the feature vectors to generate a second image with a mosaic. According to the embodiment of the application, the generated mosaic image has a unique artistic style while the features of the source image are retained to the greatest extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a method and device for generating an image mosaic. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application recited in the claims. The description herein does not constitute admission that the information provided herein is prior art to the present application.

[0003] In the field of computer vision, a mosaic is a new image composed of small images called "tiles" that are pieced together to recreate the visual pattern of the original image. Image mosaics have been an important research topic in the field of computer vision, which often refers to dividing an input image into tiles, and then retrieving the most similar images from a database or network to replace the tiles to produce an impressive mosaic effect.

[0004] Most of the current image mosaic generation techniques mostly use a single image segmentation mechanism and a single color feature metric. Therefore, it is difficult to maximize the preservation of the features of the input image in the mosaic image. In addition, since a large number of images need to be retrieved in the database to find the most matching images to fill the original image, it will consume a considerable amount of time, so most of the current image mosaic generation techniques are difficult to achieve a good balance between image retrieval time and splicing effect. SUMMARY

[0005] Embodiments of the present application provide a method and device for generating an image mosaic, which divides an input image into a first image having multiple blocks of different sizes, and then retrieves matching images from a database according to the feature vectors of the first image to fill them, thereby maximizing the preservation of the features of the source image while making the generated mosaic image have a unique artistic style.

[0006] According to a first aspect of embodiments of the present application, a method for generating an image mosaic is provided, wherein the method for generating an image mosaic comprises: adaptively dividing an input image to obtain a first image having multiple blocks of at least different sizes; performing feature description on the first image to establish a feature vector of the first image; retrieving matching feature vectors from a database for the multiple blocks in the first image according to the feature vector of the first image; and filling the multiple blocks in the first image with matching images corresponding to the feature vectors retrieved to generate a second image having a mosaic.

[0007] According to a second aspect of embodiments of the present application, a device for generating an image mosaic is provided, wherein the device for generating an image mosaic comprises:

[0008] an image segmentation unit, which adaptively segments an input image to obtain a first image having a plurality of blocks with at least different sizes;

[0009] a feature description unit, which performs feature description on the first image to establish a feature vector of the first image;

[0010] a retrieval unit, which retrieves, according to the feature vector of the first image, matched feature vectors from a database for the plurality of blocks in the first image respectively;

[0011] a filling unit, which fills the plurality of blocks in the first image with matched images corresponding to the feature vectors retrieved to generate a second image having a mosaic.

[0012] According to a third aspect of the embodiments of the present application, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image mosaic generation method as described above when executing the computer program.

[0013] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program for executing the image mosaic generation method as described above.

[0014] An advantage of the present application is that the generated mosaic image has a unique artistic style while the features of the source image are retained to the greatest extent.

[0015] Specific embodiments of the present application are disclosed herein, and represented in the accompanying drawings, illustrating the principles of the present application in a manner that those skilled in the relevant art can apply its principles. It should be understood that the embodiments of the present application are not limited in scope to the specific embodiments described. Embodiments of the present application encompass many changes, modifications, and equivalents within the spirit and scope of the appended claims as well as embodiments applicable to a variety of applications.

[0016] Features described and / or illustrated with respect to one implementation can be used in the same or similar manner in one or more other implementations, in combination with or in place of features in other implementations.

[0017] It should be emphasized that the term "comprises / comprising" when used in this text is taken to mean the presence of stated features, integers, operations or components but not to the exclusion of one or more other features, integers, operations or components, that are stated or implied. BRIEF DESCRIPTION OF DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a schematic diagram of an image mosaic generation method according to an embodiment of this application;

[0020] Figure 2 This is a diagram of a quadtree data structure according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a quadtree image segmentation method according to an embodiment of this application;

[0022] Figure 4 This is a diagram of the quadtree image segmentation structure and result according to an embodiment of this application;

[0023] Figure 5 This is a diagram of an image similarity retrieval method according to an embodiment of this application;

[0024] Figure 6 This is an example diagram of the quadtree image partitioning result according to an embodiment of this application;

[0025] Figure 7 This is an example diagram illustrating the changing parameters in an embodiment of this application;

[0026] Figure 8 This is another example diagram of the quadtree image segmentation result according to an embodiment of this application;

[0027] Figure 9 This is an example image of a mosaic image generated according to an embodiment of this application;

[0028] Figure 10 This is an example image that uses a traditional mosaic image based on the RGB color space;

[0029] Figure 11 This is a diagram of an image mosaic generation apparatus according to an embodiment of this application. Detailed Implementation

[0030] It should be noted that the image mosaic generation method and apparatus disclosed in this application can be used in the field of financial technology, or in any field other than financial technology. This application does not limit the application field of the disclosed image mosaic generation method and apparatus.

[0031] The specific implementation of the image mosaic generation method and apparatus of this application will be described below with reference to the accompanying drawings and embodiments. Embodiments of this application provide a method for generating image mosaics.

[0032] Figure 1 This is a schematic diagram of an image mosaic generation method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0033] Step 101: Adaptively segment the input image to obtain a first image with multiple blocks of at least different sizes;

[0034] Step 102: Perform feature description on the first image and establish the feature vector of the first image;

[0035] Step 103: Based on the feature vector of the first image, retrieve matching feature vectors from the database for each of the multiple blocks in the first image;

[0036] Step 104: Use the retrieved matching image corresponding to the feature vector to fill multiple blocks in the first image to generate a second image with mosaic.

[0037] Therefore, by adaptively segmenting the image and retrieving it based on the feature vector, the generated mosaic image can retain the features of the source image to the greatest extent while also giving it a unique artistic style.

[0038] In some embodiments, a top-down quadtree image segmentation algorithm is used to achieve adaptive segmentation, dividing the input image into image blocks of different sizes. That is, adaptive segmentation of the input image is used to obtain a first image with multiple blocks of at least different sizes. Furthermore, the segmented first image includes uniform and non-uniform regions; wherein the blocks in the uniform regions are of uniform size, and the blocks in the non-uniform regions are of inconsistent size.

[0039] Quadtrees are an efficient image processing and data structure. Quadtree partitioning algorithms divide images into uniform and non-uniform regions. They require relatively fewer segmentations in uniform regions and more segmentations in regions containing additional details and important features.

[0040] For example, in portraits, facial feature regions are segmented more often than background regions. This implies that facial feature regions are more valuable and important than background regions. Therefore, quadtree technology segments the input image into first images of different sizes, where the smaller first image more accurately represents the detailed features of the input image. Consequently, quadtree-based image segmentation techniques are faster and can preserve the detailed features of the image.

[0041] Quadtree segmentation algorithms are based on quadtree data structures, which are an efficient method for processing two-dimensional images. Because in two-dimensional space, planar pixels can be repeatedly divided into four parts, the depth of the tree is determined by the complexity of the image, computer memory, and graphics. A quadtree structure is defined as having four child nodes under each node. It typically subdivides a portion of the two-dimensional space into four quadrants or regions, storing relevant information within each region in the quadtree nodes. Regions can be squares, rectangles, or any shape.

[0042] Figure 2 This is a diagram of a quadtree data structure according to an embodiment of this application, such as... Figure 2 As shown, the quadtree structure has a parent node 21, child nodes 210, and leaf nodes 2100. Parent node 21 represents the source image, i.e., the input image. The four child nodes 211-214 of parent node 21 represent the four quadrants of the source image, respectively. Leaf node 2100 represents the homogeneous quadrant (where the difference between pixels within this quadrant is less than a certain threshold), and the intermediate nodes represent the heterogeneous quadrant (where the difference between pixels within this quadrant is greater than a threshold). Furthermore, Figure 2 Each layer in the image represents a QT layer, and the number of QT layers is directly proportional to the feature complexity of the image: the more features an image has, the more QT layers it has, and vice versa.

[0043] In some embodiments, a top-down approach is used to implement the quadtree data structure. The top-down construction method first determines whether a single leaf node can describe the entire region or whether it can be divided into four child nodes. This operation is then repeated, sequentially determining each region and checking whether it needs further partitioning. The specific steps are as follows:

[0044] a. The region that needs to be divided is region R.

[0045] b. Select an image descriptive feature as the basis for segmentation. This technique uses RGB variance and image entropy, and sets a threshold and conditional function P.

[0046] c. If P(R) = FALSE, the image region is divided into four quadrants, and so on, until all regions in the image satisfy the consistency criterion.

[0047] Finally, the result of quadtree partitioning may contain blocks of various sizes.

[0048] In some embodiments, in the quadtree image segmentation algorithm, the decision to stop quadrant segmentation is made by specifying the maximum number of iterations D and the minimum block size S; wherein the maximum number of iterations D is used to limit the maximum depth of segmentation, and the minimum block size S is used to allow the width and height of the minimum quadrant.

[0049] For example, the segmentation algorithm in this embodiment no longer relies entirely on the settings of RGB variance and image entropy thresholds, but instead determines whether quadrant segmentation stops by specifying the maximum number of iterations D and the minimum block size S. For an input image, the segmentation is determined by the maximum number of iterations D and the minimum block size S. For a color-rich region, the segmentation is dense because the quadrant with the largest RGB variance value is always within that region. However, in some overly colorful image regions, if the maximum number of iterations D and the minimum quadrant size S are not set, it may lead to overly dense segmentation, which would be meaningless to the naked eye when viewed as a mosaic.

[0050] Therefore, a maximum iteration count parameter D is introduced to limit the maximum depth of partitioning, preventing partitions in certain areas of the image from becoming too dense; a minimum quadrant size parameter S is introduced to prevent quadrant partitions from being too small, which would affect the image mosaic result. Specifically, in each iteration, the quadrant with the largest RGB variance and image entropy is selected, and it is divided into four smaller quadrants, with the RGB variance and image entropy of the existing quadrants being dynamically updated continuously. When the number of iterations reaches the upper limit, or the size of the square reaches the minimum value, the partitioning stops, thus achieving the optimal partitioning of the quadtree image.

[0051] Figure 3 This is a schematic diagram of a quadtree image segmentation method according to an embodiment of this application.

[0052] like Figure 3 As shown, the quadtree image segmentation method includes:

[0053] Step 3011: Set the maximum number of iterations D and the minimum quadrant size S.

[0054] For example, setting the maximum number of iterations D for a quadtree limits the maximum depth of a quadtree partition, and setting the minimum quadrant size S represents the minimum allowed quadrant width and height.

[0055] Step 3012: Divide the image into quadrants with the highest RGB variance and image entropy.

[0056] For example, the source image is constructed as the root quadrant, its RGB variance and image entropy are calculated, and added to a heap data structure. After each iteration, the quadrant with the largest RGB variance and image entropy in the heap data structure is selected to continue partitioning.

[0057] Step 3013: Determine whether the number of iterations has reached the upper limit D or whether the size of the first image has reached the minimum value S. If the result is yes, proceed to step 3014; if the result is no, return to 3012 and continue dividing the quadrants.

[0058] Step 3014: End image segmentation.

[0059] For example, when the height or width of a quadrant reaches S, the quadrant will no longer be further subdivided. Furthermore, this quadtree algorithm recursively segments the image until the RGB variance and image entropy within all image quadrants cease to update, or the number of iterations corresponding to that block reaches the maximum number of iterations D.

[0060] In the quadtree segmentation algorithm of this application embodiment, the quadrant segmentation is stopped by specifying the maximum number of iterations D and the minimum block size S; wherein the maximum number of iterations D is used to limit the maximum depth of segmentation, and the minimum block size S is used to allow the width and height of the minimum quadrant.

[0061] Figure 4 This is a diagram of the quadtree image segmentation structure and result of Example 1. The left side (a) shows the segmentation result, and the right side (b) shows the segmentation structure. For example, as... Figure 4 As shown in the segmentation result on the left (a), the input image is adaptively segmented using a top-down quadtree image segmentation algorithm, and the first segmented image includes a uniform region 41 and a non-uniform region 42; wherein, the blocks in the uniform region 41 are of the same size, and the blocks in the non-uniform region 42 are of different sizes.

[0062] The above provides an illustrative explanation of adaptive segmentation. In this application, a quadtree segmentation algorithm is used to segment the image into blocks of different sizes. As a result, the image blocks of the feature regions of the input image are smaller and denser, which helps to preserve the detailed features of the input image, while the image blocks of the non-feature regions of the input image are larger, maintaining the clarity of the filled image, thereby providing users with more useful visual information and presenting different styles.

[0063] The following is an illustrative description of the features of the images in the embodiments of this application.

[0064] In step 102, the first image is characterized to establish a feature vector.

[0065] In some embodiments, an HSV-based image feature descriptor is used. HSV can accurately describe the color features of an image, and the HSV color space (hue, saturation, and brightness values) is consistent with human visual perception, making it a suitable model for the human eye to distinguish between the two. However, HSV cannot reflect the roughness of an image. If only HSV color components are considered, it is possible to obtain images with similar average color values ​​but different roughness, which will seriously affect the visual effect of the image stitching result.

[0066] Therefore, this application introduces image information entropy as a texture feature of the image, adds information entropy as a roughness factor to the image descriptor to describe the image, and combines the image HSV color and information entropy to obtain a balanced description of color and roughness.

[0067] In some embodiments, the feature vector of the first image includes a color feature vector and a texture feature vector; wherein, the HSV color space is used to describe the color features of the first image, and information entropy is used to describe the texture features of the first image. This allows for the acquisition of stable and reliable image features.

[0068] For example, although calculations based on the RGB color space are simple, the RGB color space differs from human visual perception, leading to a significant difference between the resulting image mosaic and the colors of the input image. Therefore, embodiments of this application employ HSV-based image feature descriptors to explore the differences from RGB color feature descriptors.

[0069] Existing research has shown that the HSV color space (hue, saturation, and brightness values) is consistent with human visual perception and is a suitable model for distinguishing between the two. However, Python reads color images in RGB mode, therefore, it is necessary to convert the RGB space to the HSV space.

[0070] Specifically, we can first obtain the average RGB values ​​of the image, and then convert them to HSV values ​​using functions provided by the PIL library and OpenCV. This transforms the RGB vector (r, g, b) into the HSV vector (h, s, v).

[0071] Furthermore, while HSV is accurate in describing the color features of an image, it does not reflect the image's roughness. For example, consider an input image with a very smooth tile pattern. When searching for the best-matching tile from the dataset, if only HSV color components are considered, it is likely to result in images with similar average color values ​​but different roughness, which will severely affect the visual quality of the image stitching result.

[0072] Therefore, this application introduces image information entropy as a new dimension of image feature vector, adds information entropy as a roughness factor to the image descriptor to describe the image, and combines the image HSV color and information entropy to obtain a balanced description of color and roughness.

[0073] For example, a 28-dimensional image descriptor based on HSV and image entropy can be obtained. The first 27 dimensions are color feature components, and the last dimension is the entropy component. The PIL library and OpenCV do not directly extract image entropy values, but another Python library called Skimage provides a `shannon entropy()` function to calculate image entropy. Therefore, you can use the `cv2.imread()` function provided by OpenCV to read the image first, and then use the first argument of the `entropy()` function, which returns the image entropy.

[0074] The following is an illustrative description of image similarity retrieval in the embodiments of this application.

[0075] In step 103, based on the feature vector of the first image, matching feature vectors are retrieved from the database for each of the multiple blocks in the first image.

[0076] The feature vector matching process is equivalent to finding the feature point that is closest to the input feature point among a set of feature points. For image mosaic, the feature vector of each square in the input image is matched with all feature vectors in the image database based on similarity, and the feature with the shortest distance is the matching result.

[0077] Furthermore, image databases contain thousands of high-quality images, resulting in thousands of feature vectors after their features are described. Therefore, finding a suitable data indexing structure to store these feature vectors and matching vectors is crucial to significantly reduce the time required. The matching process of feature description vectors is equivalent to finding the feature point that is closest to the input feature point among a set of feature points. Two common methods are:

[0078] A. Linear search: This method compares a feature description vector with each vector in the database. It is simple but inefficient, with a time complexity of O(N), where N is the number of vectors in the database.

[0079] B. Create a data index and then match the feature description vectors. This mosaic generation technique uses a KD-tree data structure to store feature vectors and search images. Based on this technique, the time complexity of image search is significantly reduced, more data is indexed under the same memory constraints, and more importantly, matching accuracy and coverage are significantly improved.

[0080] KD-trees are k-dimensional indexed tree data structures commonly used for nearest neighbor search in large-scale, high-dimensional data spaces, such as k-nearest neighbor search and matching of high-dimensional image feature vectors in image retrieval and recognition. K-dimensional data is stored in the tree. The root node divides the space into two subspaces, and child nodes further divide the space into smaller parts. The partitioning of a child node does not cross the partitioning of its parent node. The top-level node is partitioned into one dimension, the next-level node into another dimension, and so on, until the amount of data within a node is less than a set upper limit threshold. The search time complexity of a KD-tree is O(log₂N).

[0081] In some embodiments, the database uses a KD-tree structure to store feature vectors; wherein the leaf nodes of the KD-tree structure store data, and the non-leaf nodes store the current division dimension, as well as the decomposition values ​​of the current dimension and the left and right nodes of the current node.

[0082] Once all images in the dataset have their feature vectors, they are constructed into a KD-tree structure to facilitate searching for the best-matching image for each square. Furthermore, this process is performed only once in the system because it does not depend on the source images. A KD-tree is a data structure that partitions data points in k-dimensional space. Essentially, a KD-tree is a complete binary tree, except that data points are stored only in leaf nodes, while non-leaf nodes store only the current division dimension and the decomposition values ​​of the current dimension and its left and right neighbors.

[0083] This application provides a dataset in which the feature vectors of all images are used to construct the KD-tree. By calculating the variance of the dataset in each dimension, the dimension with the largest variance is selected as the partitioning domain. The largest variance indicates that the data points are sparsely distributed along that dimension, and partitioning the data in that direction yields the best resolution. Then, all sample points are sorted according to the value of their first division dimension, and the middle data point is selected as the division domain of the partitioning node.

[0084] The specific steps are as follows:

[0085] a. Select the dimension K with the largest variance in the k-dimensional dataset. For this technique, K is 28-dimensional. Then, select the median value m as the pivot to split the dataset and obtain two subsets. At the same time, create tree nodes for storage.

[0086] b. Repeat step a for both subsets until all subsets cannot be subdivided further; if a subset cannot be subdivided further, save the data in that subset to a leaf node.

[0087] In some embodiments, to ensure the efficiency of KD-tree queries, the height of the tree must be limited because query efficiency drops sharply when the vector dimension exceeds a certain value, such as 100. Furthermore, calculating the variance and mean value of each dimension for a large dataset would be a very time-consuming task; considering the specificity of images, the divisor for each dimension can simply be set to the median.

[0088] After implementing the descriptive feature vectors for all images in the dataset and storing them in a KD-tree structure, another crucial part of image mosaicking needs to be implemented: image similarity measurement. Image similarity retrieval is summarized as the process of retrieving the feature vector of each square in the input image and all vectors in the KD-tree, finding the vector with the highest similarity, and returning its corresponding image.

[0089] To calculate the similarity between feature vectors, the simplest method is L2 distance (Euclidean distance), which calculates the similarity between corresponding feature vectors. Euclidean distance (also known as the Euclidean metric) is a commonly used definition of distance, representing the actual distance between two points in m-dimensional space, or the natural length of a vector (i.e., the distance from that point to the origin). Assuming two image feature vectors are denoted as (p1, p2, p3, p4, ...) and (q1, q2, q3, q4, ...), the Euclidean distance is calculated as follows:

[0090]

[0091] In this application, a weighted Euclidean distance function is used to measure the similarity of 28-dimensional feature vectors instead of directly using the Euclidean distance method. Therefore, color or texture features can be highlighted in the distance measurement, enabling more professional matching.

[0092] For example, the formula for calculating this weighted Euclidean distance:

[0093]

[0094] In equation (1-1), this formula is used to measure the similarity of 28-dimensional feature vectors. V is the corresponding grid index in I∩T, K is the number of elements in V, set to 9, because the image needs to be divided into 3×3 grids, H=8, C=255, and S is a square in the input image. α and β are weighting factors for the region information entropy and the 27-dimensional RGB color components. Preferably, α=0.4 and β=0.6, thus achieving a better balance between information entropy and HSV color.

[0095] Figure 5 This is a diagram of the image similarity retrieval method in Example 1, as shown below. Figure 5 As shown, the image similarity retrieval method includes:

[0096] Step 5011: Describe image features and construct a KD tree structure.

[0097] For example, image descriptors are used to describe the features of images in the dataset, and the feature vectors are constructed into a KD-tree structure. The same image descriptors are used to describe the features of the first input image. Therefore, the image descriptors of the feature vectors of the images in the database are consistent with the descriptors of the feature vectors of the first image; where the HSV color space is used to describe the color features of the images in the database, and information entropy is used to describe the texture features of the images in the database.

[0098] Step 5012: Use the weighted Euclidean distance function to find the feature vector of the first image in the KD tree.

[0099] Step 5013: Store the matched feature vectors in a dictionary structure.

[0100] In some embodiments, the matched feature vector is the vector with the highest similarity between the feature vector of the image in the database and the feature vector of the first image; wherein, a weighted Euclidean distance function is used to measure the similarity between feature vectors.

[0101] Furthermore, in the dictionary structure, the Key represents the first input image, and the Value represents the address corresponding to that first image. That is, the matched feature vectors are stored in the dictionary structure, which contains the correspondence information between the first and second images.

[0102] The following is an illustrative description of image filling in the embodiments of this application.

[0103] In step 104, the retrieved matching images corresponding to the feature vectors are used to fill the plurality of blocks in the first image to generate a second image with a mosaic effect. This process is also known as the mosaic image generation process. This process has two key aspects: the image filling position is correct; and the image size needs to be adjusted to the same size as the corresponding square.

[0104] As mentioned earlier, after performing image similarity search, each image has a best-matching image. The corresponding matching relationships are stored in a dictionary structure, which is a key-value pair, an efficient way to represent the correspondence between image patches and the best-matching image. Therefore, when it's necessary to generate a mosaic image by filling blocks with the best-matching image, each item in the dictionary can be read sequentially to obtain their correspondence, and then the corresponding image can be filled into the specified position. Furthermore, the key in the dictionary represents the first input image, i.e., a quadrant of the quadtree. After the quadtree is partitioned, each quadrant has a position attribute, representing its four-point coordinates at a specific location in the input image.

[0105] For example, the specific implementation process can be summarized as follows:

[0106] (1) Read a quadrant object and obtain its four point coordinates (l, t, r, b).

[0107] (2) Use four-point coordinates to calculate the size of each texture, width = rl, height = b – t.

[0108] (3) Use the Open(path) function to read the best matching image for the texture, mainly by reading the values ​​in the dictionary.

[0109] (4) Use the resize() function provided by the PIL library to adjust the size of the best matching image, which is the same size as the corresponding first image.

[0110] (5) Use the past paste() function provided by the PIL library to fill the best matching image.

[0111] The steps of the embodiments of this application have been illustrated above. The technical features and effects of this application will be further described below.

[0112] In some embodiments, the purpose of using a quadtree segmentation algorithm to segment the input image is to divide the image into blocks of different sizes. The image blocks in the feature regions are smaller and denser, which helps to preserve the detailed features of the input image, while the image blocks in the non-feature regions are quite large, maintaining the clarity of the filled image, thereby providing users with more useful visual information and presenting different styles.

[0113] In some embodiments, the quadtree partitioning result depends on the maximum number of iterations D and the minimum size S. The impact of setting different values ​​for parameters D and S is investigated experimentally. Appropriate parameters are then selected as the basis for the subsequent image stitching generation process.

[0114] Figure 6 This is an example diagram of a quadtree image segmentation result according to an embodiment of this application. The input image is a portrait image, and black lines are drawn on the segmentation result using drawing tools. Figure 6 As can be seen, quadtree image segmentation can indeed segment an image into squares of different sizes. Particularly in smooth background areas, the tiles are larger due to the subtle differences in RGB color and information entropy, while in foreground portrait areas, especially around the beard, collar, and hands, the tiles are smaller. This is because these areas contain more image feature information and exhibit greater RGB variance variation, especially at image edges, where the segmented squares are smaller than those in other parts.

[0115] Figure 7This is an example diagram illustrating the changing parameters according to an embodiment of this application. From left to right, it shows the results of changing two parameters, D and S. When the maximum number of iterations D is 1000, it can be observed that although the size of the background area of ​​the squares is sufficient to express the feature information, in the face area of ​​the image, if the image is filled with these squares, it will be too large to express the feature information of the face area.

[0116] The number of iterations was further increased to 3500, and the minimum size S was reduced to 4. The results show that as the number of iterations increases, some areas of the background region become too small. Merging four adjacent squares improves the representation of the background region. Simultaneously, it can be observed that the minimum size of the squares is now sufficient; further reducing the minimum tile size will cause the mosaic result to fail to recognize the content of the filled image, visually reducing it to a single dot.

[0117] To observe further partitioning results, the maximum number of iterations was further reduced to 2500, while the minimum size remained at 4. As can be seen from the right side of the graph, the background region is now appropriately partitioned because merging four adjacent squares can affect the edge features of the human region. Conversely, in certain parts of the portrait region, such as the mouth and eyes, the squares were segmented smaller to preserve the small features of these areas without making them so small that the filled image content is lost. Therefore, the maximum number of iterations was set to 2500, and the minimum square size was set to 4.

[0118] The user information in this application embodiment is obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the customer.

[0119] Figure 8 This is another example diagram of the quadtree image partitioning result according to an embodiment of this application, showing the partitioning result of applying the above parameters to another flower image.

[0120] Figure 9 This is an example image of the generated mosaic image according to an embodiment of this application. The input images are a high-resolution portrait image and a flower image. For the mosaic effect, the overall color of the images can be observed compared to the input images, and details such as the beard and eyes in the portrait can be distinguished. Because this embodiment uses quadtree adaptive segmentation, a large fill image can be seen in the background areas of both images. Conversely, the fill image size must be very small for more core parts of the middle image, such as the edges of the yellow stamens of the flower, the edges of the hand portrait image, and the collar. Therefore, the application of the adaptive segmentation method results in a better mosaic effect.

[0121] Figure 10This is an example image using a traditional RGB color space-based mosaic, without employing a KD-tree to speed up the image retrieval process. To the naked eye, the RGB-based image mosaic reproduces the image features of the input image.

[0122] However, it can be observed that the overall color style of the RGB-based image mosaic differs significantly from that of the HSV-based image mosaic in this embodiment. Firstly, in the background area of ​​the flower image, the RGB-based image mosaic result is yellow, while in the HSV-based result of this embodiment, the overall tone becomes white. Furthermore, this difference is even more pronounced in portrait images. In the beard area of ​​this image, the RGB-based result shows a bluish-white area, which clearly does not match the beard color of the source image, while the HSV-based result shows a beard color that is closer to the source image.

[0123] For the most common image RGB color descriptor, the matching of blocks in the dataset and the image only considers the similarity of red, green and blue. Their colors still have some differences from the input image. Compared with the RGB color space, the HSV color space in this application is more in line with human visual perception and is closest to the human visual system.

[0124] Furthermore, embodiments of this application also consider the color difference and mutual information entropy between the mosaic result and the input image. Color difference refers to the mean square error of color between the mosaic image and the input image.

[0125] For example, the formula is shown below:

[0126]

[0127] Where M is the mosaic result and I is the input image. Mutual information entropy refers to the amount of information about one variable about another, and its formula is:

[0128] I m (M, I) = H(M) + H(I) - H j (M, I)

[0129] Based on the image mosaic result evaluation method defined above, this application embodiment considers the following three aspects to comprehensively evaluate the mosaic result: color difference and mutual information entropy between the mosaic result and the input image; the number of filled images; and the mosaic generation time.

[0130] As the dimension of the image feature descriptor increases, the number of fill images selected for image similarity retrieval gradually decreases. This means that the mosaic image will have more repeating fill images, making the mosaic result look less cluttered, especially in areas of the same color, and better maintaining uniformity. This technique uses the fewest images possible.

[0131] Based on the color difference between the mosaic result and the input image, the mosaic image generated in this embodiment has the smallest color difference from the source image. Simultaneously, the mosaic image generated in this embodiment has the largest mutual information entropy value. This demonstrates that the mosaic image generation technique of this embodiment achieves better results when considering both color and texture features simultaneously.

[0132] Furthermore, the mosaic image generation technique in this embodiment generates the results in the shortest time. This is because an efficient KD-tree data structure is used in the image similarity retrieval process, which greatly accelerates the generation process.

[0133] The following is a summary of some of the beneficial effects of the embodiments of this application:

[0134] Efficient image segmentation algorithms can divide an input image into regular or irregular blocks. This application employs a top-down quadtree image segmentation algorithm, which enables adaptive segmentation, dividing the input image into blocks of different sizes.

[0135] Appropriate image feature descriptors are used to describe the color or texture features of an image. All images are represented using the same image descriptor. This application's embodiments employ the HSV color space, consistent with human visual perception, to describe the color features of the image, and extend Shannon's information entropy formula to obtain a one-dimensional information entropy formula, used to measure texture information in an image, thus describing the image's content information from two dimensions.

[0136] The images in the dataset require preprocessing. This application embodiment saves memory and improves processing time by appropriately cropping the images and preserving useful parts, and uses the same image descriptors as the input images to describe the features of the images in the dataset.

[0137] An efficient data structure is used to store feature vectors, making the search process faster and more efficient. This application's embodiments employ a novel KD-tree structure.

[0138] An appropriate image similarity measurement method is used. This application embodiment employs a weighted distance measurement function to make image similarity measurement more accurate. After similarity search, each square is replaced with the best matching image; this application embodiment uses a fast and efficient image tiling method to achieve this.

[0139] Therefore, in this embodiment of the application, by adaptively segmenting the image and retrieving it based on the feature vector, it is possible to retain the features of the source image to the greatest extent while also giving the generated mosaic image a unique artistic style.

[0140] This application also provides an image mosaic generation apparatus, corresponding to the image mosaic generation method in the above embodiments.

[0141] Figure 11 This is a diagram of an image mosaic generation apparatus according to an embodiment of this application, such as... Figure 11 As shown, the image mosaic generation device 60 includes: an image segmentation unit 610, a feature description unit 620, a retrieval unit 630, and a filling unit 640.

[0142] In this embodiment, the image segmentation unit 610 adaptively segments the input image to obtain a first image with multiple blocks of at least different sizes; the feature description unit 620 performs feature description on the first image to establish a feature vector of the first image; the retrieval unit 630 retrieves matching feature vectors from the database for each block in the first image based on the feature vector of the first image; and the filling unit 640 fills the multiple blocks in the first image with the retrieved matching images corresponding to the feature vectors to generate a second image with mosaic.

[0143] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image mosaic generation method described in the above embodiments.

[0144] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image mosaic generation method described above.

[0145] The present invention has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present invention. Those skilled in the art can make various modifications and variations to the present invention based on its spirit and principles, and these modifications and variations are also within the scope of the present invention.

[0146] The image mosaic generation apparatus described in conjunction with the embodiments of the present invention can be directly embodied in hardware, software modules executed by a processor, or a combination of both. These hardware modules can be implemented, for example, by embedding these software modules using a field-programmable gate array (FPGA).

[0147] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the electronic device uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.

[0148] The image mosaic generation apparatus described in this embodiment can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. It can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors communicating with a DSP, or any other such configuration.

[0149] The embodiments of the present invention also relate to storage media for storing the above programs, such as hard disks, magnetic disks, optical disks, DVDs, flash memory, etc.

[0150] It should be noted that the limitations on each step involved in this solution are not considered as limiting the order of steps, provided that they do not affect the implementation of the specific solution. The steps listed first can be executed first, later, or even simultaneously. As long as this solution can be implemented, it should be considered to fall within the scope of protection of this application.

Claims

1. A method for generating image mosaic, characterized in that, The method for generating the image mosaic includes: A top-down quadtree image segmentation algorithm is used to adaptively segment the input image to obtain a first image with multiple blocks of at least different sizes; The first image is characterized by features, and a feature vector of the first image is established. Based on the feature vector of the first image, matching feature vectors are retrieved from the database for each of the multiple blocks in the first image. The multiple blocks in the first image are filled with the matching images corresponding to the retrieved matching feature vectors to generate a second image with mosaic. The quadtree image segmentation algorithm includes: Set a maximum number of iterations and a minimum quadrant size, wherein the maximum number of iterations is used to limit the depth of the maximum segmentation of the quadtree, and the minimum quadrant size is used to represent the width and height of the minimum allowed quadrant; Construct the input image as the root quadrant, calculate the RGB variance and image entropy of the input image, add them to the heap data structure, and select the quadrant with the largest RGB variance and image entropy in the heap data structure for partitioning; Iteratively execute the step of dividing the heap data structure into quadrants with the largest RGB variance and image entropy, and determine whether the number of iterations has reached the maximum number of iterations or whether the size of the first image has reached the minimum quadrant size. If the judgment result is yes, the image segmentation ends; otherwise, the quadrant segmentation continues.

2. The method for generating image mosaic according to claim 1, characterized in that, The segmented first image includes uniform and non-uniform regions; wherein, the blocks in the uniform regions are of uniform size, and the blocks in the non-uniform regions are of inconsistent size.

3. The method for generating image mosaic according to claim 1, characterized in that, The feature vector of the first image includes a color feature vector and a texture feature vector; The HSV color space is used to describe the color features of the first image, and information entropy is used to describe the texture features of the first image.

4. The method for generating image mosaic according to claim 1, characterized in that, The image descriptor of the feature vector of the image in the database is consistent with the descriptor of the feature vector of the first image; wherein, the HSV color space is used to describe the color features of the image in the database, and information entropy is used to describe the texture features of the image in the database.

5. The method for generating image mosaic according to claim 1, characterized in that, The database uses a KD-tree structure to store feature vectors; The leaf nodes of the KD tree structure store data, while the non-leaf nodes store the current division dimension, the current dimension, and the decomposition values ​​of the left and right nodes of the current node.

6. The method for generating image mosaic according to claim 1, characterized in that, The matched feature vector is the vector with the highest similarity between the feature vector of the image in the database and the feature vector of the first image; wherein, a weighted Euclidean distance function is used to measure the similarity between feature vectors.

7. The method for generating image mosaic according to claim 1, characterized in that, The matched feature vectors are stored in a dictionary structure, which contains the correspondence information between the first image and the second image.

8. An apparatus for generating image mosaic, characterized in that, The image mosaic generation device includes: The image segmentation unit uses a top-down quadtree image segmentation algorithm to adaptively segment the input image to obtain a first image with multiple blocks of at least different sizes. A feature description unit describes the features of the first image and establishes a feature vector of the first image. The retrieval unit retrieves matching feature vectors from the database for each of the multiple blocks in the first image based on the feature vectors of the first image. A filling unit fills the plurality of blocks in the first image with the retrieved matching image corresponding to the matching feature vector to generate a second image with mosaic. The quadtree image segmentation algorithm includes: Set a maximum number of iterations and a minimum quadrant size, wherein the maximum number of iterations is used to limit the depth of the maximum segmentation of the quadtree, and the minimum quadrant size is used to represent the width and height of the minimum allowed quadrant; Construct the input image as the root quadrant, calculate the RGB variance and image entropy of the input image, add them to the heap data structure, and select the quadrant with the largest RGB variance and image entropy in the heap data structure for partitioning; Iteratively execute the step of dividing the heap data structure into quadrants with the largest RGB variance and image entropy, and determine whether the number of iterations has reached the maximum number of iterations or whether the size of the first image has reached the minimum quadrant size. If the judgment result is yes, the image segmentation ends; otherwise, the quadrant segmentation continues.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image mosaic generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the image mosaic generation method according to any one of claims 1 to 7.

Images