A method, system, device and storage medium for similar image retrieval

Abstract

The present invention discloses a method, system, device and storage medium for similar image retrieval. In this method, the to-be-tested defective painting image is input into a preset semantic repair model to obtain a roughly repaired painting image output by the preset semantic repair model; pixel neural features and pixel RGB features at multiple scales in the roughly repaired painting image are extracted; according to the pixel neural features and pixel RGB features at multiple scales, a first preset number of similar pixels are obtained; the spatial consistency of the first preset number of similar pixels is verified, and a second preset number of spatially consistent feature matching pairs are selected; according to the feature matching pairs, the dense correspondence relationship between the roughly repaired painting image and other painting images is calculated, and similar image retrieval is performed according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image. The present invention can improve the accuracy of similar image retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image retrieval, and in particular to a method, system, device and storage medium for retrieving similar images. Background Art

[0002] Painting is one of the historical and cultural heritages recording human civilization. Due to improper protection, humidity and light, most of the current painting art cultural relics have defects, such as aging, scratches and peeling. Peeling and cracks are the most common defects in ancient murals. Peeling related to aging time and machinery often leads to the loss of large areas of the mural content, thus seriously affecting the visual effect of the whole mural. Painting field knowledge has important reference value for restoring damaged painting art cultural relics. Painting knowledge mainly includes two categories: painting content and painting tools. The painting knowledge related to the painting content mainly involves replicas and similar content existing in the painting itself. Painting cultural relics are precious works among many paintings, such as Dunhuang murals, Van Gogh oil paintings, etc. These precious painting art cultural relics are usually copied by many artists in various historical periods, which may provide complete painting information for the restoration of the original painting works. Painting cultural relics seem unique, but painters often reuse the same elements and patterns in different works, and the painting styles and techniques of the same painter also exist repeatedly in multiple works. For example, Dunhuang murals are all painted by strictly trained painters, which makes the murals of the same period similar in painting style and content.

[0003] Currently, most image retrieval methods are for complete images, and the supervised deep neural network image retrieval method requires a large amount of labeled clean training data. Painting artworks have problems such as defects and stains and limited data, and the existing deep learning methods cannot effectively retrieve similar patterns in damaged painting images. Traditional feature-based image retrieval methods only consider the visual effect similarity of images and cannot perform accurate painting pattern searches. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. For this purpose, the present invention provides a method, system, device and storage medium for retrieving similar images, which can improve the accuracy of retrieving similar images.

[0005] In a first aspect, an embodiment of the present invention provides a method for retrieving similar images, and the method for retrieving similar images includes:

[0006] Inputting a to-be-tested damaged painting image into a preset semantic repair model to obtain a roughly repaired painting image output by the preset semantic repair model;

[0007] Extract the pixel neural features and pixel RGB features at multiple scales in the roughly repaired painting image;

[0008] Obtain a first preset number of similar pixels according to the pixel neural features and the pixel RGB features at multiple scales;

[0009] Verify the spatial consistency of the first preset number of the similar pixels, and select a second preset number of spatially consistent feature matching pairs;

[0010] Calculate the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image.

[0011] Compared with the prior art, the first aspect of the present invention has the following beneficial effects:

[0012] In order to improve the accuracy of similar image retrieval, this method inputs the to-be-tested defective painting image into a preset semantic repair model to obtain the roughly repaired painting image output by the preset semantic repair model; extracts the pixel neural features and pixel RGB features at multiple scales in the roughly repaired painting image; obtains a first preset number of similar pixels according to the pixel neural features and pixel RGB features at multiple scales; verifies the spatial consistency of the first preset number of similar pixels, and selects a second preset number of spatially consistent feature matching pairs; calculates the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and performs similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image. This method obtains feature matching pairs based on the spatial consistency of similar pixels; calculates the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and performs similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image, which can improve the accuracy of similar image retrieval.

[0013] According to some embodiments of the present invention, the semantic repair model is obtained in the following manner:

[0014] Based on the U-Net network, construct an image repair model and a loss function of the image repair model;

[0015] Train the image repair model through an image repair data set and a defect mask until the loss function converges to obtain a trained image repair model;

[0016] Fine-tune the parameters of the trained image repair model using defective painting image samples to obtain the semantic repair model.

[0017] According to some embodiments of the present invention, the loss function for constructing the image inpainting model includes:

[0018]

[0019] where M represents the defect mask, G(w,x) represents the complete inpainted image output by the first image inpainting model, y represents the complete ground truth image corresponding to the complete inpainted image, and λ i represents the weight factor of the loss function i, and L i represents the loss function i.

[0020] According to some embodiments of the present invention, the extraction of pixel neural features and pixel RGB features at multiple scales from the coarsely inpainted painting image includes:

[0021] Input the coarsely inpainted painting image into the VGG19 model pre-trained on ImageNet;

[0022] Extract the pixel neural features and the pixel RGB features at multiple scales from the coarsely inpainted painting image through the VGG19 model pre-trained on ImageNet.

[0023] According to some embodiments of the present invention, the obtaining of a first preset number of similar pixels based on the pixel neural features and the pixel RGB features at multiple scales includes:

[0024] Taking a first pixel block of a first preset size centered on any pixel p;

[0025] Extracting the pixel neural features and the pixel RGB features corresponding to the first pixel block through the VGG19 model pre-trained on ImageNet;

[0026] Matching the pixel neural features and the pixel RGB features corresponding to the first pixel block with the images in the complete database to construct a feature matching correspondence;

[0027] Selecting a first preset number of similar pixels from the complete database according to the feature matching correspondence:

[0028]

[0029] where k represents the first preset number, q represents the similar pixels, and x t,p represents the first pixel block, and x m,n represents the image x in the complete database m a second pixel block of a second preset size centered on n, S(x t,p ,x m,n) represents the similarity measure between the first pixel block and the second pixel block, and the similarity measure is:

[0030]

[0031] where φ(x t,p ) represents the pixel neural feature and pixel RGB feature corresponding to the first pixel block, and φ(x m,n ) represents the pixel neural feature and pixel RGB feature corresponding to the second pixel block.

[0032] According to some embodiments of the present invention, the verifying the spatial consistency of the first preset number of similar pixels and selecting the second preset number of spatially consistent feature matching pairs includes:

[0033] Define the adjacent pixels of the pixel p as the verification area;

[0034] For each pixel in the first preset number of similar pixels, take the adjacent pixels at the same position for individual matching to obtain the adjacent pixel matching degree;

[0035] According to the adjacent pixel matching degree, sort the first preset number of similar pixels and the pixel p according to similarity, and use the K-nearest neighbor method to select the second preset number of spatially consistent feature matching pairs.

[0036] According to some embodiments of the present invention, the calculating the dense correspondence relationship between the coarsely repaired painting image and other painting images according to the feature matching pairs includes:

[0037] According to the feature matching pairs, obtain the regions of multiple spatially consistent feature matching pairs;

[0038] Use the Hough transform to identify multiple regions, and use the random sample consensus method to perform affine transformation recovery on the inliers of each region's data to obtain other painting images similar to the coarsely repaired painting image;

[0039] Calculate the dense correspondence relationship between the coarsely repaired painting image and the similar other painting images:

[0040]

[0041] where I represents the index of the internal matching pairs, e i represents the error between the geometric model and the corresponding i-th matching pair, s i represents the feature similarity of the i-th matching pair, N represents the number of the i-th group of internal matching pairs, and σ represents the standard variance of e i .

[0042] Second aspect, an embodiment of the present invention further provides a similar image retrieval system, and the similar image retrieval system includes:

[0043] An image restoration unit, configured to input a to-be-tested defective painting image into a preset semantic restoration model, and obtain a roughly restored painting image output by the preset semantic restoration model;

[0044] A feature extraction unit, configured to extract pixel neural features and pixel RGB features at multiple scales in the roughly restored painting image;

[0045] A similar pixel acquisition unit, configured to acquire a first preset number of similar pixels according to the pixel neural features and the pixel RGB features at multiple scales;

[0046] A feature matching unit, configured to verify the spatial consistency of the first preset number of similar pixels, and select a second preset number of spatially consistent feature matching pairs;

[0047] A similar image retrieval unit, configured to calculate a dense correspondence relationship between the roughly restored painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly restored painting image.

[0048] Third aspect, an embodiment of the present invention further provides a similar image retrieval device, including at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a similar image retrieval method as described above.

[0049] Fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute a similar image retrieval method as described above.

[0050] It can be understood that the beneficial effects of the above second aspect to the fourth aspect compared with the related art are the same as those of the above first aspect compared with the related art. For relevant descriptions, reference can be made to the relevant descriptions in the above first aspect, and details will not be repeated here. Description of the Drawings

[0051] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0052] Figure 1 is a flowchart of a similar image retrieval method according to an embodiment of the present invention;

[0053] Figure 2 It is the structural diagram of the U-Net network model according to an embodiment of the present invention;

[0054] Figure 3 It is the structural diagram of the pre-trained VGG19 model according to an embodiment of the present invention;

[0055] Figure 4 It is the structural diagram of a similar image retrieval system according to an embodiment of the present invention. Detailed implementation manners

[0056] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0057] In the description of the present invention, if the first, second, etc. are described only for the purpose of distinguishing technical features, they should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features.

[0058] In the description of the present invention, it should be understood that for the orientation description, such as up, down, etc., the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present invention.

[0059] In the description of the present invention, it should be noted that unless otherwise clearly defined, words such as setting, installation, connection, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0060] Currently, most image retrieval methods are for complete images, and supervised deep neural network image retrieval methods require a large amount of labeled clean training data. There are problems such as defects and stains in painting artworks and the data is limited. Existing deep learning methods cannot effectively retrieve similar patterns in defective painting images. Traditional feature-based image retrieval methods only consider the visual effect similarity of images and cannot perform precise painting pattern searches.

[0061] To solve the above problems, in order to improve the accuracy of similar image retrieval, the present invention inputs the to-be-tested defective painting image into a preset semantic repair model to obtain a roughly repaired painting image output by the preset semantic repair model; extracts pixel neural features and pixel RGB features of multiple scales in the roughly repaired painting image; obtains a first preset number of similar pixels according to the pixel neural features and pixel RGB features of multiple scales; verifies the spatial consistency of the first preset number of similar pixels, and selects a second preset number of spatially consistent feature matching pairs; calculates the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and performs similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image. The present invention obtains feature matching pairs based on the spatial consistency of similar pixels; calculates the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and performs similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image, which can improve the accuracy of similar image retrieval.

[0062] Referring to Figure 1 , an embodiment of the present invention provides a method for similar image retrieval, and the method for similar image retrieval includes:

[0063] Step S100, input the to-be-tested defective painting image into a preset semantic repair model to obtain a roughly repaired painting image output by the preset semantic repair model.

[0064] Specifically, referring to Figure 2 , based on the U-Net network, construct an image repair model and a loss function of the image repair model; train the image repair model through an image repair data set and a defective mask until the loss function converges to obtain a trained image repair model. The specific process is as follows:

[0065] Select an image repair data set with rich data, for example, CelebA-HQ and Place2, and model the semantic repair of the defective painting image as an image completion problem. Randomly generate a defective mask for brush strokes by setting the number of brush stroke turning points, brush width, brush length, and brush turning corners, and use the public image data CelebA-HQ or Place2 and the randomly generated defective mask to train the U-Net network model to minimize the loss function of the model. After the loss function converges, the decoder of the U-Net network model can effectively extract the features of image completion, and its encoder can use the features learned by the decoder to complete the defective area of the image.

[0066] Construct the loss function of the image repair model in the following way:

[0067]

[0068] Among them, M represents the defect mask, G(w, x) represents the complete repaired image output by the trained image inpainting model, y represents the complete real image corresponding to the complete repaired image, and λ i represents the weight factor of the loss function i, and L i represents the loss function i.

[0069] L i can be various loss functions, mainly including the reconstruction loss function L R and the adversarial loss function L adv , where the reconstruction loss function L R is:

[0070] L R = |M⊙G(w, x) - M⊙y| + |G(w, x) - y|

[0071] The adversarial loss function L adv is:

[0072]

[0073] Among them, y represents the real complete image, w represents the parameters of the inpainting network G, x represents the input image, ⊙ represents element-wise multiplication, u1 and u2 represent weights, and P real represents the real data distribution, P fake represents the forged data distribution, and E represents the expectation.

[0074] Based on the trained image inpainting model, and based on transfer learning, the obtained trained image inpainting model is used as the learning starting point for the semantic repair of the defective painting image, and the defective painting image samples are used to fine-tune the parameters of the trained image inpainting model to obtain the semantic repair model.

[0075] Through the obtained semantic repair model, the repair model can perform semantic repair. The defective painting image to be tested is input into the preset semantic repair model, and the rough repaired painting image output by the preset semantic repair model is obtained.

[0076] In this embodiment, the small sample data of the defective painting image is used to fine-tune the model parameters, so that the U-Net network adapts to the semantic repair of the defective painting image.

[0077] In this embodiment, the semantic repair of the small sample defective painting image can improve the feasibility of finding similar patterns in the defective area of the defective painting image.

[0078] Step S200: Extract the pixel neural features and pixel RGB features at multiple scales in the rough repaired painting image.

[0079] Specifically, referring to Figure 3 , input the coarsely repaired painting image into the VGG19 model pre-trained on ImageNet;

[0080] Extract the pixel neural features and pixel RGB features at multiple scales in the coarsely repaired painting image through the VGG19 model pre-trained on ImageNet.

[0081] Step S300: Obtain the first preset number of similar pixels according to the pixel neural features and pixel RGB features at multiple scales.

[0082] Specifically, taking any pixel p in the coarsely repaired painting image as the center, take the first pixel block x of the first preset size s×s t,p ;

[0083] Extract the pixel neural features and pixel RGB features corresponding to the first pixel block through the VGG19 model pre-trained on ImageNet;

[0084] Match the pixel neural features and pixel RGB features corresponding to the first pixel block with the images in the complete database to construct a feature matching correspondence;

[0085] Select the first preset number of similar pixels from the complete database according to the feature matching correspondence:

[0086]

[0087] Among them, k represents the first preset number, q represents the similar pixels, x t,p represents the first pixel block, x m,n represents the image x in the complete database m the second pixel block of the second preset size centered on n, S(x t,p , x m,n ) represents the similarity measure between the first pixel block and the second pixel block, and the similarity measure is:

[0088]

[0089] Among them, φ(x t,p ) represents the pixel neural features and pixel RGB features corresponding to the first pixel block, and φ(x m,n ) represents the pixel neural features and pixel RGB features corresponding to the second pixel block.

[0090] It should be noted that in this embodiment, the first preset number of similar pixels ranked at the front in the complete database is selected. The first preset number can be changed according to actual needs, and this embodiment does not make specific limitations.

[0091] Step S400: Verify the spatial consistency of the first preset number of similar pixels, and select the second preset number of spatially consistent feature matching pairs.

[0092] Specifically, define the adjacent pixels of pixel p as the verification area; for each pixel in the first preset number of similar pixels, take the adjacent pixels at the same position for individual matching to obtain the adjacent pixel matching degree; according to the adjacent pixel matching degree, sort the similarity between the first preset number of similar pixels and pixel p, and use the K-nearest neighbor method to select the second preset number of spatially consistent feature matching pairs. Specifically:

[0093] Further verify the effectiveness of K nearest neighbor pixels based on the spatial consistency of adjacent pixel feature matching. Define the adjacent pixels of pixel p as the verification area, take the adjacent pixels at the same position for each pixel in the first preset number of similar pixels for individual matching to obtain the adjacent pixel matching degree; re-sort the similarity between the K nearest neighbor pixels (i.e., the first preset number of similar pixels) and pixel p according to the matching degrees of all adjacent pixels with spatial structure, and select the nearest neighbor second preset number K1 (K1 < K) of spatially consistent and effective feature matching pairs based on the K-nearest neighbor method.

[0094] It should be noted that in this embodiment, the nearest neighbor second preset number of spatially consistent and effective feature matching pairs is selected, and the second preset number can be changed according to actual needs, which is not specifically limited in this embodiment.

[0095] Step S500: Calculate the dense correspondence relationship between the coarsely repaired painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the coarsely repaired painting image.

[0096] Specifically, according to the feature matching pairs, obtain the regions of multiple spatially consistent feature matching pairs;

[0097] Use the Hough transform to identify multiple regions, and use the random sample consensus method to perform affine transformation recovery on the inliers of each region's data to obtain other painting images similar to the coarsely repaired painting image. Specifically:

[0098] First, check all the images in the complete database, calculate the features between the coarsely repaired painting image and other painting images pairwise and perform feature matching, then screen out the spatially consistent feature matching pairs and find the regions with multiple spatially consistent matching pairs. The region search method is: use the Hough transform to identify possible consistent correspondence regions, and then for each region, use the random sample consensus (RANSAC) method to perform affine transformation recovery on the inliers of the data of this region of the two images. Thus, find other painting images that are content-consistent but have some structural distortions and are similar to the coarsely repaired painting image.

[0099] Calculate the dense correspondence between the coarsely repaired painting image and other similar painting images:

[0100]

[0101] where I represents the index of the internal matching pair, and e i represents the error between the geometric model and the corresponding i-th matching pair, and s i represents the feature similarity of the i-th matching pair, N represents the number of internal matching pairs in the I-th group, and σ represents the standard variance of e i of.

[0102] Perform similar image retrieval based on the dense correspondence to obtain similar painting images similar to the coarsely repaired painting image. Specifically:

[0103] Classify the feature matching groups based on the correspondence S(I), connect the matching regions in the similar painting images with overlapping Intersection over Union (IoU) scores greater than a given threshold, and extract the connected component graphics therein to implement the retrieval of similar painting images of the coarsely repaired painting image.

[0104] In this embodiment, based on the semantic repair of the damaged painting image and the selection of feature matching pairs using spatial consistency, it is possible to realize the retrieval of existing painting patterns in small-sample painting damaged images, and it can help to discover the correlation of painting patterns in painting artworks. Based on the semantic repair learning of transfer learning, the preliminary semantic repair of small-sample damaged painting images can improve the feasibility of finding similar patterns in the damaged areas of damaged painting images. Based on the feature matching pairs of multi-scale neural features and feature matching spatial consistency, it is possible to extract effective feature matching pairs between painting images, and it can improve the accuracy of similar painting image retrieval.

[0105] Referring to Figure 4 , the embodiment of the present invention also provides a similar image retrieval system. This similar image retrieval system includes an image repair unit 100, a feature extraction unit 200, a similar pixel acquisition unit 300, a feature matching unit 400, and a similar image retrieval unit 500, where:

[0106] The image repair unit 100 is configured to input the to-be-tested damaged painting image into a preset semantic repair model to obtain a coarsely repaired painting image output by the preset semantic repair model;

[0107] The feature extraction unit 200 is configured to extract pixel neural features and pixel RGB features at multiple scales in the coarsely repaired painting image;

[0108] A similar pixel acquisition unit 300 is configured to acquire a first preset number of similar pixels according to pixel neural features and pixel RGB features at multiple scales;

[0109] A feature matching unit 400 is configured to verify the spatial consistency of the first preset number of similar pixels and select a second preset number of spatially consistent feature matching pairs;

[0110] A similar image retrieval unit 500 is configured to calculate a dense correspondence relationship between a roughly restored painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly restored painting image.

[0111] It should be noted that since a similar image retrieval system in this embodiment and the above-mentioned similar image retrieval method are based on the same inventive concept, the corresponding content in the method embodiment is equally applicable to the system embodiment of the present invention, and will not be elaborated here.

[0112] An embodiment of the present invention further provides a similar image retrieval device, including: at least one control processor and a memory communicatively connected to the at least one control processor.

[0113] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The non-transitory software programs and instructions required to implement a similar image retrieval method in the above embodiments are stored in the memory. When executed by the processor, the method steps of a similar image retrieval method in the above embodiments are executed. For example, the method steps S100 to S500 described above are executed. Figure 1 in the above.

[0115] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] An embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more control processors, the one or more control processors can be caused to execute a similar image retrieval method in the above method embodiment. For example, execute the functions of method steps S100 to S500 described above. Figure 1 in the method steps S100 to S500.

[0117] Those of ordinary skill in the art can understand that all or some of the steps and systems in the above-disclosed methods can be implemented as software, firmware, hardware, and their appropriate combinations. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or can be implemented as hardware, or can be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or a non-transitory medium) and a communication medium (or a transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium generally includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0118] The above is a specific description of the preferred embodiment of the present application. However, the embodiments of the present application are not limited to the above-described embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the embodiments of the present application. These equivalent deformations or substitutions are all included within the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for retrieving similar images, characterized in that, The described similar image retrieval method includes: Input the to-be-tested defective painting image into a preset semantic repair model to obtain the roughly repaired painting image output by the preset semantic repair model, where the semantic repair model is obtained through the following steps: Based on the U-Net network, construct an image repair model and the loss function of the image repair model; Train the image repair model with an image repair dataset and a defect mask until the loss function converges to obtain a trained image repair model; Fine-tune the parameters of the trained image repair model with defective painting image samples to obtain the semantic repair model; Extract the pixel neural features and pixel RGB features at multiple scales in the roughly repaired painting image. Specifically, Input the roughly repaired painting image into the VGG19 model pre-trained on ImageNet; Extract the pixel neural features and the pixel RGB features at multiple scales in the roughly repaired painting image through the VGG19 model pre-trained on ImageNet; Obtain a first preset number of similar pixels according to the pixel neural features and the pixel RGB features at multiple scales; Verify the spatial consistency of the first preset number of similar pixels and select a second preset number of spatially consistent feature matching pairs; Calculate the dense correspondence relationship between the roughly repaired painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly repaired painting image, where Obtain the regions of multiple spatially consistent feature matching pairs according to the feature matching pairs; Use the Hough transform to identify multiple regions, and use the random sample consensus method to perform affine transformation recovery on the inliers of each region's data to obtain other painting images similar to the roughly repaired painting image; Calculate the dense correspondence relationship between the roughly repaired painting image and the similar other painting images: where I represents the index of the internal matching pair, e i represents the error between the geometric model and the corresponding i-th matching pair, s i represents the feature similarity of the i-th matching pair, N represents the number of internal matching pairs in the I-th group, and σ represents e i 's standard deviation.

2. The similar image retrieval method according to claim 1, wherein Constructing the loss function of the image repair model includes: Among them, M represents the defect mask, G(w,x) represents the complete restored image output by the trained image inpainting model, y represents the complete true image corresponding to the complete restored image, and λ i represents the weight factor of loss function i, and L i represents the loss function i.

3. The similarity image retrieval method according to claim 1, wherein The obtaining of a first preset number of similar pixels according to the pixel neural features and the pixel RGB features at multiple scales includes: Taking a first pixel block with a first preset size centered on any pixel p; Extracting the pixel neural features and pixel RGB features corresponding to the first pixel block through the VGG19 model pre-trained on ImageNet; Matching the pixel neural features and pixel RGB features corresponding to the first pixel block with the images in the complete database to construct a feature matching correspondence relationship; Selecting a first preset number of similar pixels from the complete database according to the feature matching correspondence relationship: Among them, k represents the first preset quantity, q represents the similar pixels, and x t,p represents the first pixel block, and x m,n represents the image in the complete database, x m is the second pixel block of the second preset size centered on n, S(x t,p , x m,n ) represents the similarity measure between the first pixel block and the second pixel block, and the similarity measure is: Among them, φ(x t,p ) represents the pixel neural feature and pixel RGB feature corresponding to the first pixel block, and φ(x m,n ) represents the pixel neural feature and pixel RGB feature corresponding to the second pixel block.

4. The similar image retrieval method according to claim 3, wherein The verifying of the spatial consistency of the first preset number of similar pixels and selecting a second preset number of spatially consistent feature matching pairs includes: Defining the adjacent pixels of the pixel p as the verification region; Taking the adjacent pixels at the same position for each of the first preset number of similar pixels for individual matching to obtain the adjacent pixel matching degree; According to the adjacent pixel matching degree, sort the first preset number of similar pixels and the pixel p in terms of similarity, and use the K-nearest neighbor method to select the second preset number of spatially consistent feature matching pairs.

5. A similar image retrieval system, characterized in that, The similar image retrieval system includes: An image restoration unit, configured to input a to-be-tested defective painting image into a preset semantic restoration model to obtain a roughly restored painting image output by the preset semantic restoration model, wherein the semantic restoration model is obtained by the following method: Based on the U-Net network, construct an image restoration model and a loss function of the image restoration model; Train the image restoration model with an image restoration data set and a defect mask until the loss function converges to obtain a trained image restoration model; Fine-tune the parameters of the trained image restoration model with defective painting image samples to obtain the semantic restoration model; A feature extraction unit, configured to extract pixel neural features and pixel RGB features of multiple scales in the roughly restored painting image. Specifically, Input the roughly restored painting image into the VGG19 model pre-trained on ImageNet; Extract the pixel neural features and the pixel RGB features of multiple scales in the roughly restored painting image through the VGG19 model pre-trained on ImageNet; A similar pixel acquisition unit, configured to obtain the first preset number of similar pixels according to the pixel neural features and the pixel RGB features of multiple scales; A feature matching unit, configured to verify the spatial consistency of the first preset number of similar pixels and select the second preset number of spatially consistent feature matching pairs; A similar image retrieval unit, configured to calculate a dense correspondence relationship between the roughly restored painting image and other painting images according to the feature matching pairs, and perform similar image retrieval according to the dense correspondence relationship to obtain similar painting images similar to the roughly restored painting image, wherein, Obtain regions of multiple spatially consistent feature matching pairs according to the feature matching pairs; Use the Hough transform to identify multiple regions, and use the random sample consensus method to perform affine transformation recovery on the inliers of each region's data to obtain other painting images similar to the roughly restored painting image; Calculate the dense correspondence relationship between the roughly restored painting image and the similar other painting images: where I represents the index of the internal matching pairs, e i represents the error between the geometric model and the corresponding i-th matching pair, s i represents the feature similarity of the i-th matching pair, N represents the number of internal matching pairs in the I-th group, and σ represents the standard variance of e i ​ 6. A similar image retrieval device, characterized in that, Comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the similar image retrieval method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the similar image retrieval method according to any one of claims 1 to 4.

Images