A robust representation method for dense domain images

Abstract

The present invention discloses a method for robust characterization of dense domain images, comprising the following steps. 1) Preprocessing: converting the input image into a grayscale image; 2) Convolution kernel preparation: calculating the convolution kernel according to a given basis function formula and given scale and frequency parameters; 3) Image decomposition: calculating the spatial convolution of the grayscale image and the above kernel, and decomposing the image into multi-scale and multi-frequency coefficients; 4) Invariant construction: according to the required invariance requirements, the covariance formula can be used to further derive rotation, flip, and scale invariant dense features from the above coefficients to achieve robust characterization of the image. The present invention has the characteristics of high characterization coverage, strong robustness and good interpretability, while showing good computational efficiency, and thus has application potential in many problems of computer vision (especially safety-critical scenarios and pixel-level tasks).

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and image processing, and specifically relates to a method for robust representation of dense domain images. Background Art

[0002] Since entering the information age, multimedia data generated by human activities has been increasing at an unimaginable rate. Among them, digital images have become the key information carrier of modern society, active in all aspects of human life and production. In order to more efficiently manage, retrieve and use massive digital images, academia and industry have long been committed to the research and development of various automated artificial vision systems. Among them, image representation plays a fundamental role. Specifically, a typical artificial vision system first converts digital images into useful digital representations for a given task, and then extracts knowledge on this basis and finally realizes high-level visual understanding applications, such as image classification, instance retrieval, object detection, semantic segmentation, etc.

[0003] Starting from the semantics of the target task, we can abstract the basic characteristics that an effective image representation should have: robustness to intra-class changes and discriminability to inter-class changes. At the implementation level, the representation should also have reasonable implementation cost and accuracy. In addition, when the target scene involves safety-critical tasks, the interpretability of the representation will become an important goal; when the target scene involves pixel-level tasks, the coverage of the representation on the image plane will become another important goal.

[0004] In view of the above goals, many researchers have proposed numerous image representation methods from different perspectives, which can be roughly divided into the following three categories. 1) Representations based on deep learning, such as convolutional neural networks (CNNs), have shown impressive performance on various high-level visual tasks. Although they are highly discriminative, they have two inherent weaknesses that make them difficult to use in safety-critical scenarios. The first is low robustness, especially for geometric variants and adversarial samples. The second is poor interpretability, and this black-box nature reduces the credibility of the results. 2) Representations based on key points, such as scale-invariant feature transforms (SIFTs), are very popular and competitive methods in feature engineering. Key point detectors and descriptors are generally designed under various geometric invariance constraints, and therefore usually have good robustness. However, such key points are sparse on the image plane, resulting in information loss in some areas. This is obviously not suitable for pixel-level tasks. 3) Representations based on dense sampling, such as DAISY descriptors. This type of method is built on a regular sampling grid on the image plane (and scale), which not only guarantees coverage but also simple spatial relationships between descriptors. This can be useful in modeling Markov processes, constructing spatially correlated representations, accelerating feature matching, etc. In dense representations, the main challenges are geometric invariance and computational efficiency. Existing methods are not stable under geometric distortions, even common rotations, scaling, and flipping. In addition, due to the computation in the dense domain, the implementation efficiency of such methods is low, which also limits the use of computationally expensive invariant features. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned prior art, a robust representation method for dense domain images is proposed.

[0006] Technical solution: The technical solution of the present invention is a dense domain image robust representation (DIR) method, which is characterized by the following steps:

[0007] a. Initial Setup

[0008] a.1 Get the image f to be analyzed and initialize the translation parameters (u, v), scale parameters w and frequency parameters (n, m). The above parameters are manually set according to the task requirements. Here we assume that (u, v) ∈ {1, 2, ..., M} × {1, 2, ..., N}, w ∈ {2 c :c=1,2,...,7}, (n,m)∈{(n,m):|n|,|m|<5};

[0009] b. Pretreatment

[0010] b.1 If the image to be analyzed is a color image, convert it to a grayscale image;

[0011] c. Convolution kernel preparation

[0012] c.1 Calculate the convolution kernel according to the following formula

[0013]

[0014] in (·) * Represents the complex conjugate, which can be approximated in practice D ij is the pixel area, i.e. D ij ={(x,y)∈[i-0.5,i+0.5]×[j-0.5,j+0.5]};

[0015] c.2 In c.1, the basis function is defined as:

[0016]

[0017] Where R is the radial basis function, which can be set manually according to the task requirements. Here we assume exp is the exponential function; j is the imaginary unit, the basis function The domain of is D = {(x,y):(xu) 2 +(yv) 2 ≤w 2}.

[0018] d. Image decomposition

[0019] d.1 Calculate the decomposition coefficient of image f according to the following formula

[0020]

[0021] in is the convolution symbol, (·) T Represents matrix transpose;

[0022] d.2 In practice, the direct calculation of d.1 may be inefficient, and can be derived using the following quick calculation formula:

[0023]

[0024] where ⊙ is the matrix point-wise multiplication symbol, represents the fast Fourier transform, represents the inverse fast Fourier transform; the factor term It can also be determined by the known items The interpolation is derived as:

[0025]

[0026] Where w0 is a known term The scale parameter, (ξ i ,ξ j ) is a known item Fourier transform of the independent variable;

[0027] e. Invariant construction

[0028] e.1 Decomposition coefficients Satisfies image translation equivariance, that is:

[0029]

[0030] Where (Δx, Δy) is the offset vector of the image translation, so it can be decomposed into coefficients The same coefficients of the translated version of a given pattern are found in the (u, v) domain, and the translation invariant can be implemented by performing a permutation invariant mapping (such as average pooling) in the (u, v) domain;

[0031] e.2 Decomposition coefficients Satisfies the image rotation covariance, that is:

[0032]

[0033] Where φ is the angle of image rotation, r′,θ′ is the version of r,θ translated by the offset (u,v) and scaled by the factor w, so it can be decomposed into The rotation angle of a given pattern is derived in the phase domain, and the rotation invariant can be implemented by performing a permutation invariant mapping (such as taking the amplitude) in the phase domain; similar conclusions also hold for image flipping;

[0034] e.3 Decomposition coefficients Satisfy image scaling covariance, that is:

[0035]

[0036] Where s is the factor of image scaling, so it can be decomposed into The same coefficients of the scaled version of a given pattern are found in the w domain, while the scaling invariant can be implemented by performing a permutation-invariant mapping (such as average pooling) in the w domain;

[0037] The present invention discloses a method for robust representation of dense domain images (DIR), comprising the following steps. 1) Preprocessing: converting the input image into a grayscale image; 2) Convolution kernel preparation: calculating the convolution kernel according to a given basis function formula and given scale and frequency parameters; 3) Image decomposition: calculating the spatial convolution of the grayscale image and the above kernel, and decomposing the image into multi-scale and multi-frequency coefficients; 4) Invariant construction: according to the required invariance requirements, the covariance formula can be used to further derive rotation, flip, and scale invariant dense features from the above coefficients to achieve robust representation of the image. The present invention has the characteristics of high representation coverage, strong robustness and good interpretability, while showing good computational efficiency, and thus has application potential in many problems of computer vision (especially safety-critical scenarios and pixel-level tasks).

[0038] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0039] First, the present invention has four advantages: high representation coverage (local dense sampling), strong robustness (covariance and invariance of various geometric transformations), good interpretability (both theory and implementation are formalized), and high computational efficiency (complexity of constant order relative to the convolution kernel size), which are not available in existing representation methods. Therefore, the present invention can be better applied in safety-critical scenarios and pixel-level tasks.

[0040] Second, the present invention provides a unified image representation framework, which includes a variety of existing dense representation methods as special cases, that is, the representation robustness and fast calculation strategies involved in the present invention are generalized and can be applied to the definition of a variety of basis functions; BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the present invention;

[0042] Figure 2 (a) is a schematic diagram of the translation transformation of the present invention;

[0043] Figure 2 (b) is a schematic diagram of the rotation transformation of the present invention;

[0044] Figure 2 (c) is a schematic diagram of the flip transformation of the present invention;

[0045] Figure 2 (d) is a schematic diagram of the scaling transformation of the present invention;

[0046] Figure 3 (a) is a schematic diagram of an image f to be analyzed obtained by the present invention;

[0047] Figure 3 (b) The convolution kernel calculated by the present invention Schematic diagram;

[0048] Figure 3 (c) is a schematic diagram of the amplitude of the decomposition coefficient calculated by the present invention;

[0049] Figure 3 (d) is a phase diagram of the decomposition coefficients calculated by the present invention;

[0050] Figure 4 This is a graph showing the calculation speed results of the present invention on the image Lena;

[0051] Figure 5 This is a diagram showing the calculation accuracy results of the present invention on the image Lena;

[0052] Figure 6 This is a graph showing the dense matching experimental results of the present invention in the image library INRIAHolidays. DETAILED DESCRIPTION

[0053] The present invention will be further explained below in conjunction with the accompanying drawings.

[0054] The method of the present invention comprises four steps: preprocessing, convolution kernel preparation, image decomposition and invariant construction.

[0055] Convention: f refers to the image to be analyzed; M and N refer to the number of rows and columns of the image to be analyzed; i, j represent the discrete position coordinates of the pixel; x, y represent the continuous position coordinates of the pixel, that is, (x, y) ∈ [1, M] × [1, N]; r, θ represent the polar coordinates corresponding to x, y; (u, v) are the translation parameters of the basis function; w is the scale parameter of the basis function; (n, m) are the frequency parameters of the basis function; r′, θ′ are the versions of r, θ after being translated by the offset (u, v) and scaled by the factor w; exp is the exponential function; (·) * represents the complex conjugate; j is the imaginary unit; <·,·> represents the inner product; is the convolution symbol; (·) T represents matrix transpose; ⊙ is the symbol for point-by-point matrix multiplication; stands for Fast Fourier Transform.

[0056] a. Initial Setup

[0057] a.1 Get the image f to be analyzed and initialize the translation parameters (u, v), scale parameters w and frequency parameters (n, m). The above parameters are manually set according to the task requirements. Here we assume that (u, v) ∈ {1, 2, ..., M} × {1, 2, ..., N}, w ∈ {2 c :c=1,2,...,7}, (n,m)∈{(n,m):|n|,|m|<5};

[0058] b. Pretreatment

[0059] b.1 If the image to be analyzed is a color image, convert it to a grayscale image;

[0060] c. Convolution kernel preparation

[0061] c.1 Calculate the convolution kernel according to the following formula

[0062]

[0063] in (·) * Represents the complex conjugate, which can be approximated in practice D ij is the pixel area, i.e. D ij ={(x,y)∈[i-0.5,i+0.5]×[j-0.5,j+0.5]};

[0064] c.2 In c.1, the basis function is defined as:

[0065]

[0066] Where R is the radial basis function, which can be set manually according to the task requirements. Here we assume exp is the exponential function; j is the imaginary unit, the basis function The domain of is D = {(x,y):(xu) 2 +(yv) 2 ≤w 2}.

[0067] d. Image decomposition

[0068] d.1 Calculate the decomposition coefficient of image f according to the following formula

[0069]

[0070] in is the convolution symbol, (·) T Represents matrix transpose;

[0071] d.2 In practice, the direct calculation of d.1 may be inefficient, and can be derived using the following quick calculation formula:

[0072]

[0073] where ⊙ is the matrix point-wise multiplication symbol, represents the fast Fourier transform, represents the inverse fast Fourier transform; the factor term It can also be determined by the known items The interpolation is derived as:

[0074]

[0075] Where w0 is a known term The scale parameter, (ξ i ,ξ j ) is a known item Fourier transform of the independent variable;

[0076] e. Invariant construction

[0077] e.1 Decomposition coefficients Satisfies image translation equivariance, that is:

[0078]

[0079] Where (Δx, Δy) is the offset vector of the image translation, so it can be decomposed into coefficients The same coefficients of the translated version of a given pattern are found in the (u, v) domain, and the translation invariant can be implemented by performing a permutation invariant mapping (such as average pooling) in the (u, v) domain;

[0080] e.2 Decomposition coefficients Satisfies the image rotation covariance, that is:

[0081]

[0082] Where φ is the angle of image rotation, r′,θ′ is the version of r,θ translated by the offset (u,v) and scaled by the factor w, so it can be decomposed into The rotation angle of a given pattern is derived in the phase domain, and the rotation invariant can be implemented by performing a permutation invariant mapping (such as taking the amplitude) in the phase domain; similar conclusions also hold for image flipping;

[0083] e.3 Decomposition coefficients Satisfy image scaling covariance, that is:

[0084]

[0085] Where s is the factor of image scaling, so it can be decomposed into The same coefficients of the scaled version of a given pattern are found in the w domain, while the scaling invariant can be implemented by performing a permutation-invariant mapping (such as average pooling) in the w domain;

[0086] Experimental test and parameter setting: The experiment was performed in Matlab R2021a environment. The image library involved in the experiment is INRIAHolidays, which has been made public and can be searched and downloaded online. As shown in the following table, the robustness experimental results of the dense matching of the present invention in the image library INRIAHolidays are as follows:

[0087]

[0088] The specific implementation scheme described above further describes the purpose and technical solutions of the present invention in detail. Those skilled in the art will understand that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Various modifications can be made to the details without departing from the limitations of the attached claims, which should all be deemed to belong to the scope of protection of the present invention.

Claims

1. A method for robust representation of dense domain images, characterized in that: The dense domain image robust representation method comprises the following steps: 1) Obtain the image to be analyzed and initialize the parameters; 2) Preprocessing the image to be analyzed. If the image to be analyzed is in color, it is converted into a grayscale image; 3) Obtain the convolution kernel of the grayscale image; 4) Calculate the spatial convolution of the grayscale image and the convolution kernel, and decompose the grayscale image into multi-scale and multi-frequency decomposition coefficients; 5) Derive dense features that are invariant to translation, rotation, flipping and scaling from the multi-scale and multi-frequency decomposition coefficients to achieve robust representation of the image to be analyzed; The implementation process of step 5) is: a. Multi-scale and multi-frequency decomposition coefficients Satisfies image translation equivariance, that is: Where (Δx, Δy) is the offset vector of the image translation, so in the decomposition coefficient The same coefficients of the translation version of the given pattern are found in the domain where the (u, v) translation parameters are located, and the translation invariant is implemented by performing a permutation invariant mapping in the domain where (u, v) is located, that is, the translation invariant is derived; b. The multi-scale and multi-frequency decomposition coefficients satisfy the image rotation covariance, that is: Where φ is the angle of image rotation, r′,θ′ is the version of r,θ after translation by translation parameter (u,v) and scaling by scale parameter w, so in multi-scale and multi-frequency decomposition coefficients The rotation angle of a given mode is derived from the phase domain, and the rotation invariant is obtained by taking The amplitude of is realized, that is, the rotation invariant is derived; The multi-scale and multi-frequency decomposition coefficients satisfy the image flip covariance, that is: Where f(r,-θ) is the vertical flip version of image f, so the multi-scale and multi-frequency decomposition coefficients The flipping of a given mode is derived in the phase domain, and the flipping invariant is obtained by taking The amplitude of is realized, that is, the flip invariant is derived; c. Multi-scale and multi-frequency decomposition coefficients Satisfy image scaling covariance, that is: Where s is the image scaling factor, so in the decomposition coefficient The same coefficients of the scaled version of the given pattern are found in the domain where the scale parameter w is located, and the scaling invariant is implemented by performing a permutation invariant mapping in the domain where the scale parameter w is located, that is, the scaling invariant is derived.

2. A method for robust representation of dense domain images as claimed in claim 1, characterized in that: The specific steps of step 1) are: obtain the image f to be analyzed, and initialize the translation parameters (u, v), scale parameters w and frequency parameters (n, m), assuming that: (u,v)∈{1,2,...,M}×{1,2,...,N},w∈{2 c :c=1,2,...,7}, (n,m)∈{(n,m):|n|,|m|<5}; Where: c is a variable, M and N are the number of rows and columns of the image f to be analyzed, and |·| is an absolute value.

3. A method for robust representation of dense domain images as claimed in claim 2, characterized in that: Step 3) Get the convolution kernel of the grayscale image Where: i, j represent the discrete position coordinates of the pixel f of the image to be analyzed, st represents the auxiliary condition, D is the domain of the basis function, D ij For the image f pixel area to be analyzed: D ij ={(x,y)∈[i-0.5,i+0.5]×[j-0.5,j+0.5]}, The definition is as follows: in:(·) * Represents the complex conjugate, taking the approximate value represents the basis function, x, y represent the continuous position coordinates of the pixel f of the image to be analyzed, that is, (x, y)∈[1,M]×[1,N]; at the same time, the basis function is defined as: R n is the radial basis function, assuming that: Among them: r is the independent variable, exp is the exponential function; j is the imaginary unit, the basis function The domain of is D = {(x,y):(xu) 2 +(yv) 2 ≤w 2 }.

4. A method for robust representation of dense domain images as claimed in claim 3, characterized in that: Step 4) Obtain the multi-scale and multi-frequency decomposition coefficients of the image to be analyzed f through the formula in is the convolution symbol, (·) T represents the matrix transpose, and f(i,j) is the coordinate of the discrete position of the image f to be analyzed.

5. A method for robust representation of dense domain images as claimed in claim 4, characterized in that: Step 4) Obtain the multi-scale and multi-frequency decomposition coefficients of the image to be analyzed f Another approach: Where: ⊙ is the point-by-point multiplication symbol of the matrix, represents the fast Fourier transform, represents inverse fast Fourier transform; Where w0 is a known term The scale parameter, (ξ i ,ξ j ) is a known item The Fourier transform of the independent variable.

Images