Image Data Fusion Feature Extraction Method, Device, Electronic Device and Storage Medium

Abstract

The present invention relates to the technical field of image data feature recognition, and in particular to an image data fusion feature extraction method, apparatus, electronic device and storage medium. The method of the present invention first obtains a plurality of image blocks; then respectively performs two-dimensional frequency domain feature extraction on the plurality of image blocks to obtain a plurality of first frequency domain matrices; then respectively cuts the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices, and adjusts the data of the cut plurality of first frequency domain matrices to a preset interval to obtain a plurality of second frequency domain matrices; finally, according to the frequency of data repetition in the plurality of second frequency domain matrices, data fusion is performed on the data of the plurality of second frequency domain matrices to obtain a plurality of feature vectors. The frequency domain matrices in the embodiments of the present invention have the characteristics of being featureized. Therefore, when performing image block retrieval, the feature data can be directly used for retrieval, and data comparison can be completed without image restoration, improving the efficiency of image retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image data feature recognition, and in particular, to an image data fusion feature extraction method, apparatus, electronic device, and storage medium. Background Art

[0002] In public security business scenarios, capturing and searching images by image have attracted more and more attention. In general application scenarios, the local image features and global image features captured are independently modeled and retrieved, and retrieval is required in some scenarios.

[0003] The related image storage method is to compress and store images through basic image compression algorithms. When performing image retrieval, the compressed and stored images are restored, the search key image is segmented into image blocks, and the image blocks are compared with the restored images to find the images of interest.

[0004] The traditional image storage method makes the amount of data to be compared during retrieval large, resulting in either more images being compared or the target not being compared, leading to retrieval failure.

[0005] Based on this, it is necessary to develop and design an image data fusion feature extraction method. Summary of the Invention

[0006] Embodiments of the present invention provide an image data fusion feature extraction method, apparatus, electronic device, and storage medium, which are used to solve the problem of low retrieval efficiency caused by unclear image data characterization in the prior art.

[0007] In a first aspect, embodiments of the present invention provide an image data fusion feature extraction method, including:

[0008] Obtaining a plurality of image blocks, where the plurality of image blocks are obtained based on the segmentation of an image;

[0009] Respectively performing two-dimensional frequency domain feature extraction on the plurality of image blocks to obtain a plurality of first frequency domain matrices, where each first frequency domain matrix corresponds to an image block;

[0010] Respectively cropping the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices, and adjusting the data of the cropped plurality of first frequency domain matrices to a preset interval to obtain a plurality of second frequency domain matrices;

[0011] According to the frequency of data repetition in the plurality of second frequency domain matrices, performing data fusion on the data of the plurality of second frequency domain matrices to obtain a plurality of feature vectors, where each feature vector corresponds to a frequency domain matrix.

[0012] In a possible implementation manner, the two-dimensional frequency domain feature extraction of the multiple image blocks respectively to obtain multiple first frequency domain matrices includes:

[0013] Traversingly extract an image block from the multiple image blocks as a to-be-processed image block, and after each extraction is completed, perform the following steps:

[0014] Traversingly extract positions in the first frequency domain matrix, and use the row number and column number of the extracted position as the target row number and target column number;

[0015] Traversingly extract image data from the to-be-processed image block as to-be-processed image data in a predetermined order;

[0016] Generate a feature extraction operator according to the position of the to-be-processed image data in the to-be-processed image block, the target row number, and the target column number;

[0017] Add the product of the feature extraction operator and the to-be-processed image data to the operator extraction result queue;

[0018] If the traversal of the to-be-processed image block is not completed, jump to the step of traversingly extracting image data from the to-be-processed image block as to-be-processed image data in a predetermined order;

[0019] Otherwise, calculate the sum of multiple data in the operator extraction result queue;

[0020] Add the sum to the first frequency domain matrix according to the extraction position;

[0021] If the traversing extraction of the positions in the first frequency domain matrix is not completed, jump to the step of traversingly extracting positions in the first frequency domain matrix and using the row number and column number of the extracted position as the target row number and target column number.

[0022] In a possible implementation manner, the generating a feature extraction operator according to the position of the to-be-processed image data in the to-be-processed image block, the target row number, and the target column number includes:

[0023] Generate a feature extraction operator according to the first formula, the position of the to-be-processed image data in the to-be-processed image block, the target row number, and the target column number, where the first formula is:

[0024]

[0025] In the formula, is the target row number, is the target column number, is the natural constant, is the imaginary unit, is the ratio of a circle's circumference to its diameter, is the number of columns of the image block, is the number of rows of the image block, is the number of columns of the image data to be processed in the image block to be processed, is the number of rows of the image data to be processed in the image block to be processed.

[0026] In a possible implementation, the step of respectively cropping the low-amplitude edge rows and low-amplitude edge columns of the multiple first frequency-domain matrices includes:

[0027] Sequentially extract a matrix from the multiple first frequency-domain matrices as a first matrix to be determined, and perform the following steps after extraction:

[0028] Obtain a first threshold, where the first threshold is determined proportionally according to the maximum value in the first matrix to be determined in the frequency domain;

[0029] Set the values in the matrix to be determined in the frequency domain that are lower than the first threshold to zero to obtain a second matrix to be determined;

[0030] Check whether there are target edge rows and target edge columns in the second matrix to be determined, where the target edge rows and the target edge columns are respectively the edge rows and edge columns of the second matrix to be determined with all element values being zero;

[0031] If there are target edge rows or target edge columns, delete the target edge rows and target edge columns from the second matrix to be determined, use the second matrix to be determined after deleting the target edge rows and target edge columns as the second matrix to be determined, and jump to the step of checking whether there are target edge rows and target edge columns in the second matrix to be determined;

[0032] Otherwise, use the second matrix to be determined after deleting the target edge rows and target edge columns as the cropped first frequency-domain matrix.

[0033] In a possible implementation, the step of adjusting the data of the multiple cropped first frequency-domain matrices to a preset interval to obtain multiple second frequency-domain matrices includes:

[0034] Obtain the lower limit of the preset interval;

[0035] Group the elements with the same value in each first frequency-domain matrix into the same value class to obtain multiple first classes;

[0036] Count the number of the first classes in each first frequency-domain matrix to obtain multiple first counts;

[0037] Select the largest value from the multiple first counts as the interval span value;

[0038] Sequentially extract matrices from the multiple first frequency-domain matrices after cutting as matrices to be processed, and perform the following steps after extraction:

[0039] Obtain the minimum value and the maximum value of the non-zero values in the matrix to be processed as the maximum value and the minimum value respectively;

[0040] According to the second formula, the lower limit of the interval, the interval span value, the maximum value, and the minimum value, numerically adjust the multiple non-zero value elements in the matrix to be processed, and use the adjusted matrix to be processed as the second frequency-domain matrix, where the second formula is:

[0041]

[0042] In the formula, is the adjusted non-zero value element, is the lower limit of the preset interval, is the interval span value, is the non-zero value element before adjustment, is the maximum value, is the minimum value, is the rounding function.

[0043] In a possible implementation manner, the data fusion and arrangement of the data in the multiple second frequency-domain matrices according to the frequency of data repetition in the multiple second frequency-domain matrices to obtain multiple feature vectors includes:

[0044] Allocate data mapping units for multiple values in the second frequency-domain matrix according to the number of elements with the same value in the multiple second frequency-domain matrices;

[0045] Traversingly extract the second frequency-domain matrix from the multiple second frequency-domain matrices as the target second frequency-domain matrix, and after each extraction, perform the following steps:

[0046] Sequentially extract elements from the target second frequency-domain matrix as elements to be fused;

[0047] Extract the data mapping unit of the second category where the element to be fused is located and add it to the feature vector;

[0048] If the traversal of the target second frequency-domain matrix is not completed, jump to the step of sequentially extracting elements from the target second frequency-domain matrix as elements to be fused.

[0049] In a possible implementation manner, the allocation of data mapping units for multiple values in the second frequency-domain matrix according to the number of elements with the same value in the multiple second frequency-domain matrices includes:

[0050] Obtain the number of data segments;

[0051] Group the elements with the same value in the multiple second frequency domain matrices into the same value class to obtain multiple second classes;

[0052] Count the number of the multiple second classes as the count number;

[0053] Determine multiple segment lengths according to the number of data segments and the count number;

[0054] Determine multiple maximum expression values according to the multiple segment lengths, where each maximum expression value corresponds to the maximum value that can be expressed by a segment length;

[0055] Arrange the multiple maximum expression values in ascending order to obtain an expression value arrangement;

[0056] Sort the multiple second classes in descending order of the number of elements to obtain a class arrangement;

[0057] Successively take out the maximum expression value from the expression value arrangement as the target value;

[0058] Successively take out the second classes with the number of the target value from the class arrangement as multiple classes to be processed;

[0059] Arrange the multiple classes to be processed according to the values of the classes, add a discontinuous identifier before the ranking serial number, and fill it into the data mapping unit corresponding to the segment length of the target value;

[0060] If the class arrangement is not empty, jump to the step of successively taking out the second classes with the number of the target value from the class arrangement as multiple classes to be processed;

[0061] If the expression value arrangement is not empty, jump to the step of successively taking out the maximum expression value from the expression value arrangement as the target value.

[0062] In a second aspect, an image data fusion feature extraction device provided by an embodiment of the present invention is used to implement the image data fusion feature extraction method described in the above first aspect or any one of the possible implementation manners of the first aspect. The image data fusion feature extraction device includes:

[0063] An image block acquisition module, configured to acquire a plurality of image blocks, where the plurality of image blocks are obtained based on the segmentation of an image;

[0064] A frequency domain matrix construction module, configured to perform two-dimensional frequency domain feature extraction on the plurality of image blocks respectively to obtain a plurality of first frequency domain matrices, where each first frequency domain matrix corresponds to an image block;

[0065] A frequency domain matrix adjustment module, configured to crop the low-amplitude edge rows and low-amplitude edge columns of the multiple first frequency domain matrices respectively, and adjust the data of the cropped multiple first frequency domain matrices to a preset interval to obtain multiple second frequency domain matrices;

[0066] And,

[0067] A feature fusion module, configured to perform data fusion on the data of the multiple second frequency domain matrices according to the frequency of data repetition in the multiple second frequency domain matrices to obtain multiple feature vectors, where each feature vector corresponds to a frequency domain matrix.

[0068] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor. A computer program that can run on the processor is stored in the memory. When the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation manner of the first aspect above are implemented.

[0069] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation manner of the first aspect above are implemented.

[0070] The beneficial effects of the embodiment of the present invention compared with the prior art are as follows:

[0071] The embodiment of the present invention discloses an image data fusion feature extraction method. First, a plurality of image blocks are obtained, where the plurality of image blocks are obtained based on the segmentation of an image; then two-dimensional frequency domain feature extraction is respectively performed on the plurality of image blocks to obtain a plurality of first frequency domain matrices, where each first frequency domain matrix corresponds to an image block; then the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices are respectively cropped, and the data of the cropped plurality of first frequency domain matrices is adjusted to a preset interval to obtain a plurality of second frequency domain matrices; finally, data fusion is performed on the data of the plurality of second frequency domain matrices according to the frequency of data repetition in the plurality of second frequency domain matrices to obtain a plurality of feature vectors, where each feature vector corresponds to a frequency domain matrix. The embodiment of the present invention extracts two-dimensional frequency domain features, constructs a frequency domain feature matrix, and performs data fusion based on the matrix. Since the frequency domain matrix has the characteristic of being featureized, when performing image block retrieval, the feature data can be directly used for retrieval, and data comparison can be completed without image restoration, improving the efficiency of image retrieval. And since the matrix data after being featureized is smaller in amount, more storage space is saved. And when needed, inverse feature transformation can be performed according to the matrix data for image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0073] Figure 1 is a flowchart of the image data fusion feature extraction method provided by the embodiment of the present invention;

[0074] Figure 2 is a schematic diagram of the two-dimensional frequency domain feature extraction process provided by the embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of the data mapping unit construction process provided by the embodiment of the present invention;

[0076] Figure 4 is a functional block diagram of the image data fusion feature extraction device provided by the embodiment of the present invention;

[0077] Figure 5 is a functional block diagram of the electronic device provided by the embodiment of the present invention. Specific Embodiments

[0078] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0079] To make the purpose, technical solutions, and advantages of the present invention clearer, the following will be described through specific embodiments in conjunction with the drawings.

[0080] The following will elaborate on the embodiments of the present invention. This example is implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0081] Figure 1 is a flowchart of the image data fusion feature extraction method provided by the embodiment of the present invention.

[0082] As Figure 1 shown, it shows the implementation flowchart of the image data fusion feature extraction method provided by the embodiment of the present invention, which is described in detail as follows:

[0083] In step 101, multiple image patches are obtained, where the multiple image patches are obtained based on the segmentation of an image.

[0084] In step 102, two-dimensional frequency domain feature extraction is respectively performed on the multiple image patches to obtain multiple first frequency domain matrices, where each first frequency domain matrix corresponds to an image patch.

[0085] In some embodiments, step 102 includes:

[0086] Traversingly extract an image patch from the multiple image patches as the image patch to be processed, and after each extraction is completed, perform the following steps:

[0087] Traversingly extract positions in the first frequency domain matrix, and use the row number and column number of the extracted position as the target row number and target column number;

[0088] Traversingly extract image data from the image patch to be processed as the image data to be processed in a predetermined order;

[0089] Generate a feature extraction operator according to the position of the image data to be processed in the image patch to be processed, the target row number, and the target column number;

[0090] Add the product of the feature extraction operator and the image data to be processed to the operator extraction result queue;

[0091] If the traversal of the image patch to be processed is not completed, jump to the step of traversingly extracting image data from the image patch to be processed as the image data to be processed in the predetermined order;

[0092] Otherwise, calculate the sum of multiple data in the operator extraction result queue;

[0093] Add the sum to the first frequency domain matrix according to the extraction position;

[0094] If the traversing extraction of the positions in the first frequency domain matrix is not completed, jump to the step of traversingly extracting positions in the first frequency domain matrix and using the row number and column number of the extracted position as the target row number and target column number.

[0095] In some embodiments, the generating a feature extraction operator according to the position of the image data to be processed in the image patch to be processed, the target row number, and the target column number includes:

[0096] Generate a feature extraction operator according to a first formula, the position of the image data to be processed in the image patch to be processed, the target row number, and the target column number, where the first formula is:

[0097]

[0098] In the formula, is the target number of rows, is the target number of columns, is the natural constant, is the imaginary unit, is the pi, is the number of columns of the image block, is the number of rows of the image block, is the number of columns of the image data to be processed in the image block to be processed, is the number of rows of the image data to be processed in the image block to be processed.

[0099] Exemplarily, as described above, in the related art, the storage of images adopts compressed storage, and the images are not expressed or represented. When the images need to be retrieved, they need to be decompressed and restored, and then retrieved by comparing image blocks.

[0100] The embodiments of the present invention are intended to characterize images, fuse the characterized images, and store them based on the fusion results. The advantage of such storage is that when image retrieval is required, the feature fusion data can be directly extracted for image comparison, without decompression, with low computational cost, small data volume, and high comparison success rate because the data is compared according to features. Moreover, since the images are characterized, the data is compressed after characterization, occupying less storage space, and the images can be restored through the inverse process of characterization when needed. In addition, the embodiments of the present invention also perform data fusion again after the characterization process to reduce the data dimension, improve the storage space utilization rate and retrieval efficiency.

[0101] To achieve the above object, in terms of feature extraction, a frequency domain matrix is constructed for each image block respectively. Figure 2 shows the construction process of each frequency domain matrix. First, determine the position of the frequency domain matrix 201 to be filled, then sequentially extract pixel data 203 (image data) from the image block 202. According to the position of the extracted pixel data 203 in the image block and the position of the frequency domain matrix 201 to be filled, construct an operator 204, and perform a multiplication operation on the operator 204 and the pixel data 203. The obtained result is added to the operator extraction result queue 205. When the traversal of the image block 202 is completed, the operator extraction result queue 205 is constructed. Calculate the sum of all elements of the operator extraction result queue 205, and add this sum to the position of the frequency domain matrix 201 to be filled. Repeat in the above order, and this frequency domain matrix 201 is constructed.

[0102] The above operator is determined according to the first formula:

[0103]

[0104] In the formula, is the target number of rows, is the target number of columns, is the natural constant, is the imaginary unit, is the pi, is the number of columns of the image block, is the number of rows of the image block, is the number of columns of the image data to be processed in the image block to be processed, is the number of rows of the image data to be processed in the image block to be processed.

[0105] The above frequency domain matrix is actually the two-dimensional frequency domain matrix of the image block, which extracts the features of the image in different frequency domains. Through the analysis of the frequency domain matrix, we can know the different frequency domain features contained in the image. Through some representative values in the frequency domain features, the image can be retrieved and compared without image restoration, improving the retrieval efficiency and saving storage space.

[0106] In step 103, the low-amplitude edge rows and low-amplitude edge columns of the multiple first frequency domain matrices are respectively cropped, and the data of the cropped multiple first frequency domain matrices are adjusted to a preset interval to obtain multiple second frequency domain matrices.

[0107] In some embodiments, the respectively cropping the low-amplitude edge rows and low-amplitude edge columns of the multiple first frequency domain matrices includes:

[0108] Sequentially extract a matrix from the multiple first frequency domain matrices as a first matrix to be determined, and perform the following steps after extraction:

[0109] Obtain a first threshold, where the first threshold is determined according to the maximum value in the first matrix to be determined in the frequency domain;

[0110] Set the values in the matrix to be determined in the frequency domain that are lower than the first threshold to zero to obtain a second matrix to be determined;

[0111] Check whether there are target edge rows and target edge columns in the second matrix to be determined, where the target edge rows and the target edge columns are respectively the edge rows and edge columns of the second matrix to be determined with all element values being zero;

[0112] If there are target edge rows or target edge columns, delete the target edge rows and target edge columns from the second matrix to be determined, use the second matrix to be determined after deleting the target edge rows and target edge columns as the second matrix to be determined, and jump to the step of checking whether there are target edge rows and target edge columns in the second matrix to be determined;

[0113] Otherwise, the second undetermined matrix of the target edge row and the target edge column will be deleted as the first frequency domain matrix after cropping.

[0114] In some embodiments, adjusting the data of the cropped multiple first frequency domain matrices to a preset interval to obtain multiple second frequency domain matrices includes:

[0115] Obtain the lower limit of the preset interval;

[0116] Group the elements with the same value in each first frequency domain matrix into the same value class to obtain multiple first classes;

[0117] Count the number of the first classes in each first frequency domain matrix to obtain multiple first quantities;

[0118] Select the largest value from the multiple first quantities as the interval span value;

[0119] Extract matrices from the cropped multiple first frequency domain matrices in sequence as the matrices to be processed, and perform the following steps after extraction:

[0120] Obtain the minimum value and the maximum value of the non-zero values in the matrix to be processed as the maximum value and the minimum value respectively;

[0121] According to the second formula, the lower limit of the interval, the interval span value, the maximum value and the minimum value, adjust the multiple non-zero value elements in the matrix to be processed, and use the adjusted matrix to be processed as the second frequency domain matrix, where the second formula is:

[0122]

[0123] In the formula, is the adjusted non-zero value element, is the lower limit of the preset interval, is the interval span value, is the non-zero value element before adjustment, is the maximum value, is the minimum value, is the rounding function.

[0124] Exemplarily, the frequency domain feature matrix contains some low-amplitude frequency domain features. The low-amplitude frequency domain features located at the matrix edge and below a specific condition have little impact on the expression and retrieval of the entire image block. However, on the other hand, these low-amplitude edge features will affect the storage space of the image and further affect the retrieval efficiency.

[0125] Therefore, in one implementation, the feature with the largest value in the frequency domain matrix is extracted, and a threshold is set according to this value at a predetermined ratio, for example, the predetermined ratio is 0.1. Then, the two edge rows and two edge columns in the frequency domain matrix are checked. If the element values in these edge rows or edge columns are all lower than the threshold, the edge rows and edge columns are deleted. After deletion, the new edge rows and edge columns are checked again until there are elements higher than the threshold in the edge rows and edge columns of the newly generated frequency domain matrix.

[0126] After cropping the frequency domain matrix, the embodiments of the present invention also adjust the data intervals of multiple cropped frequency domain matrices, so that the data in the frequency domain matrix repeats as much as possible, facilitating higher fusion degree of the fused data and reducing the data space occupancy.

[0127] To achieve the above object, the embodiments of the present invention first perform numerical statistics on each cropped matrix to check the number of values it contains. One statistical method is to classify the same values in the cropped matrix into one category, and then check the number of categories. Obviously, each matrix corresponds to a number of values, and we use the maximum value of the number of values as the interval span value.

[0128] Then, according to the interval span value and the lower limit of the preset interval, each cropped matrix is intervalized using the second formula. The second formula is:

[0129]

[0130] In the formula, is the adjusted non-zero value element, is the lower limit of the preset interval, is the interval span value, is the non-zero value element before adjustment, is the maximum value, is the minimum value, is the rounding function.

[0131] After intervalization, a frequency domain matrix (the second frequency domain matrix) with low-amplitude feature cropping and high data overlap rate is obtained.

[0132] In step 104, according to the frequency of data repetition in the multiple second frequency domain matrices, the data of the multiple second frequency domain matrices are data-fused to obtain multiple feature vectors, where each feature vector corresponds to a frequency domain matrix.

[0133] In some embodiments, step 104 includes:

[0134] Allocating data mapping units to multiple values in the second frequency domain matrix according to the number of elements with the same value in the multiple second frequency domain matrices;

[0135] Traversingly extract a second frequency-domain matrix from the multiple second frequency-domain matrices as the target second frequency-domain matrix, and after each extraction, perform the following steps:

[0136] Sequentially extract elements from the target second frequency-domain matrix as the elements to be fused;

[0137] Extract the data mapping units of the second type where the elements to be fused are located, and add them to the feature vector;

[0138] If the traversal of the target second frequency-domain matrix is not completed, jump to the step of sequentially extracting elements from the target second frequency-domain matrix as the elements to be fused.

[0139] In some embodiments, the step of allocating data mapping units to multiple values in the second frequency-domain matrix according to the number of elements with the same value in the multiple second frequency-domain matrices includes:

[0140] Obtain the number of data segments;

[0141] Group the elements with the same value in the multiple second frequency-domain matrices into the same value class to obtain multiple second classes;

[0142] Count the number of multiple second classes as the counting number;

[0143] Determine multiple segment lengths according to the number of data segments and the counting number;

[0144] Determine multiple maximum expression values according to the multiple segment lengths, where each maximum expression value corresponds to the maximum value that a segment length can express;

[0145] Arrange the multiple maximum expression values in ascending order to obtain an expression value arrangement;

[0146] Sort the multiple second classes in descending order of the number of elements to obtain a class arrangement;

[0147] Sequentially take out the maximum expression value from the expression value arrangement as the target value;

[0148] Sequentially take out the second classes with the number of the target value from the class arrangement as multiple classes to be processed;

[0149] Arrange the multiple classes to be processed according to the values of the classes, add a discontinuous identifier to the front of the ranking number, and fill it into the data mapping units corresponding to the segment lengths of the target value;

[0150] If the class arrangement is not empty, jump to the step of sequentially taking out the second classes with the number of the target value from the class arrangement as multiple classes to be processed;

[0151] If the arrangement of the expression values is non-empty, jump to the step of sequentially taking out the maximum expression value from the arrangement of the expression values as the target value.

[0152] Exemplarily, in the embodiment of the present invention, data lengths of different lengths are used to represent elements in a matrix. Among them, data with a higher occurrence frequency is represented by shorter data, while data with a lower occurrence frequency is represented by longer data. For example, the data '0' has the highest occurrence frequency, so it is represented by 7-bit (7bit) data, and the data '237' has a lower occurrence frequency, so it is represented by 21-bit (21bit) data. The advantage of doing this is that most data is represented by short data, improving the storage efficiency. Moreover, since data with a high occurrence frequency appears in multiple matrices, in other words, these eigenvalues with a high frequency appear in multiple image blocks, the retrieval significance is much smaller than that of data with a low frequency. Therefore, when selecting key features from a matrix and screening from multiple search keywords, preferentially screening long data can improve the screening efficiency.

[0153] To achieve the above object, in the embodiment of the present invention, the values of the elements in multiple second frequency domain matrices are counted, and those belonging to the same value are classified into the same class, so that multiple classes are obtained, and the number of classes is used as the counting quantity.

[0154] Then, according to the counting quantity and the number of segments, the segment length is determined. One determination method is to use the following formula:

[0155]

[0156] In the above formula, is the counting quantity, is the number of segments, is the segment length.

[0157] After the above steps, the segment length can be determined. In fact, the segment length is the length of the data storage space after data mapping. Then, the above classes are arranged according to the number of elements in the class, and the quantity that can be expressed by the storage space of the segment length is extracted. The class with the most forward expression quantity in the arrangement is extracted, and then these classes are arranged according to the numerical size, the serial number of the arrangement is extracted, and the discontinuous identifier is added and stored in the mapped value of the class.

[0158] As Figure 3 shown, it provides a schematic diagram of the above process. Arranged according to the number of elements, a set 301 of multiple classes is obtained (the first set in the figure has 3 classes, and the second has 5 classes). Then, the set with the largest number of classes in the set 301 is assigned to the shortest data mapping unit 302.

[0159] For example, if the length of the data mapping unit is 3, then the maximum number of expressions is , at this time, among multiple classes, find the 8 classes with the most elements. For example, these 8 classes are the classes of the values 13, 21, 14, 19, 17, 16, 23, and 22. Then sort these classes of values according to the values. The sorting order is 13, 14, 16, 17, 19, 21, 22, 23. Then, the class number of the value 13 is 1, store 0 in its data mapping unit (the expression is from 0 to 7, and the first bit corresponds to 0), and add an intermittent identifier 0 at the front end to form a four-bit (4-bit) data. The binary representation is 0000 (the first 0 is the intermittent identifier). And so on, the binary representation of the data stored in the data mapping unit of 16 is 0010 (the first 0 is the intermittent identifier), and the binary representation of the data stored in the data mapping unit of 23 is 0111 (the first 0 is the intermittent identifier).

[0160] After the mapping unit is constructed, each frequency domain matrix extracts data from the matrix in turn, looks up the data in the data mapping unit and fills it into the feature vector, and then the construction of the fused feature vector is completed.

[0161] In the implementation manner of the method for extracting the fused feature of image data of the present invention, first, a plurality of image blocks are obtained, wherein the plurality of image blocks are obtained based on the segmentation of the image; then, two-dimensional frequency domain feature extraction is respectively performed on the plurality of image blocks to obtain a plurality of first frequency domain matrices, wherein each first frequency domain matrix corresponds to an image block; then, the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices are respectively cut, and the data of the cut plurality of first frequency domain matrices are adjusted to a preset interval to obtain a plurality of second frequency domain matrices; finally, according to the frequency of data repetition in the plurality of second frequency domain matrices, the data of the plurality of second frequency domain matrices are fused to obtain a plurality of feature vectors, wherein each feature vector corresponds to a frequency domain matrix. In the implementation manner of the present invention, two-dimensional frequency domain features are extracted, a frequency domain feature matrix is constructed, and data fusion is performed based on the matrix. Since the frequency domain matrix has the characteristic of being featureized, when performing image block retrieval, the feature data can be directly used for retrieval, and data comparison can be completed without image restoration, improving the efficiency of image retrieval. And because the matrix data after being featureized is smaller, more storage space is saved. And when needed, inverse feature transformation can be performed according to the matrix data for image restoration.

[0162] It should be understood that the magnitudes of the sequence numbers of the steps in the above implementation manner do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the implementation manner of the present invention.

[0163] The following is the device implementation manner of the present invention. For details not described in detail herein, reference may be made to the corresponding method implementation manner above.

[0164] Figure 4 is a functional block diagram of an image data fusion feature extraction device provided by an embodiment of the present invention. Referring to Figure 4 , the image data fusion feature extraction device includes: an image block acquisition module 401, a frequency domain matrix construction module 402, a frequency domain matrix adjustment module 403, and a feature fusion module 404, where:

[0165] The image block acquisition module 401 is configured to acquire a plurality of image blocks, where the plurality of image blocks are obtained based on segmentation of an image;

[0166] The frequency domain matrix construction module 402 is configured to perform two-dimensional frequency domain feature extraction on the plurality of image blocks respectively to obtain a plurality of first frequency domain matrices, where each first frequency domain matrix corresponds to an image block;

[0167] The frequency domain matrix adjustment module 403 is configured to crop the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices respectively, and adjust the data of the cropped plurality of first frequency domain matrices to a preset interval to obtain a plurality of second frequency domain matrices;

[0168] The feature fusion module 404 is configured to perform data fusion on the data of the plurality of second frequency domain matrices according to the frequency of data repetition in the plurality of second frequency domain matrices to obtain a plurality of feature vectors, where each feature vector corresponds to a frequency domain matrix.

[0169] Figure 5 is a functional block diagram of an electronic device provided by an embodiment of the present invention. As Figure 5 shown, the electronic device 5 of this embodiment includes: a processor 500 and a memory 501, and a computer program 502 that can run on the processor 500 is stored in the memory 501. When the processor 500 executes the computer program 502, the steps in the above-mentioned various image data fusion feature extraction methods and embodiments are implemented, such as Figure 1 the steps 101 to 104 shown.

[0170] Exemplarily, the computer program 502 may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 501 and executed by the processor 500 to complete the present invention.

[0171] The electronic device 5 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 5 may include, but is not limited to, a processor 500 and a memory 501. Those skilled in the art can understand thatFigure 5 This is only an example of the electronic device 5, which does not constitute a limitation on the electronic device 5. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the electronic device 5 may also include input / output devices, network access devices, buses, etc.

[0172] The so-called processor 500 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0173] The memory 501 may be an internal storage unit of the electronic device 5, such as the hard disk or memory of the electronic device 5. The memory 501 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 501 may also include both the internal storage unit and the external storage device of the electronic device 5. The memory 501 is used to store the computer program 502 and other programs and data required by the electronic device 5. The memory 501 may also be used to temporarily store data that has been output or is to be output.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0175] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not described or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0177] In the embodiments provided by the present invention, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0178] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in various embodiments of the present invention may be integrated into a processing unit, may exist physically as individual units, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-described embodiments of the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method and apparatus embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0181] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An image data fusion feature extraction method, characterized in that Including: Obtaining a plurality of image patches, where the plurality of image patches are obtained based on the segmentation of an image; Performing two-dimensional frequency domain feature extraction on the plurality of image patches respectively to obtain a plurality of first frequency domain matrices, including: Traversing and extracting an image patch from the plurality of image patches as a to-be-processed image patch, and after each extraction is completed, performing the following steps: Traversing and extracting positions in the first frequency domain matrix, and taking the row number and column number of the extracted position as the target row number and target column number; Traversing and extracting image data from the to-be-processed image patch as to-be-processed image data in a predetermined order; Generating a feature extraction operator according to a first formula, the position of the to-be-processed image data in the to-be-processed image patch, the target row number, and the target column number, where the first formula is: In the formula, is the target row number, is the target column number, is the natural constant, is the imaginary unit, is the pi, is the column number of the image block, is the row number of the image block, is the column number of the image data to be processed in the image block to be processed, is the row number of the image data to be processed in the image block to be processed; Adding the product of the feature extraction operator and the to-be-processed image data to an operator extraction result queue; If the traversal of the to-be-processed image patch is not completed, jump to the step of traversing and extracting image data from the to-be-processed image patch as to-be-processed image data in the predetermined order; Otherwise, calculating the sum of multiple data in the operator extraction result queue; Adding the sum to the first frequency domain matrix according to the extraction position; If the position of the first frequency domain matrix has not been traversed and extracted, jump to the step of traversing and extracting positions in the first frequency domain matrix, and taking the row number and column number of the extracted position as the target row number and target column number; Cropping the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices respectively, and adjusting the data of the cropped plurality of first frequency domain matrices to a preset interval to obtain a plurality of second frequency domain matrices; Performing data fusion on the data of the plurality of second frequency domain matrices according to the frequency of data repetition in the plurality of second frequency domain matrices to obtain a plurality of feature vectors, including: Allocating data mapping units to multiple values in the second frequency domain matrix according to the number of elements with the same value in the plurality of second frequency domain matrices; Traversing and extracting a second frequency domain matrix from the plurality of second frequency domain matrices as a target second frequency domain matrix, and after each extraction, performing the following steps: Sequentially extracting elements from the target second frequency domain matrix as elements to be fused; Extracting the data mapping unit of the element to be fused and adding it to the feature vector; If the traversal of the target second frequency domain matrix is not completed, jump to the step of sequentially extracting elements from the target second frequency domain matrix as elements to be fused.

2. The image data fusion feature extraction method according to claim 1, wherein The cropping of the low-amplitude edge rows and low-amplitude edge columns of the plurality of first frequency domain matrices respectively includes: Sequentially extracting a matrix from the plurality of first frequency domain matrices as a first to-be-determined matrix, and after extraction, performing the following steps: Obtaining a first threshold, where the first threshold is determined according to the maximum value in the first to-be-determined matrix in proportion; Setting the values in the first to-be-determined matrix that are lower than the first threshold to zero to obtain a second to-be-determined matrix; Check whether there are target edge rows and target edge columns in the second undetermined matrix, where the target edge rows and the target edge columns are respectively the edge rows and edge columns of the second undetermined matrix with multiple element values all being zero; If there are target edge rows or target edge columns, then delete the target edge rows and target edge columns from the second undetermined matrix, take the second undetermined matrix after deleting the target edge rows and target edge columns as the second undetermined matrix, and jump to the step of checking whether there are target edge rows and target edge columns in the second undetermined matrix; Otherwise, take the second undetermined matrix after deleting the target edge rows and target edge columns as the cropped first frequency domain matrix.

3. The image data fusion feature extraction method according to claim 1, wherein The adjusting the data of the cropped multiple first frequency domain matrices to a preset interval to obtain multiple second frequency domain matrices includes: Obtain the lower limit of the preset interval; Group the elements with the same value in each first frequency domain matrix into the same value class to obtain multiple first classes; Count the number of the first classes in each first frequency domain matrix to obtain multiple first numbers; Select the largest value from the multiple first numbers as the interval span value; Extract matrices from the cropped multiple first frequency domain matrices in sequence as the matrix to be processed, and perform the following steps after extraction: Obtain the minimum value and the maximum value of the non-zero values in the matrix to be processed as the maximum value and the minimum value respectively; According to the second formula, the lower limit of the interval, the interval span value, the maximum value and the minimum value, adjust the multiple non-zero value elements in the matrix to be processed, and take the adjusted matrix to be processed as the second frequency domain matrix, where the second formula is: In the formula, is the adjusted non-zero value element, is the lower limit of the preset interval, is the interval span value, is the non-zero value element before adjustment, is the maximum value, is the minimum value, is the rounding function.

4. The method for extracting image data fusion features according to claim 1, wherein The allocating data mapping units to multiple values in the second frequency domain matrix according to the number of elements with the same value in the multiple second frequency domain matrices includes: Obtain the number of data segments; Group the elements with the same value in the multiple second frequency domain matrices into the same value class to obtain multiple second classes; Count the number of the multiple second classes as the counting number; Determine multiple segment lengths according to the number of data segments and the counting number; Determine multiple maximum expression values according to the multiple segment lengths, where each maximum expression value corresponds to the maximum value that can be expressed by a segment length; Arrange the multiple maximum expression values in ascending order to obtain an expression value arrangement; Sort the multiple second classes in descending order of the number of elements to obtain a class arrangement; Take out the maximum expression value from the expression value arrangement in sequence as the target value; Take out the second classes with the number of the target value from the class arrangement in sequence as multiple classes to be processed; Arrange the multiple classes to be processed according to the value of the class, add a discontinuous identifier to the front of the ranking serial number, and fill it into the data mapping unit corresponding to the segment length of the target value; If the class arrangement is not empty, jump to the step of taking out the second classes with the number of the target value from the class arrangement in sequence as multiple classes to be processed; If the expression value arrangement is not empty, jump to the step of taking out the maximum expression value from the expression value arrangement in sequence as the target value.

5. An image data fusion feature extraction device, characterized in that, For implementing the image data fusion feature extraction method described in any one of claims 1-4, the image data fusion feature extraction device includes: An image block acquisition module, configured to acquire a plurality of image blocks, wherein the plurality of image blocks are obtained based on the segmentation of an image; A frequency domain matrix construction module, configured to perform two-dimensional frequency domain feature extraction on the plurality of image blocks respectively to obtain a plurality of first frequency domain matrices, wherein each first frequency domain matrix corresponds to an image block; A frequency domain matrix adjustment module, configured to crop the low amplitude edge rows and low amplitude edge columns of the plurality of first frequency domain matrices respectively, and adjust the data of the cropped plurality of first frequency domain matrices to a preset interval to obtain a plurality of second frequency domain matrices; And, A feature fusion module, configured to perform data fusion on the data of the plurality of second frequency domain matrices according to the frequency of data repetition in the plurality of second frequency domain matrices to obtain a plurality of feature vectors, wherein each feature vector corresponds to a frequency domain matrix.

6. An electronic device, comprising a memory and a processor, wherein a computer program that can run on the processor is stored in the memory, and is characterized in that When the processor executes the computer program, the steps of the method described in any one of claims 1 to 4 above are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method described in any one of claims 1 to 4 above are implemented.

Images