Deep learning data compression method and device for image storage

Abstract

The invention provides a deep learning data compression method and device for image storage, and the method comprises the steps: extracting a plurality of high-dimensional image features from a subset for each subset in a plurality of similar data subsets, and classifying and screening the image data in each subset according to the distance between each high-dimensional image feature and the similarity to obtain a new similar data subset, decoding the new similar data subset, and sequentially performing image fusion and image splicing on the images with the similarity greater than a second distance threshold in the decoded similar data subset to obtain a first to-be-coded image, carrying out image splicing on image pairs with the distance smaller than a second distance threshold in the decoded similar data subsets to obtain a second to-be-coded image; and compressing the first to-be-coded image and the second to-be-coded image to obtain a compressed data set. According to the method, the data redundancy is effectively reduced, the coding efficiency and the image compression accuracy are improved, the image data compression ratio is improved, and the data storage and transmission cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image compression technology, and in particular to a deep learning data compression method and device for image storage. Background Art

[0002] With the popularization and application of various digital products and visual sensors, the number of images on the Internet has increased dramatically. With the development of multimedia technology, the resolution of various views is also constantly improving, and the size of images is also increasing, which has brought huge storage pressure to users and enterprises.

[0003] Compared with other data types such as text data, database data, and log data, image data contains a large amount of redundant data. This is mainly because objects in nature and artificial scenes often have similar shapes and colors, which leads to local consistency, resulting in similarities between adjacent pixels of image data, repeated elements in image content, and the coexistence of high-frequency and low-frequency information; in addition, the uneven frequency of occurrence of certain pixel values ​​in the image, coupled with the correlation between brightness and chromaticity in the color model, further increases redundancy.

[0004] Image redundancy removal methods in related technologies usually digitally perceive the color and structure of graphics to determine the similarity of content, and further implement differential storage. These methods are prone to misjudge similar but actually different graphic data obtained by digital encoding of the image as redundant, and erroneously remove data, thereby reducing the compression accuracy. Summary of the invention

[0005] The present invention provides a deep learning data compression method and device for image storage, which is used to solve the problem that the existing technology digitally perceives the color and structure of graphics and judges the content similarity, further realizes differential storage, and easily misjudges similar but actually different graphic data obtained by digital encoding of images as redundant, resulting in low compression accuracy, thereby improving the image compression accuracy.

[0006] The present invention provides a deep learning data compression method for image storage, comprising: For each of the multiple similar data subsets, multiple high-dimensional image features are extracted from the subset, and the image data in each subset is classified and screened according to the similarity based on the distance between the high-dimensional image features to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; Decoding the new similar data subset to obtain a decoded similar data subset, and sequentially performing image fusion and image stitching on images in the decoded similar data subset whose similarity is greater than a second distance threshold to obtain a first image to be encoded, and performing image stitching on image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; The first to-be-encoded image and the second to-be-encoded image are compressed to obtain a compressed data set.

[0007] According to a deep learning data compression method for image storage provided by the present invention, the multiple similar data subsets are obtained by the following steps: Performing image scaling on the image data set to be processed and converting it into a grayscale image; Performing discrete cosine transform (DCT) on the grayscale image and extracting multiple image features; The hash values ​​of the multiple image features are calculated respectively according to the perceptual hash algorithm, and the image data set to be processed is segmented according to the distance between the hash values ​​of each image feature to obtain the multiple similar data subsets.

[0008] According to a deep learning data compression method for image storage provided by the present invention, the image data in each subset is classified and screened according to similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset including: Calculate the distance measurement parameters between each high-dimensional image feature; Classifying the image data in each subset according to the distance measurement parameter, the first distance threshold and the second distance threshold to obtain classified similar data subsets; Image data whose distance measurement parameter is less than the first distance threshold are eliminated from the classified similar data subset to obtain the new similar data subset.

[0009] According to a deep learning data compression method for image storage provided by the present invention, decoding the new similar data subset to obtain the decoded similar data subset includes: Decoding the new similar data subset by an entropy decoder to obtain the decoded similar data subset; Compressing the first to-be-encoded image and the second to-be-encoded image to obtain a compressed data set includes: The first to-be-encoded image and the second to-be-encoded image are compressed by a differential compression encoder to obtain the compressed data set.

[0010] According to a deep learning data compression method for image storage provided by the present invention, the first image to be encoded and the second image to be encoded include image blocks; Compressing the first to-be-encoded image and the second to-be-encoded image by the differential compression encoder to obtain the compressed data set includes: The relevant areas in the image block are encoded once by a differential compression encoder, and the other data sets in the first image to be encoded and the second image to be encoded except the image block are subjected to target encoding operations to obtain the compressed data set; wherein the target encoding includes at least one of DCT transformation, quantization, run-length encoding, variable-length integer encoding, differential encoding and Huffman coding.

[0011] According to a deep learning data compression method for image storage provided by the present invention, extracting multiple high-dimensional image features from the subset includes: Feature extraction is performed on the subset based on a pre-trained ResNet model to obtain the multiple high-dimensional image features.

[0012] The present invention also provides a deep learning data compression device for image storage, comprising: A screening module is used to extract multiple high-dimensional image features from each subset of multiple similar data subsets, and classify and screen the image data in each subset according to the similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; an integration module, configured to decode the new similar data subset to obtain a decoded similar data subset, and sequentially perform image fusion and image stitching on images in the decoded similar data subset whose similarity is greater than a second distance threshold to obtain a first image to be encoded, and perform image stitching on image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; A compression module is used to compress the first image to be encoded and the second image to be encoded to obtain a compressed data set.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the deep learning data compression method for image storage as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the deep learning data compression methods for image storage as described above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the deep learning data compression methods for image storage as described above.

[0016] The deep learning data compression method and device for image storage provided by the present invention classify and screen the images in each subset by extracting the distances between multiple high-dimensional image features from multiple similar data subsets, decode the screened data subsets, and perform image fusion and image stitching in sequence on images whose similarity is greater than a second distance threshold, perform image stitching on image pairs in the subset whose distance is less than the second distance threshold, and finally compress the image to be encoded, thereby effectively reducing data redundancy, improving encoding efficiency and image compression accuracy, and at the same time improving the image data compression ratio and reducing data storage and transmission costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the deep learning data compression method for image storage provided by the present invention.

[0019] Figure 2 This is the second flow chart of the deep learning data compression method for image storage provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the deep learning data compression device for image storage provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 3 The present invention describes a deep learning data compression method and device for image storage.

[0024] Figure 1 This is one of the flow charts of the deep learning data compression method for image storage provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: for each of the multiple similar data subsets, extract multiple high-dimensional image features from the subset, and classify and screen the image data in each subset according to similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by using a perceptual hashing algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold.

[0025] In this step, the perceptual hash function can be used to convert the initialized segmented regions in the image data set to be processed into binary fingerprints, so as to compare the similarities between regions, merge or segment the regions based on the similarities, and finally obtain multiple similar data sets.

[0026] In this step, the target neural network model may include but is not limited to a convolutional neural network. For example, the target neural network model is a ResNet (Deep Residual Learning Network) model.

[0027] Specifically, extracting multiple high-dimensional image features from the subset includes: extracting features from the subset based on a pre-trained ResNet model to obtain multiple high-dimensional image features for identifying deep similarities in image content.

[0028] In this embodiment, the distance between image features is calculated and then compared with a preset distance threshold to determine the similarity of each image data. Based on the comparison between the similarity and the distance threshold, the image data with a similarity higher than the distance threshold is further divided according to the attributes of "similar" and "convergent". For image data with a similarity lower than the distance threshold, no de-redundancy processing is performed in the current image processing batch.

[0029] For example, after extracting multiple high-dimensional image features from the subset, the Euclidean distance between the image features is calculated, and then the Euclidean distance is compared with different distance thresholds to obtain image pairs with different similarities, and the image data with similarity values ​​lower than the preset distance threshold is eliminated to obtain a new similar data subset.

[0030] Step 120: decode the new similar data subset to obtain a decoded similar data subset, and perform image fusion and image stitching in sequence on the images in the decoded similar data subset whose similarity is greater than the second distance threshold to obtain a first image to be encoded, and perform image stitching on the image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold.

[0031] It should be noted that, due to the different redundancy levels of different image data, quantitative analysis can be performed based on the above-mentioned similarity. After distance comparison, for image pairs with higher similarity, they can be integrated through image fusion and stitching, and then the integrated images can be de-redundant. For image pairs with the second highest similarity, image integration can be performed only through image stitching, and then the integrated images can be de-redundant, which can effectively reduce data redundancy.

[0032] In this step, the image decoding methods include but are not limited to entropy decoding, inverse DCT (Discrete Cosine Transform), inverse wavelet transform, predictive decoding, inverse process of transform coding, lossless decoding or lossy decoding and other methods.

[0033] In this step, the values ​​of the first distance threshold and the second distance threshold may be set according to user requirements. For example, the second distance threshold may be set to 0.9, and the first distance threshold may be set to 0.7.

[0034] In this embodiment, image features are first extracted using a deep learning model (such as a convolutional neural network) or traditional image processing technology. The similarity between feature vectors is calculated to find image pairs with high similarity, and then, according to specific needs and application scenarios, a suitable image fusion method (such as pixel-level fusion, feature-level fusion, or decision-level fusion) is selected to fuse images with high similarity; for images that need to be spliced, first a registration process is performed to ensure that they can be correctly aligned when spliced; then, according to the content and layout of the image, a suitable splicing method is selected, such as horizontal splicing, vertical splicing, or a custom splicing mode, and then the selected images are spliced ​​using an image processing library in an image processing software or programming language.

[0035] In one embodiment, a threshold K may be set to judge the degree of similarity of images from low to high as dissimilar, similar, and convergent. For a data set, dissimilar images mean a perceived hash matching error, and graphic data that is similar in digital coding but actually different is misjudged as redundant. Convergent images are decoded and then fused and de-redundant. Similar images are decoded and then spliced ​​and de-redundant. For the same data set, if both convergent and similar features appear at the same time, the convergent images may be decoded first and then fused and de-redundant, and then the fused images may be spliced ​​and de-redundant.

[0036] Step 130: compress the first image to be encoded and the second image to be encoded to obtain a compressed data set.

[0037] In this step, the image compression methods include but are not limited to DCT transformation, quantization, run-length coding, variable-length integer coding, differential coding and Huffman coding.

[0038] In this embodiment, similar image data generated by perceptual hashing and convolutional neural network matching are decoded to restore the original code before compression coding. The images are then fused and spliced, and then uniformly compressed and coded to avoid the redundancy of pre-coding data between different images caused by compression coding being concealed.

[0039] In this embodiment, after obtaining the above-mentioned new similar data subset, each image is first decoded once to restore the original code before compression encoding, and then the decoded images are subjected to at least one of image fusion and image splicing according to the similarity through the method of the above-mentioned step 120 to obtain an integrated image, and finally the integrated image is compression encoded to obtain a compressed data set.

[0040] The deep learning data compression method for image storage provided by the embodiment of the present invention classifies and screens the images in each subset by extracting the distances between multiple high-dimensional image features from multiple similar data subsets, decodes the screened data subsets, and sequentially performs image fusion and image stitching on images whose similarity is greater than a second distance threshold, performs image stitching on image pairs in the subset whose distance is less than the second distance threshold, and finally compresses the image to be encoded, thereby effectively reducing data redundancy, improving encoding efficiency and image compression accuracy, and at the same time improving the image data compression ratio and reducing data storage and transmission costs.

[0041] In some embodiments, multiple similar data subsets are obtained by the following steps: performing image scaling on the image data set to be processed and converting it into a grayscale image; performing discrete cosine transform (DCT) on the grayscale image and extracting multiple image features; calculating hash values ​​of multiple image features respectively according to the perceptual hash algorithm, and segmenting the image data set to be processed according to the distance between the hash values ​​of each image feature to obtain multiple similar data subsets.

[0042] In this embodiment, Python tools and common image processing libraries (such as PIL or OpenCV) can be used to scale the images to the same target size (eg, 32 mm×32 mm) to eliminate the impact of different resolutions.

[0043] In this embodiment, a grayscale image conversion tool (e.g., cv2.cvtColor function of OpenCV) can be used to convert a color image into a grayscale image to reduce computational complexity; a DCT result can be obtained by calculating a two-dimensional fast Fourier transform (FFT) of the grayscale image and then performing a mathematical transformation; or the scipy.fftpack.dct function in the SciPy library can be used to directly perform a two-dimensional DCT transformation on the grayscale image.

[0044] It should be noted that since the DCT transformation result is a matrix of the same size as the original image, the low-frequency coefficients are usually located in the upper left corner of the matrix; this embodiment extracts these low-frequency coefficients by selecting a sub-region of the matrix, that is, extracts the first N low-frequency coefficients from the DCT transformation result as image features.

[0045] In this embodiment, the hash value corresponding to the image feature is calculated by the perceptual hash algorithm, and the similarity of each image is measured by calculating the Hamming distance between two hash values. Finally, the original image data set to be processed is divided into multiple digitally perceptually similar subsets, i.e., similar data subsets, by comparing the similarity of each image with a preset threshold.

[0046] The deep learning data compression method for image storage provided by the embodiment of the present invention divides a relatively large image data set to be processed into multiple similar data subsets through a hash-aware algorithm, so as to reduce the size of the data set that is subsequently processed by a feature extraction algorithm in a single process, improve the algorithm processing efficiency, and thereby improve the image compression efficiency.

[0047] In some embodiments, the image data in each subset is classified and screened according to similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset, which includes: calculating the distance measurement parameter between each high-dimensional image feature; classifying the image data in each subset according to the distance measurement parameter, a first distance threshold and a second distance threshold to obtain a classified similar data subset; and eliminating image data with a distance measurement parameter less than the first distance threshold from the classified similar data subset to obtain a new similar data subset.

[0048] In this embodiment, the distance metric parameter includes but is not limited to cosine similarity, Euclidean distance, Manhattan distance or Hamming distance.

[0049] Taking the Euclidean distance as an example, the Euclidean distance between any two image features in each subset is first calculated, and compared at multiple distance thresholds; for example, the distance thresholds are set to 0.7 and 0.9 respectively. When the Euclidean distance L between the two image features satisfies: 0.7≤L<0.9, the image pair is a "similar" image; if it satisfies: L≥0.9, the image pair is a "convergent" image; if it satisfies L<0.7, the image pair is a "dissimilar" image; retain the "convergent" images and "similar" images in each subset, remove the "dissimilar" images, and obtain a new similar data subset.

[0050] The deep learning data compression method for image storage provided by the embodiment of the present invention calculates the distance measurement parameter between each high-dimensional image feature; classifies the image data in each subset according to the distance measurement parameter, the first distance threshold and the second distance threshold to obtain a classified similar data subset; removes the image data whose distance measurement parameter is less than the first distance threshold from the classified similar data subset to obtain a new similar data subset, thereby further reducing the interference of irrelevant images on image compression and improving image compression efficiency.

[0051] In some embodiments, decoding a new similar data subset to obtain a decoded similar data subset includes: decoding the new similar data subset through an entropy decoder to obtain a decoded similar data subset; compressing a first image to be encoded and a second image to be encoded to obtain a compressed data set includes: compressing the first image to be encoded and the second image to be encoded through a differential compression encoder to obtain a compressed data set.

[0052] In this embodiment, the entropy decoder uses the entropy redundancy between information (i.e., large amounts of information are represented by long code streams, and small amounts of information are represented by short code streams) to recover the original image data from the compressed bit stream; during the decoding process, the entropy decoder first parses the coded information in the bit stream, and recovers various parts of the image data based on the information, including parsing the quantized transform coefficients, performing inverse quantization and inverse transformation operations, so as to reconstruct the residual data of the image; finally, these residual data are added to the prediction data (generated by intra-frame prediction or inter-frame prediction) to obtain the final decoded image.

[0053] In this embodiment, the differential compression encoder can adopt encoding methods such as differential pulse code modulation DPCM or adaptive differential pulse code modulation ADPCM; in DPCM encoding, the encoder predicts the value of the next pixel (usually based on the previous encoded and decoded pixel value) and quantizes the difference between the actual value and the predicted value, which will be encoded and transmitted or stored; at the decoding end, the decoder receives this difference and adds it to the previous predicted value to reconstruct the original pixel value.

[0054] Taking the differential compression encoder using differential pulse code modulation DPCM to encode a decoded data set as an example, the first image to be encoded and the second image to be encoded include image blocks; compressing the decoded data set by the differential compression encoder to obtain a compressed data set includes: single encoding of relevant areas in the image blocks by the differential compression encoder, and target encoding operations are performed on other data sets in the first image to be encoded and the second image to be encoded except the image blocks to obtain a compressed data set; wherein the target encoding includes at least one of DCT transform, quantization, run-length coding, variable-length integer coding, differential coding and Huffman coding.

[0055] The deep learning data compression method for image storage provided by the embodiment of the present invention decodes the first image to be encoded and the second image to be encoded through an entropy decoder, and compresses the decoded data set through a differential compression encoder, thereby making full use of the characteristics of the image file, exploring the redundant data of the image, and improving the compression ratio of the image encoding.

[0056] Figure 2 This is the second flow chart of the deep learning data compression method for image storage provided by the present invention. Figure 2In the illustrated embodiment, the perceptual hash algorithm (corresponding to the calculation of perceptual hash) divides the image data set to be processed into multiple similar data subsets (corresponding to the digital perceptual similar image data set), obtains a deep feature similar data set by calculating the high-dimensional features corresponding to each image data in the subset, and then determines the correlation between different image pairs by the distance between the high-dimensional features (corresponding to CNN feature matching; then, images of different similarities (including convergent, similar and dissimilar) are determined by similarity threshold determination, and then the images of different similarities are decoded by an entropy decoder to obtain a decoded data set; in the decoded data set, image fusion and image splicing (corresponding to image splicing) are performed in sequence on the convergent image pairs, image splicing is directly performed on the similar image pairs, and the dissimilar image pairs are defaulted to "perceptual hash matching misalignment" and no compression processing is performed, and finally the fused or spliced ​​data set is compressed by a differential compression encoder to obtain a compressed data set.

[0057] The deep learning data compression device for image storage provided by the present invention is described below. The deep learning data compression device for image storage described below and the deep learning data compression method for image storage described above can be referenced to each other.

[0058] Figure 3 is a schematic diagram of the structure of the deep learning data compression device for image storage provided by the present invention, such as Figure 3 As shown, the deep learning data compression device for image storage includes: a screening module 310, an integration module 320 and a compression module 330.

[0059] The screening module 310 is used to extract multiple high-dimensional image features from each subset of the multiple similar data subsets, and classify and screen the image data in each subset according to the similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to the first distance threshold; The integration module 320 is used to decode the new similar data subset to obtain a decoded similar data subset, and perform image fusion and image stitching on the images in the decoded target similar data subset whose similarity is greater than the second distance threshold in sequence to obtain a first image to be encoded, and perform image stitching on the image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; The compression module 330 is used to compress the first image to be encoded and the second image to be encoded to obtain a compressed data set.

[0060] The deep learning data compression device for image storage provided by the embodiment of the present invention classifies and screens the images in each subset by extracting the distances between multiple high-dimensional image features from multiple similar data subsets, decodes the screened data subsets, and sequentially performs image fusion and image stitching on images whose similarity is greater than a second distance threshold, performs image stitching on image pairs in the subset whose distance is less than the second distance threshold, and finally compresses the image to be encoded, thereby effectively reducing data redundancy, improving encoding efficiency and image compression accuracy, and at the same time improving the image data compression ratio and reducing data storage and transmission costs.

[0061] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a deep learning data compression method for image storage, which includes: for each subset of multiple similar data subsets, extracting multiple high-dimensional image features from the subset, and classifying and screening the image data in each subset according to the similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; the distance between each image pair in the new similar data subset is greater than or equal to the first distance threshold; decoding the new similar data subset to obtain a decoded similar data subset, and performing image fusion and image stitching on the images in the decoded similar data subset whose similarity is greater than the second distance threshold in sequence to obtain a first image to be encoded, and performing image stitching on the image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; compressing the first image to be encoded and the second image to be encoded to obtain a compressed data set.

[0062] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0063] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deep learning data compression method for image storage provided by the above methods, the method including: for each subset of multiple similar data subsets, extracting multiple high-dimensional image features from the subset, and classifying and screening the image data in each subset according to the similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by a perceptual hashing algorithm; the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; decoding the new similar data subset to obtain a decoded similar data subset, and performing image fusion and image stitching on the images in the decoded similar data subset whose similarity is greater than the second distance threshold in sequence to obtain a first image to be encoded, and performing image stitching on the image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; compressing the first image to be encoded and the second image to be encoded to obtain a compressed data set.

[0064] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the deep learning data compression method for image storage provided by the above-mentioned methods, the method comprising: for each subset of multiple similar data subsets, extracting multiple high-dimensional image features from the subset, and classifying and screening the image data in each subset according to the similarity based on the distance between each high-dimensional image feature, to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by a perceptual hashing algorithm; the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; the new similar data subset is decoded to obtain a decoded similar data subset, and the images in the decoded similar data subset whose similarity is greater than the second distance threshold are image fused and image stitched in sequence to obtain a first image to be encoded, and the image pairs in the decoded similar data subset whose distance is less than the second distance threshold are image stitched to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; the first image to be encoded and the second image to be encoded are compressed to obtain a compressed data set.

[0065] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning data compression method for image storage, characterized in that: include: For each of the multiple similar data subsets, multiple high-dimensional image features are extracted from the subset, and the image data in each subset is classified and screened according to the similarity based on the distance between the high-dimensional image features to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; Decoding the new similar data subset to obtain a decoded similar data subset, and sequentially performing image fusion and image stitching on image pairs in the decoded similar data subset whose distance is greater than a second distance threshold to obtain a first image to be encoded, and performing image stitching on image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; The first to-be-encoded image and the second to-be-encoded image are compressed to obtain a compressed data set.

2. The deep learning data compression method for image storage according to claim 1, characterized in that: The multiple similar data subsets are obtained by the following steps: Performing image scaling on the image data set to be processed and converting it into a grayscale image; Performing discrete cosine transform (DCT) on the grayscale image and extracting multiple image features; The hash values ​​of the multiple image features are calculated respectively according to the perceptual hash algorithm, and the image data set to be processed is segmented according to the distance between the hash values ​​of each image feature to obtain the multiple similar data subsets.

3. The deep learning data compression method for image storage according to claim 1, characterized in that: The image data in each subset is classified and screened according to the similarity based on the distance between each high-dimensional image feature, and a new similar data subset is obtained, including: Calculate the distance measurement parameters between each high-dimensional image feature; Classifying the image data in each subset according to the distance measurement parameter, the first distance threshold and the second distance threshold to obtain classified similar data subsets; Image data whose distance measurement parameter is less than the first distance threshold are eliminated from the classified similar data subset to obtain the new similar data subset.

4. The deep learning data compression method for image storage according to claim 1, characterized in that: The decoding of the new similar data subset to obtain the decoded similar data subset comprises: Decoding the new similar data subset by an entropy decoder to obtain the decoded similar data subset; Compressing the first to-be-encoded image and the second to-be-encoded image to obtain a compressed data set includes: The first image to be encoded and the second image to be encoded are compressed by a differential compression encoder to obtain the compressed data set.

5. The deep learning data compression method for image storage according to claim 4, characterized in that: The first to-be-encoded image and the second to-be-encoded image include image blocks; The compressing the first to-be-encoded image and the second to-be-encoded image by a differential compression encoder to obtain the compressed data set comprises: The differential compression encoder is used to perform a single encoding on the relevant area in the image block, and a target encoding operation is performed on other data sets in the first image to be encoded and the second image to be encoded except the image block, so as to obtain the compressed data set; wherein the target encoding includes at least one of DCT transformation, quantization, run-length encoding, variable-length integer encoding, differential encoding and Huffman coding.

6. The deep learning data compression method for image storage according to claim 1, characterized in that: The extracting a plurality of high-dimensional image features from the subset comprises: Feature extraction is performed on the subset based on a pre-trained ResNet model to obtain the multiple high-dimensional image features.

7. A deep learning data compression device for image storage, characterized in that: include: A screening module is used to extract multiple high-dimensional image features from each subset of multiple similar data subsets, and classify and screen the image data in each subset according to the similarity based on the distance between each high-dimensional image feature to obtain a new similar data subset; wherein the multiple similar data subsets are obtained by segmenting the image data set to be processed by the perceptual hash algorithm; and the distance between each image pair in the new similar data subset is greater than or equal to a first distance threshold; an integration module, configured to decode the new similar data subset to obtain a decoded similar data subset, and sequentially perform image fusion and image stitching on images in the decoded similar data subset whose similarity is greater than a second distance threshold to obtain a first image to be encoded, and perform image stitching on image pairs in the decoded similar data subset whose distance is less than the second distance threshold to obtain a second image to be encoded; wherein the first distance threshold is less than the second distance threshold; A compression module is used to compress the first image to be encoded and the second image to be encoded to obtain a compressed data set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the deep learning data compression method for image storage as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep learning data compression method for image storage as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the deep learning data compression method for image storage as described in any one of claims 1 to 6 is implemented.

Images