Light field image data set processing method and system based on differential privacy

Abstract

The invention relates to the technical field of image processing, and discloses a light field image data set processing method and system based on differential privacy, and the method comprises the steps: grouping a light field image data set to obtain a light field image data group and a group number, numbering sub-images to obtain an image number, and carrying out the pixel extraction to obtain a first pixel matrix; encoding the image number, the first pixel matrix and the group number to obtain one-dimensional encoding data of the sub-image, and storing the one-dimensional encoding data; processing the one-dimensional coded data based on a differential privacy processing algorithm to obtain one-dimensional coded data of the sub-images; and rearranging the sub-images based on the processed one-dimensional coding data of the sub-images to obtain a light field image data set after differential privacy protection. Completeness of light field image array information is ensured through one-dimensional coding and rearrangement, the problem that position information is lost due to the fact that differential privacy processing is independently carried out on independent images is avoided, and the availability of a light field image data set is ensured while privacy is protected.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a light field image data set processing method and system based on differential privacy. Background Art

[0002] Image protection methods in related technologies, such as data encryption, have problems such as high computational overhead and low data utility, while methods based on image perturbations such as noise, blur, and pixelation can be easily restored by deep learning models.

[0003] At present, differential privacy (DP) technology can solve the problems mentioned in the above-mentioned related technologies by adding controllable noise to protect privacy. Image differential privacy protection methods mainly include two categories: spatial domain processing and frequency domain processing. Among them, the frequency domain-based processing method decomposes image pixel information through Fourier transform, wavelet transform and other means to perform differentiated privacy protection for different frequency bands. However, the existing image differential privacy protection technology mainly processes independent images and cannot effectively protect image data with spatial correlation, which will cause the position information of the light field image to be lost or confused, resulting in the inability of subsequent refocusing, three-dimensional imaging and other applications to correctly restore the light field of a lens group, affecting the availability of the light field data set. Summary of the invention

[0004] In view of this, the present invention provides a light field image dataset processing method and system based on differential privacy to solve how to effectively improve the availability of the light field image dataset in the process of light field image protection.

[0005] In a first aspect, the present invention provides a method for processing a light field image dataset based on differential privacy, the method comprising:

[0006] Grouping the light field image data sets according to the shooting time sequence and the lens array of the light field camera to which they belong, to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group includes a plurality of sub-images shot by the light field camera at the same time;

[0007] Numbering all sub-images in each light field image data group to obtain image numbers;

[0008] Performing pixel extraction on all sub-images in each light field image data group to obtain a first pixel matrix;

[0009] Encoding the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store the data;

[0010] The one-dimensional coded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing;

[0011] Based on the one-dimensional encoded data of each sub-image after processing, each sub-image is rearranged according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection.

[0012] The embodiment of the present invention groups light field images according to shooting time and lens array, and numbers and extracts pixels of sub-images in each group, and then performs one-dimensional encoding and differential privacy processing, and finally rearranges them according to the lens array, so as to protect the privacy of light field images while maintaining the spatial correlation between sub-images. Because the group processing method is adopted, the problem of position information loss caused by differential privacy processing of independent images is avoided. At the same time, the integrity of the light field image array information is ensured through one-dimensional encoding and rearrangement, thereby solving the problem that the existing technology cannot effectively protect image data with spatial correlation, and achieving the availability of light field image data sets while protecting privacy, so that subsequent applications such as refocusing and three-dimensional imaging can correctly restore the light field.

[0013] In an optional implementation, encoding the image number, the first pixel matrix, and the group number to obtain and store one-dimensional coded data of each sub-image specifically includes:

[0014] Associating the image number, the first pixel matrix and the group number according to the lens array, obtaining and storing sub-image metadata of each associated sub-image;

[0015] The sub-image metadata is in the form of a key-value pair, the key is path information consisting of at least an image number and a group number, and the value is the first pixel matrix.

[0016] The embodiment of the present invention associates the image number, pixel matrix and group number according to the lens array, and stores the sub-image metadata in the form of a key-value pair, where the key contains the path information of the image and the value is the pixel matrix, so that the position information and pixel data of each sub-image can correspond one to one. The key-value pair storage method avoids the confusion of position information during data storage, ensures that sub-images in different groups and positions can be accurately identified, and provides a data organization basis for subsequent differential privacy processing at the data storage level, which helps to achieve efficient processing of light field image data sets.

[0017] In an optional implementation, the one-dimensional coded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing, specifically including:

[0018] Aggregate metadata of sub-images in the same group based on the group number;

[0019] Reading a first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information;

[0020] Performing wavelet transform on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image;

[0021] Selecting the sub-image at the center position of each group as a reference image, and using the frequency domain matrix of the reference image as a feature matrix;

[0022] Generate a noise matrix based on the feature matrix of the reference image, and add the noise matrix to the frequency domain matrices of other sub-images of the non-reference image in the same group;

[0023] Performing inverse wavelet transform on the frequency domain matrices of all sub-images after adding noise to obtain a processed second pixel matrix;

[0024] The image number, the second pixel matrix and the group number are encoded to obtain one-dimensional encoded data after each sub-image is processed.

[0025] The embodiment of the present invention performs wavelet transform on sub-images in the same group to obtain a frequency domain matrix, selects the sub-image at the center position as a reference to generate a noise matrix, then adds noise to other sub-images, and finally performs an inverse transform, so that the sub-images in the same group can maintain relatively consistent noise characteristics, avoiding the destruction of spatial correlation caused by inconsistent noise of each sub-image. At the same time, noise addition in the frequency domain is realized through wavelet transform, the visual quality of the image is guaranteed, the spatial correlation of the light field image is maintained while protecting privacy, and the usability of the processed light field image data set in practical applications is improved.

[0026] In an optional implementation, the key in the sub-image metadata is path information consisting of at least an image number, an image encoding method, and a group number; if the image encoding method is in YUV format, then performing pixel extraction on all sub-images in each light field image data group to obtain a first pixel matrix specifically includes:

[0027] Encode each sub-image in YUV format to obtain the brightness component Y and the chrominance component UV;

[0028] Pixels of the brightness component Y are extracted to obtain a first pixel matrix, and the chrominance component UV is stored in a preset path according to the path information.

[0029] The embodiment of the present invention encodes the sub-image in YUV format, processes the brightness component Y and the chrominance component UV separately, and stores the UV component separately, so that the differential privacy processing can be concentrated on the brightness component, avoiding redundant processing of the chrominance component, reducing the computational overhead, and ensuring that the chrominance information is also complete, thereby improving the processing efficiency while ensuring the visual quality of the image.

[0030] In an optional implementation, the one-dimensional coded data of each sub-image after processing is rearranged according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection, specifically including:

[0031] Read the chroma component UV corresponding to each sub-image from the preset path;

[0032] Restore the processed one-dimensional encoded data to the brightness component Y;

[0033] The restored brightness component Y is combined with the corresponding chrominance component UV to obtain a sub-image in YUV format;

[0034] All sub-images in YUV format are arranged according to the lens array of the light field camera to obtain a light field image dataset with differential privacy protection.

[0035] The embodiment of the present invention re-combines the processed brightness component Y with the originally stored chrominance component UV and arranges them according to the lens array, so that the light field image after differential privacy protection can be completely restored. The method of component separation and re-combination is adopted to avoid image distortion caused by unnecessary processing of chrominance information. At the same time, the spatial structure of the light field image is further guaranteed to be intact by arranging according to the lens array. Thus, the visual quality and spatial information of the image are maintained to the greatest extent while protecting privacy, thereby improving the availability of the processed light field image dataset.

[0036] In an optional implementation, generating a noise matrix based on a feature matrix of a reference image specifically includes:

[0037] The feature matrix of the reference image is weighted based on a preset differential privacy protection strength parameter;

[0038] Generate a Laplace distributed noise matrix based on the weighted feature matrix.

[0039] The embodiment of the present invention performs weighted processing on the feature matrix of the reference image and generates a Laplace distributed noise matrix, so that the added noise intensity can be freely adjusted according to the differential privacy protection requirements. The weighted noise generation method based on the feature matrix avoids the redundant disturbance that may be caused by random noise. At the same time, the Laplace distribution is used to ensure that the statistical characteristics of the noise meet the differential privacy requirements. The combination of the two achieves a balance between protecting privacy and maintaining data availability, so that the restored light field dataset can both protect privacy and improve data availability.

[0040] In a second aspect, the present invention provides a light field image dataset processing system based on differential privacy, the system comprising a data upload module and a MapReduce module;

[0041] The data uploading module is used to group the light field image data set according to the shooting time sequence and the lens array of the light field camera to which it belongs, to obtain multiple light field image data groups and group numbers, wherein each light field image data group contains multiple sub-images shot by the light field camera at the same time; number all the sub-images in each light field image data group to obtain an image number; perform pixel extraction on all the sub-images in each light field image data group to obtain a first pixel matrix; encode the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store it;

[0042] The MapReduce module includes a Map unit and a Reduce unit;

[0043] The Map unit is used to process the one-dimensional coded data based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing;

[0044] The Reduce unit is used to rearrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each sub-image after processing, so as to obtain a light field image data set after differential privacy protection.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the light field image dataset processing method based on differential privacy according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the light field image dataset processing method based on differential privacy according to the first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the light field image dataset processing method based on differential privacy according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a first flow chart of a method for processing a light field image dataset based on differential privacy according to an embodiment of the present invention;

[0050] Figure 2 is a second flow chart of a method for processing a light field image dataset based on differential privacy according to an embodiment of the present invention;

[0051] Figure 3 is a third flow chart of a light field image dataset processing method based on differential privacy according to an embodiment of the present invention;

[0052] Figure 4 is a fourth flow chart of a light field image dataset processing method based on differential privacy according to an embodiment of the present invention;

[0053] Figure 5 is a structural block diagram of a light field image dataset processing system based on differential privacy according to an embodiment of the present invention;

[0054] Figure 6 is a fifth flow chart of a light field image dataset processing system based on differential privacy according to an embodiment of the present invention;

[0055] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] Light-field image: It is taken by a light-field camera array. It records the position and direction information of light in space through the lens array, and calculates the three-dimensional spatial coordinates through light-field imaging technology, making up for the deficiency that traditional cameras can only record two-dimensional images.

[0058] Differential Privacy: A privacy protection technique that aims to ensure that an individual's private information is not leaked or abused when processing sensitive data. It provides a mathematical framework that can quantify the impact of individual data on the output results and protect privacy by adding controlled noise or perturbations.

[0059] According to an embodiment of the present invention, an embodiment of a method for processing a light field image dataset based on differential privacy is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0060] In this embodiment, a light field image data set processing method based on differential privacy is provided, which can be used in the above-mentioned computer. Figure 1 is a flowchart of a method for processing a light field image dataset based on differential privacy according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0061] Step S101 , grouping light field image data sets according to the shooting time sequence and the lens array of the light field camera to which they belong, to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group includes a plurality of sub-images shot by the light field camera at the same time.

[0062] In an embodiment of the present invention, the light field image data set is grouped according to the time sequence and the lens array. The light field camera uses an N×N lens array (for example, 3×3, 5×5, etc.), and each lens simultaneously captures a sub-image. At the same time t, a light field camera will generate N×N sub-images, and the sub-images will be grouped into the same group. For example, for a 3×3 lens array, each group contains 9 sub-images.

[0063] Step S102: number all sub-images in each light field image data group to obtain image numbers.

[0064] Specifically, the sub-images in each group are numbered starting from the upper left corner and in row priority order. For example, in a 3×3 lens array, the sub-images in the i-th group Gi are numbered D1 to D9 from the upper left to the lower right, so as to maintain the spatial position relationship of the images in subsequent processing.

[0065] Step S103: extract pixels from all sub-images in each light field image data group to obtain a first pixel matrix.

[0066] It should be noted that pixel extraction is performed on each sub-image to obtain a pixel matrix. The two formats of RGB and YUV can be used. In the RGB format, each pixel contains the values ​​of three channels: red (R), green (G), and blue (B); the YUV format contains brightness (Y) and chrominance (UV) information. In this embodiment, only the Y channel can be processed to reduce the amount of calculation.

[0067] In a specific embodiment, if the image encoding method is YUV format, refer to Figure 2 , step S103, comprising:

[0068] a1, encode each sub-image according to the YUV format to obtain the brightness component Y and the chrominance component UV;

[0069] a2, extract pixels of the brightness component Y to obtain a first pixel matrix.

[0070] The embodiment of the present invention encodes the sub-image in YUV format, processes the brightness component Y and the chrominance component UV separately, and stores the UV component separately, so that the differential privacy processing can be concentrated on the brightness component, avoiding redundant processing of the chrominance component, reducing the computational overhead, and ensuring that the chrominance information is also complete, thereby improving the processing efficiency while ensuring the visual quality of the image.

[0071] Step S104: Encode the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store the data.

[0072] In a specific embodiment, the method includes associating an image number, a first pixel matrix, and a group number according to a lens array, obtaining and storing sub-image metadata of each associated sub-image; wherein the sub-image metadata is in the form of a key-value pair, the key is path information consisting of at least an image number and a group number, and the value is the first pixel matrix.

[0073] Exemplarily, the image number, pixel matrix and group number are encoded to obtain one-dimensional encoded data, and the encoding format is "storage path\t\tGi\tDj" as the key, and the one-dimensional data of the pixel matrix is ​​used as the value, separated by \t for storage. Among them, Gi group number and Dj image number are two indispensable information in the path information of the embodiment of the present invention, and the storage path represents the prefix storage of uploading the one-dimensional encoded data to the hdfs distributed storage, which can be flexibly adjusted according to the actual situation.

[0074] In a specific embodiment, the key in the sub-image metadata is path information consisting of at least an image number, an image encoding method, and a group number, such as "storage path\tYUV encoding method\tGi\tDj". In addition, while extracting pixels of the brightness component Y to obtain the first pixel matrix in step a2, the chrominance component UV is stored in a preset path according to the path information. It can be understood that the brightness and chrominance information are stored separately, and the YUV format can effectively reduce the resolution of the color information, thereby reducing the amount of data, and significantly improving the computational complexity and network throughput of differential privacy processing.

[0075] The embodiment of the present invention associates the image number, pixel matrix and group number according to the lens array, and stores the sub-image metadata in the form of a key-value pair, where the key contains the path information of the image and the value is the pixel matrix, so that the position information and pixel data of each sub-image can correspond one to one. The key-value pair storage method avoids the confusion of position information during data storage, ensures that sub-images in different groups and positions can be accurately identified, and provides a data organization basis for subsequent differential privacy processing at the data storage level, which helps to achieve efficient processing of light field image data sets.

[0076] Step S105 , processing the one-dimensional coded data based on the differential privacy processing algorithm to obtain one-dimensional coded data of each sub-image after processing.

[0077] In a specific embodiment, reference may be made to Figure 3 , step S105, comprising:

[0078] b1, aggregate the metadata of sub-images in the same group based on the group number;

[0079] Specifically, in the actual application process, the metadata file stored in HDFS is read according to the key value, the key-value pair information key-vblue in the file is parsed, and the group number information Gi is extracted; the vblue data with the same group number are clustered together according to the spatial relationship of the lens array, and a data structure is constructed in the memory of HDFS based on the clustered data to store the aggregation result.

[0080] b2, reading the first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information;

[0081] Specifically, in the actual application process, based on the key value read from the Pixel Di To Pixel Dn*n The first pixel matrix of each sub-image is reconstructed into a two-dimensional matrix, where Pixel Di Represents the sub-image at the upper left position in the current group, Pixel Dn*nRepresents the sub-image at the lower right position within the current group. For RGB format, reconstructed as a H×W×3 matrix, or, for YUV format, reconstructed as a H×W brightness matrix.

[0082] b3, performing wavelet transform on the first pixel matrix of all sub-images in the same group to obtain the frequency domain matrix corresponding to each sub-image;

[0083] It should be noted that for all images in a group, the wavelet transform can be performed in parallel, and the wavelet feature vector of each image after wavelet transform is recorded as Pi, Pi n Represents the nth wavelet coefficient in each eigenvector. Since the image array is taken of the same object at the same time, the image features, that is, the wavelet coefficients, are similar.

[0084] Exemplarily, a group includes 9 sub-images, the pixel matrix size of the sub-image is 256×256, and each sub-image inputs its brightness component (Y) with a size of 256×256; the Dbubechies-4 wavelet basis function is selected, and a first-level wavelet decomposition is performed, and the low-frequency component LL with a size of 128×128 and the high-frequency components LH, HL, and HH with sizes of 128×128 respectively are classified; all coefficients of the low-frequency components and high-frequency components are output, wherein the decomposition process of the 9 sub-images is completed in parallel, and all output results correspond to 9 groups of wavelet eigenvectors (frequency domain matrices).

[0085] b4, select the sub-image at the center of each group as the reference image, and use the frequency domain matrix of the reference image as the feature matrix;

[0086] Specifically, for an N×N array, select the image in the (N+1) / 2th row and (N+1) / 2th column, that is, No. The frequency domain matrix i of the sub-image, for example, D5 is selected as the reference in the 3×3 array, and the frequency domain matrix of the reference image is used as the feature matrix.

[0087] b5, generating a noise matrix based on the feature matrix of the reference image, and adding the noise matrix to the frequency domain matrices of other sub-images of the non-reference image in the same group;

[0088] Furthermore, the step b5 of generating a noise matrix based on the feature matrix of the reference image includes: weighting the feature matrix of the reference image based on a preset differential privacy protection strength parameter; and generating a Laplace distributed noise matrix based on the weighted feature matrix.

[0089] It should be noted that based on the characteristics of the image array, the same budget subprocessing is used for the wavelet coefficient vector, that is, the same privacy budget parameter is used to weight the feature vector. The privacy budget ε represents the protection strength of differential privacy. The smaller ε is, the higher the protection coefficient is, the higher the image distortion rate is, that is, the lower the data availability and image quality is.

[0090] Exemplarily, a Laplace distributed noise matrix is ​​generated according to the weighted feature matrix, which can be expressed by the following formula:

[0091]

[0092] Where F represents the noise matrix, ε, Pi n All of these have been explained in the above steps and will not be repeated here.

[0093] The embodiment of the present invention performs weighted processing on the feature matrix of the reference image and generates a Laplace distributed noise matrix, so that the added noise intensity can be freely adjusted according to the differential privacy protection requirements. The weighted noise generation method based on the feature matrix avoids the redundant disturbance that may be caused by random noise. At the same time, the Laplace distribution is used to ensure that the statistical characteristics of the noise meet the differential privacy requirements. The combination of the two achieves a balance between protecting privacy and maintaining data availability, so that the restored light field dataset can both protect privacy and improve data availability.

[0094] Exemplarily, the noise matrix F is added to the frequency domain matrices of other sub-images of the non-reference image in the same group.

[0095] b6, performing inverse wavelet transform on the frequency domain matrices of all sub-images after adding noise to obtain a processed second pixel matrix;

[0096] b7, encode the image number, the second pixel matrix and the group number to obtain one-dimensional encoded data after each sub-image is processed.

[0097] Specifically, the inverse wavelet transform is used to restore the frequency domain matrix after adding noise to a pixel matrix, and the pixel matrix is ​​reduced to one dimension. The same encoding method as in steps S102 to S104 is used to obtain the processed key-value pair key-vblue as a one-dimensional array set after differential privacy processing.

[0098] The embodiment of the present invention performs wavelet transform on sub-images in the same group to obtain a frequency domain matrix, selects the sub-image at the center position as a reference to generate a noise matrix, then adds noise to other sub-images, and finally performs an inverse transform, so that the sub-images in the same group can maintain relatively consistent noise characteristics, avoiding the destruction of spatial correlation caused by inconsistent noise of each sub-image. At the same time, noise addition in the frequency domain is realized through wavelet transform, the visual quality of the image is guaranteed, the spatial correlation of the light field image is maintained while protecting privacy, and the usability of the processed light field image data set in practical applications is improved.

[0099] Step S106 , based on the processed one-dimensional coded data of each sub-image, each sub-image is rearranged according to the lens array of the light field camera to obtain a light field image data set after differential privacy protection.

[0100] In a specific embodiment, reference may be made to Figure 4 , step S106, comprising:

[0101] c1, read the chroma component UV corresponding to each sub-image from the preset path;

[0102] c2, restore the processed one-dimensional coded data to the brightness component Y;

[0103] c3, merge the restored brightness component Y with the corresponding chrominance component UV to obtain a sub-image in YUV format;

[0104] c4, arrange all sub-images in YUV format according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection.

[0105] Exemplarily, the chromaticity components UV corresponding to the group number Gi and the sub-image number Dj are read in batches from a preset storage path, and the processed brightness components (second pixel matrix) are read from the one-dimensional encoded data generated in step S105; the brightness components Y and the chromaticity components UV of the same group number Gi and number Dj are matched; according to the YUV format, the brightness matrix Y and the chromaticity matrix UV are merged pixel by pixel to generate a complete YUV format sub-image; according to the group number Gi and number Dj, the position of each sub-image in the two-dimensional array is determined based on the arrangement rule of the lens array, and the sub-images of the same group are arranged in sequence according to the position, and spliced ​​into a complete light field image.

[0106] The embodiment of the present invention re-combines the processed brightness component Y with the originally stored chrominance component UV and arranges them according to the lens array, so that the light field image after differential privacy protection can be completely restored. The method of component separation and re-combination is adopted to avoid image distortion caused by unnecessary processing of chrominance information. At the same time, the spatial structure of the light field image is further guaranteed to be intact by arranging according to the lens array. Thus, the visual quality and spatial information of the image are maintained to the greatest extent while protecting privacy, thereby improving the availability of the processed light field image dataset.

[0107] In actual application, the processed one-dimensional encoded data generated in step S105 can be read, the group number and number of the sub-image can be extracted from the key key, and the processed second pixel matrix can be obtained from the value value; the sub-image is classified into the corresponding light field image data group based on the group number Gi, and the sub-image is arranged into a two-dimensional array according to the number Dj and the arrangement rule of the lens array (row priority or column priority). For a 3×3 array, D1 corresponds to the upper left corner, and D9 corresponds to the lower right corner. The one-dimensional pixel data of each sub-image is restored to two-dimensional image data, and the restored two-dimensional image data is combined into a multi-view data set of the light field image according to the layout of the lens array, and the rearranged light field image data set is stored in a preset format and re-uploaded to hdfs or local storage for subsequent use. If subsequent operations such as refocusing or three-dimensional modeling are required, the light field image data set after differential privacy protection is obtained from hdfs or local storage based on a preset interface.

[0108] The embodiment of the present invention groups light field images according to shooting time and lens array, and numbers and extracts pixels of sub-images in each group, and then performs one-dimensional encoding and differential privacy processing, and finally rearranges them according to the lens array, so as to protect the privacy of light field images while maintaining the spatial correlation between sub-images. Because the group processing method is adopted, the problem of position information loss caused by differential privacy processing of independent images is avoided. At the same time, the integrity of the light field image array information is ensured through one-dimensional encoding and rearrangement, thereby solving the problem that the existing technology cannot effectively protect image data with spatial correlation, and achieving the availability of light field image data sets while protecting privacy, so that subsequent applications such as refocusing and three-dimensional imaging can correctly restore the light field.

[0109] In this embodiment, a light field image data set processing system based on differential privacy is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0110] This embodiment provides a light field image data set processing system based on differential privacy. Figure 5 As shown, it includes a data upload module and a MapReduce module.

[0111] The data uploading module 501 is used to group the light field image data set according to the shooting time sequence and the lens array of the light field camera to which it belongs, to obtain multiple light field image data groups and group numbers, wherein each light field image data group contains multiple sub-images shot by the light field camera at the same time; number all the sub-images in each light field image data group to obtain an image number; perform pixel extraction on all the sub-images in each light field image data group to obtain a first pixel matrix; encode the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store it;

[0112] The MapReduce module 502 includes a Map unit 5021 and a Reduce unit 5022;

[0113] A Map unit 5021 is used to process the one-dimensional coded data based on a differential privacy processing algorithm to obtain one-dimensional coded data of each sub-image after processing;

[0114] The Reduce unit 5022 is used to rearrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each sub-image after processing, so as to obtain a light field image data set after differential privacy protection.

[0115] It should be noted that in actual applications, MapReduce is a distributed computing framework for processing large-scale data sets, suitable for scenarios that require large-scale parallel computing. Its core consists of two stages: Map stage and Reduce stage. HDFS (Hadoop Distributed File System) is the basic storage system in the MapReduce framework.

[0116] The Map stage is responsible for decomposing the input data into multiple small tasks (i.e., processing the one-dimensional encoded data), which are executed in parallel by different nodes in the distributed system. The Reduce stage processes the grouped intermediate results, summarizes, merges or further calculates them, and finally outputs the results (i.e., processing the one-dimensional encoded data to obtain the one-dimensional encoded data of each sub-image after processing).

[0117] In the application process of the present invention, reference may be made to Figure 6Light field image array->grouping (obtaining group1 to groupN)->segmentation (obtaining multiple sub-images)->(one-dimensional processing, expressed in the form of key-value)->input into MapReduce module. The Map unit calculates features based on the key-value respectively, obtains the processed key-value after inverse transformation and outputs it->Reduce unit re-encodes it into an image matrix and restores the light field image array.

[0118] In some optional embodiments, the data upload module 501 specifically includes: an image association unit, which is used to associate the image number, the first pixel matrix and the group number according to the lens array, obtain and store the sub-image metadata of each associated sub-image; wherein the sub-image metadata is in the form of a key-value pair, the key is path information consisting of at least the image number and the group number, and the value is the first pixel matrix.

[0119] In some optional implementations, the Map unit 5021 specifically includes:

[0120] A metadata aggregation subunit, used for aggregating metadata of sub-images in the same group based on the group number;

[0121] A pixel matrix reading subunit, used for reading a first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information;

[0122] A wavelet transform subunit is used to perform wavelet transform on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image;

[0123] A feature matrix extraction subunit is used to select the sub-image at the center position of each group as a reference image, and use the frequency domain matrix of the reference image as a feature matrix;

[0124] A noise adding subunit, used for generating a noise matrix based on a feature matrix of a reference image, and adding the noise matrix to frequency domain matrices of other sub-images of the non-reference image in the same group;

[0125] A wavelet inverse transformation subunit, used for performing a wavelet inverse transformation on the frequency domain matrices of all sub-images after adding noise, to obtain a processed second pixel matrix;

[0126] The coding generation subunit is used to encode the image number, the second pixel matrix and the group number to obtain one-dimensional coded data after each sub-image is processed.

[0127] In some optional implementations, the key in the sub-image metadata is path information consisting of at least an image number, an image encoding method, and a group number. If the image encoding method is in YUV format, the data uploading module 501 specifically includes:

[0128] The brightness encoding unit is used to encode each sub-image in YUV format to obtain the brightness component Y and the chrominance component UV;

[0129] The matrix generation unit is used to extract pixels of the brightness component Y to obtain a first pixel matrix, and store the chrominance component UV to a preset path according to the path information.

[0130] In some optional implementations, the Reduce unit 5022 includes:

[0131] The chroma component reading subunit is used to read the chroma component UV corresponding to each sub-image from a preset path;

[0132] A luminance component restoration subunit, used to restore the processed one-dimensional coded data to a luminance component Y;

[0133] The YUV merging subunit is used to merge the restored brightness component Y with the corresponding chrominance component UV to obtain a sub-image in YUV format;

[0134] The image arrangement subunit is used to arrange all sub-images in YUV format according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection.

[0135] In some optional implementations, the data upload module 501 specifically includes:

[0136] A weighted processing unit, used for performing weighted processing on the feature matrix of the reference image based on a preset differential privacy protection strength parameter;

[0137] The noise matrix generating unit is used to generate a Laplace distributed noise matrix according to the weighted characteristic matrix.

[0138] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0139] The light field image dataset processing system based on differential privacy in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0140] The embodiment of the present invention also provides a computer device having the above Figure 5 The light field image dataset processing system based on differential privacy is shown.

[0141] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0142] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0143] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0144] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0146] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0147] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0148] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0149] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A light field image dataset processing method based on differential privacy, characterized in that: The method comprises: Grouping the light field image data sets according to the shooting time sequence and the lens array of the light field camera to which they belong, to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group includes a plurality of sub-images shot by the light field camera at the same time; Numbering all sub-images in each light field image data group to obtain image numbers; Performing pixel extraction on all sub-images in each light field image data group to obtain a first pixel matrix; Encoding the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store the data; The one-dimensional coded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing; Based on the one-dimensional encoded data of each sub-image after processing, each sub-image is rearranged according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection.

2. The method according to claim 1, characterized in that The encoding of the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and storing the data specifically includes: Associating the image number, the first pixel matrix and the group number according to the lens array, obtaining and storing sub-image metadata of each associated sub-image; The sub-image metadata is in the form of a key-value pair, the key is path information consisting of at least an image number and a group number, and the value is the first pixel matrix.

3. The method according to claim 2, characterized in that The one-dimensional coded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing, specifically including: Aggregate metadata of sub-images in the same group based on the group number; Reading a first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information; Performing wavelet transform on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image; Selecting the sub-image at the center position of each group as a reference image, and using the frequency domain matrix of the reference image as a feature matrix; Generate a noise matrix based on the feature matrix of the reference image, and add the noise matrix to the frequency domain matrices of other sub-images of the non-reference image in the same group; Performing inverse wavelet transform on the frequency domain matrices of all sub-images after adding noise to obtain a processed second pixel matrix; The image number, the second pixel matrix and the group number are encoded to obtain one-dimensional encoded data after each sub-image is processed.

4. The method according to claim 2, characterized in that: The key in the sub-image metadata is path information consisting of at least an image number, an image encoding method, and a group number; if the image encoding method is in YUV format, then extracting pixels from all sub-images in each light field image data group to obtain a first pixel matrix specifically includes: Encode each sub-image in YUV format to obtain the brightness component Y and the chrominance component UV; Pixels of the brightness component Y are extracted to obtain a first pixel matrix, and the chrominance component UV is stored in a preset path according to the path information.

5. The method according to claim 4, characterized in that The one-dimensional coded data of each sub-image after processing is rearranged according to the lens array of the light field camera to obtain a light field image dataset after differential privacy protection, specifically including: Read the chroma component UV corresponding to each sub-image from the preset path; Restore the processed one-dimensional encoded data to the brightness component Y; The restored brightness component Y is combined with the corresponding chrominance component UV to obtain a sub-image in YUV format; All sub-images in YUV format are arranged according to the lens array of the light field camera to obtain a light field image dataset with differential privacy protection.

6. The method according to claim 3, characterized in that The generating of the noise matrix based on the feature matrix of the reference image specifically includes: The feature matrix of the reference image is weighted based on a preset differential privacy protection strength parameter; Generate a Laplace distributed noise matrix based on the weighted feature matrix.

7. A light field image dataset processing system based on differential privacy, characterized in that: The system includes a data upload module and a MapReduce module; The data uploading module is used to group the light field image data set according to the shooting time sequence and the lens array of the light field camera to which it belongs, to obtain multiple light field image data groups and group numbers, wherein each light field image data group contains multiple sub-images shot by the light field camera at the same time; number all the sub-images in each light field image data group to obtain an image number; perform pixel extraction on all the sub-images in each light field image data group to obtain a first pixel matrix; encode the image number, the first pixel matrix and the group number to obtain one-dimensional encoded data of each sub-image and store it; The MapReduce module includes a Map unit and a Reduce unit; The Map unit is used to process the one-dimensional coded data based on the differential privacy processing algorithm to obtain the one-dimensional coded data of each sub-image after processing; The Reduce unit is used to rearrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each sub-image after processing, so as to obtain a light field image data set after differential privacy protection.

8. A computer device, characterized in that: The computer device includes a memory and a processor; A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the light field image data set processing method based on differential privacy according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the light field image dataset processing method based on differential privacy according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the light field image dataset processing method based on differential privacy according to any one of claims 1 to 6.

Images