A method and system for processing light field image dataset based on differential privacy

Abstract

The application relates to the technical field of image processing, and discloses a light field image dataset processing method and system based on differential privacy. The application groups light field image datasets to obtain a light field image data group and a group number, numbers sub-images to obtain image numbers and first pixel matrices through pixel extraction; encodes the image numbers, the first pixel matrices and the group number to obtain one-dimensional encoding data of the sub-images and stores the one-dimensional encoding data; processes the one-dimensional encoding data based on a differential privacy processing algorithm to obtain one-dimensional encoding data of the sub-images; and rearranges the sub-images based on the processed one-dimensional encoding data of the sub-images to obtain a light field image dataset protected by differential privacy. The one-dimensional encoding and rearrangement ensure the integrity of light field image array information, avoid the loss of position information caused by separately processing independent images through differential privacy, and ensure the availability of the light field image dataset while protecting privacy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a light field image dataset processing method and system based on differential privacy. BACKGROUND

[0002] The image protection method such as data encryption in the related art has problems such as large calculation overhead and low data utility, and the method based on image disturbance such as noise, blur and pixelization is easy to be restored by a deep learning model.

[0003] At present, the differential privacy (DP) technology can solve the problems mentioned in the related art by adding controllable noise to protect privacy. The image differential privacy protection method mainly includes two types of spatial domain processing and frequency domain processing. Among them, the processing method based on frequency domain decomposes the image pixel information through Fourier transform, wavelet transform and other means, and differentiates the privacy protection of different frequency bands. However, the existing image differential privacy protection technology mainly processes independent images, cannot effectively protect image data with spatial correlation, and will cause the position information of the light field image to be lost or confused, so that the subsequent refocusing, three-dimensional imaging and other applications cannot correctly restore the light field of a lens group, affecting the usability of the light field dataset. SUMMARY

[0004] Therefore, the present application provides a light field image dataset processing method and system based on differential privacy to solve the problem of how to effectively improve the usability of the light field image dataset in the light field image protection process.

[0005] In a first aspect, the present application provides a light field image dataset processing method based on differential privacy, which comprises:

[0006] grouping the light field image dataset according to the shooting time sequence and the lens array of the light field camera to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group contains a plurality of sub-images shot by the light field camera at the same time;

[0007] numbering all the sub-images in each light field image data group to obtain image numbers;

[0008] pixel extraction is performed on all the sub-images in each light field image data group to obtain a first pixel matrix;

[0009] encoding the image numbers, the first pixel matrix and the group numbers to obtain one-dimensional encoding data of each sub-image and storing the one-dimensional encoding data;

[0010] processing the one-dimensional encoding data based on a differential privacy processing algorithm to obtain one-dimensional encoding data of each sub-image after processing.

[0011] Re-arrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each processed sub-image, to obtain the light field image data set after differential privacy protection.

[0012] The embodiment of the present application groups the light field image according to the shooting time and the lens array, numbers and extracts the pixels of the sub-images in each group, then performs one-dimensional encoding and differential privacy processing, and finally re-arranges according to the lens array, so that the spatial correlation between the sub-images is maintained while the privacy of the light field image is protected. Because of the grouping processing method, the problem of loss of position information caused by separate differential privacy processing of independent images is avoided, and the integrity of the light field image array information is ensured through one-dimensional encoding and re-arrangement, thereby solving the problem that the image data with spatial correlation cannot be effectively protected in the prior art, and realizing the availability of the light field image data set while protecting the privacy, so that subsequent refocusing, three-dimensional imaging and other applications can correctly restore the light field.

[0013] In an optional implementation, the one-dimensional encoded data of each sub-image is obtained by encoding the image number, the first pixel matrix, and the group number and storing, specifically including:

[0014] The image number, the first pixel matrix, and the group number are associated according to the lens array to obtain the sub-image metadata of each sub-image after association and store;

[0015] The sub-image metadata is in the form of key-value pair, the key is path information composed of at least the image number and the group number, and the value is the first pixel matrix.

[0016] The embodiment of the present application associates the image number, the pixel matrix, and the group number according to the lens array, and stores the sub-image metadata in the form of key-value pair, wherein the key contains the path information of the image, and the value is the pixel matrix, so that the position information and the pixel data of each sub-image can be one-to-one corresponding. The storage method of key-value pair avoids the confusion of position information in the data storage process, ensures that the sub-images of different groups and different positions can be accurately identified, provides a data organization basis for subsequent differential privacy processing at the data storage level, and is helpful to realize efficient processing of the light field image data set.

[0017] In an optional implementation, the one-dimensional encoded data of each sub-image is obtained by processing the one-dimensional encoded data based on the differential privacy processing algorithm, specifically including:

[0018] The sub-image metadata of the same group is aggregated based on the group number;

[0019] reading a first pixel matrix of each sub-image in the sub-image metadata of the same group based on path information;

[0020] performing wavelet transform on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image;

[0021] selecting a sub-image at a center position of each group as a reference image, and taking the frequency domain matrix of the reference image as a feature matrix;

[0022] generating a noise matrix based on the feature matrix of the reference image, and adding the noise matrix to the frequency domain matrix of other sub-images in the same group which are not the reference image;

[0023] performing inverse wavelet transform on the frequency domain matrix of all sub-images after adding the noise to obtain a processed second pixel matrix;

[0024] encoding the image number, the second pixel matrix and the group number to obtain one-dimensional encoding data of each sub-image after processing.

[0025] The embodiment of the present application obtains the frequency domain matrix by performing wavelet transform on the sub-images in the same group, selects the sub-image at the center position as the reference to generate the noise matrix, adds the noise to other sub-images, and finally performs inverse transform, so that the sub-images in the same group can maintain relatively consistent noise characteristics, avoiding the destruction of spatial correlation caused by inconsistent noise of each sub-image, and realizing noise addition in the frequency domain through wavelet transform, which guarantees the visual quality of the image, maintains the spatial correlation of the light field image while protecting privacy, and further improves the usability of the processed light field image data set in practical application.

[0026] In an optional embodiment, the key in the sub-image metadata is path information composed of at least an image number, an image encoding mode and a group number; if the image encoding mode is YUV format, the first pixel matrix is obtained by performing pixel extraction on all sub-images in each light field image data group, specifically including:

[0027] encoding each sub-image in YUV format to obtain a luminance component Y and a chrominance component UV;

[0028] performing pixel extraction on the luminance component Y to obtain the first pixel matrix, and storing the chrominance component UV to a preset path according to the path information.

[0029] The embodiment of the present application encodes the sub-image in YUV format, respectively processes the luminance component Y and the chrominance component UV, and separately stores the UV component, so that the differential privacy processing can be concentrated on the luminance component, the redundant processing on the chrominance component is avoided, the calculation overhead is reduced, and meanwhile, it is ensured that the chrominance information is also complete, thereby improving the processing efficiency while ensuring the visual quality of the image.

[0030] In an optional embodiment, the one-dimensional encoded data of each processed sub-image is rearranged according to the lens array of the light field camera to obtain a differential privacy protected light field image data set, specifically including:

[0031] reading the chrominance component UV corresponding to each sub-image from the preset path;

[0032] restoring the one-dimensional encoded data after processing to the luminance component Y;

[0033] merging the restored luminance component Y and the corresponding chrominance component UV to obtain a sub-image in YUV format;

[0034] arranging all the sub-images in YUV format according to the lens array of the light field camera to obtain a differential privacy protected light field image data set.

[0035] The embodiment of the present application restores the light field image protected by differential privacy by re-merging the processed luminance component Y and the originally stored chrominance component UV and arranging according to the lens array, adopts the component separation processing and re-merging manner, avoids the image distortion caused by unnecessary processing on the chrominance information, and further ensures the spatial structure completeness of the light field image by arranging according to the lens array. Thus, the visual quality and spatial information of the image are maintained to the greatest extent while the privacy is protected, and the usability of the processed light field image data set is improved.

[0036] In an optional embodiment, the noise matrix is generated based on the feature matrix of the reference image, specifically including:

[0037] performing weighted processing on the feature matrix of the reference image based on a preset differential privacy protection strength parameter;

[0038] generating a noise matrix with Laplace distribution according to the weighted feature matrix.

[0039] The embodiment of the present application adds noise to the feature matrix of the reference image by weighting processing and generates a noise matrix of Laplace distribution, so that the added noise intensity can be freely adjusted according to the differential privacy protection requirement, and the weighted noise generation mode based on the feature matrix avoids the redundant disturbance that the random noise may bring, and at the same time, the statistical characteristics of the noise are ensured to meet the differential privacy requirement through the Laplace distribution, and the combination of the two balances the privacy protection and data usability, so that the restored light field data set can protect privacy and improve data usability.

[0040] In a second aspect, the present application provides a light field image data set processing system based on differential privacy, the system comprising a data uploading module and a MapReduce module.

[0041] The data uploading module is configured to group the light field image data set according to the shooting time sequence and the lens array of the light field camera to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group contains a plurality of sub-images shot by the light field camera at the same time; number all the sub-images in each light field image data group to obtain image numbers; extract pixels from all the sub-images in each light field image data group to obtain a first pixel matrix; and encode the image numbers, the first pixel matrix and the group numbers to obtain one-dimensional encoded data of each sub-image and store the one-dimensional encoded data.

[0042] The MapReduce module comprises a Map unit and a Reduce unit.

[0043] The Map unit is configured to process the one-dimensional encoded data based on a differential privacy processing algorithm to obtain one-dimensional encoded data of each sub-image after processing.

[0044] The Reduce unit is configured to rearrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each sub-image after processing to obtain a light field image data set protected by differential privacy.

[0045] In a third aspect, the present application provides a computer device, comprising a memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the light field image data set processing method based on differential privacy of the first aspect or any of the corresponding embodiments thereof.

[0046] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for causing a computer to execute the light field image data set processing method based on differential privacy of the first aspect or any of the corresponding embodiments thereof.

[0047] In a fifth aspect, the present application provides a computer program product comprising computer instructions for causing a computer to perform the method for differentially private light field image dataset processing according to the first aspect or any of its possible implementation forms. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0049] Figure 1 is a first flowchart of the method for differentially private light field image dataset processing according to an embodiment of the present application;

[0050] Figure 2 is a second flowchart of the method for differentially private light field image dataset processing according to an embodiment of the present application;

[0051] Figure 3 is a third flowchart of the method for differentially private light field image dataset processing according to an embodiment of the present application;

[0052] Figure 4 is a fourth flowchart of the method for differentially private light field image dataset processing according to an embodiment of the present application;

[0053] Figure 5 is a structural block diagram of the system for differentially private light field image dataset processing according to an embodiment of the present application;

[0054] Figure 6 is a fifth flowchart of the system for differentially private light field image dataset processing according to an embodiment of the present application;

[0055] Figure 7 is a hardware structure diagram of the computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0057] Light-filed image: Captured by a light-field camera array, which records the position and direction information of light in space through a lens array, calculates the three-dimensional spatial coordinates through light-field imaging technology, and makes up for the shortcomings of traditional cameras that can only record two-dimensional images.

[0058] Differential privacy: A privacy protection technology that aims to ensure that individual privacy information is not leaked or misused when processing sensitive data. It provides a mathematical framework that can quantify the impact of individual data on the output result and protect privacy by adding controlled noise or disturbance.

[0059] According to the embodiment of the present application, a differential privacy-based light-field image dataset processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0060] A differential privacy-based light-field image dataset processing method is provided in this embodiment, which can be used in the above-mentioned computer, Figure 1 The flowchart of the differential privacy-based light-field image dataset processing method according to the embodiment of the present application is shown in Figure 1 The flowchart includes the following steps:

[0061] Step S101: Group the light-field image dataset according to the shooting time sequence and the lens array of the light-field camera to obtain a plurality of light-field image data groups and group numbers, wherein each light-field image data group contains a plurality of sub-images shot by the light-field camera at the same time.

[0062] In the embodiment of the present application, the light-field image dataset is grouped according to the time sequence and the lens array. The light-field camera uses an N×N lens array (such as 3×3, 5×5, etc.), and each lens simultaneously shoots a sub-image. At the same time t, an N×N sub-image is generated by a light-field camera, and the sub-image is divided into the same group. For example, for a 3×3 lens array, each group contains 9 sub-images.

[0063] Step S102: Number all the sub-images in each light-field image data group to obtain image numbers.

[0064] Specifically, the sub-images in each group are numbered, and the numbering method starts from the top left corner and follows the row priority order. For example, in a 3×3 lens array, the sub-images in the i-th group Gi are numbered from the top left to the bottom right as D1 to D9, so as to maintain the spatial position relationship of the images during subsequent processing.

[0065] Step S103, pixel extraction is performed on all sub-images in each light field image data set to obtain a first pixel matrix.

[0066] It should be noted that pixel extraction is performed on each sub-image to obtain a pixel matrix. RGB or YUV formats can be used, where in the RGB format, each pixel point contains values of three channels of red (R), green (G), and blue (B); and in the YUV format, luminance (Y) and chrominance (UV) information is contained, and in this embodiment, only the Y channel can be processed to reduce the amount of calculation.

[0067] In a specific embodiment, if the image encoding mode is YUV format, the following can be referred to: Figure 2 Step S103 includes:

[0068] a1, encoding each sub-image in YUV format to obtain luminance component Y and chrominance component UV;

[0069] a2, performing pixel extraction on the luminance component Y to obtain a first pixel matrix.

[0070] In this embodiment, the sub-image is encoded in YUV format, the luminance component Y and the chrominance component UV are processed respectively, and the UV component is stored separately, so that the differential privacy processing can be concentrated on the luminance component, avoiding redundant processing of the chrominance component, reducing the calculation overhead, while ensuring that the chrominance information is complete, thereby improving the processing efficiency while ensuring the visual quality of the image.

[0071] Step S104, encoding the image number, the first pixel matrix, and the group number to obtain one-dimensional encoding data of each sub-image and storing.

[0072] In a specific embodiment, the image number, the first pixel matrix, and the group number are associated according to the lens array to obtain sub-image metadata of each sub-image after association and storage; wherein the sub-image metadata is in the form of key-value pair, the key is path information composed of at least the image number and the group number, and the value is the first pixel matrix.

[0073] For example, the image number, the pixel matrix, and the group number are encoded to obtain one-dimensional encoding data, the encoding format is "storage path\tGi\tDj" as the key, and the one-dimensional data of the pixel matrix is stored as the value, separated by \. Among them, Gi and Dj are two indispensable information in the path information of this embodiment, and the storage path represents the prefix storage of the one-dimensional encoding data uploaded to the hdfs distributed storage, which can be flexibly adjusted according to the actual situation.

[0074] In a specific embodiment, the key in the sub-image metadata is path information composed of at least image number, image encoding mode, and group number, for example, "storage path\YUV encoding mode\Gi\Dj". In step a2, while the pixel extraction of the luminance component Y obtains the first pixel matrix, the chrominance components UV are stored in the preset path according to the path information. It can be understood that the luminance and chrominance information are stored separately, and the YUV format can effectively reduce the resolution of color information, thereby reducing the data amount and significantly improving the calculation amount and network throughput of differential privacy processing.

[0075] In the embodiments of the present application, the image number, pixel matrix, and group number are associated according to the lens array, and the sub-image metadata is stored in the form of key-value pairs, wherein the key contains the path information of the image, and the value is the pixel matrix, so that the position information and pixel data of each sub-image can be one-to-one corresponding. The storage mode of key-value pairs avoids the confusion of position information in the data storage process, ensures that the sub-images of different groups and different positions can be accurately identified, provides a data organization basis for subsequent differential privacy processing at the data storage level, and is helpful to realize efficient processing of the light field image data set.

[0076] In step S105, the one-dimensional encoded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional encoded data of each sub-image after processing.

[0077] In a specific embodiment, the one-dimensional encoded data can be processed by referring to Figure 3 , and step S105 comprises:

[0078] b1, aggregating the sub-image metadata of the same group based on the group number;

[0079] Specifically, in actual application, the stored metadata file is read from the hdfs according to the key value, the key-value pair information key-vblue in the file is parsed, and the group number information Gi is extracted; the vblue data of the same group number is aggregated together according to the spatial relationship of the lens array, and the aggregated data is used to construct a data structure in the memory of the hdfs to store the aggregation result.

[0080] b2, reading the first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information;

[0081] Specifically, in actual application, the first pixel matrix of each sub-image from Pixel Di to Pixel Dn*n is read based on the key value, and the one-dimensional pixel sequence is reconstructed into a two-dimensional matrix, wherein Pixel Di represents the sub-image at the top left position in the current group, and Pixel Dn*nrepresents the sub-image at the lower right position in the current group. For the RGB format, it is reconstructed into a HxWx3 matrix, or, for the YUV format, it is reconstructed into a HxW luminance matrix.

[0082] b3, wavelet transform is performed on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image;

[0083] It should be noted that the wavelet transform of all images in a group can be performed in parallel, and the wavelet feature vector of each image after wavelet transform is denoted as Pi, Pi n represents the nth wavelet coefficient in each feature vector, and since the image array is taken at the same time for the same object, the features of the image, i.e. the wavelet coefficients, have similarity.

[0084] For example, a group contains 9 sub-images, and the pixel matrix size of the sub-image is 256x256. The luminance component (Y) of each sub-image is input, and the size is 256x256. The Dbubechies-4 wavelet basis function is selected, and 1-level wavelet decomposition is performed to obtain low-frequency components LL with a size of 128x128 and high-frequency components LH, HL and HH with sizes of 128x128, respectively. All coefficients of the low-frequency components and the high-frequency components are output, wherein the decomposition process of the 9 sub-images is completed in parallel, and the total output result corresponds to 9 groups of wavelet feature vectors (frequency domain matrices).

[0085] b4, selecting a sub-image at the center position of each group as a reference image, and taking the frequency domain matrix of the reference image as a feature matrix;

[0086] Specifically, for an N*N array, the image at the (N+1) / 2 row and (N+1) / 2 column is selected, i.e. the frequency domain matrix i of the D5 sub-image, for example, the D5 is selected as the reference in a 3x3 array, and the frequency domain matrix of the reference image is taken as the feature matrix.

[0087] b5, generating a noise matrix based on the feature matrix of the reference image, and adding the noise matrix to the frequency domain matrix of other sub-images in the same group which are not reference images;

[0088] Further, the step b5 of generating a noise matrix based on the feature matrix of the reference image comprises: performing weighted processing on the feature matrix of the reference image based on a preset differential privacy protection strength parameter; and generating a noise matrix with Laplace distribution according to the weighted feature matrix.

[0089] ​It should be noted that, based on the characteristics of the image array, the same pre-operator is used for the wavelet coefficient vector, that is, the same privacy budget parameter is used to weight the feature vector. The privacy budget ε represents the protection strength of differential privacy, and the smaller the ε is, the higher the protection coefficient is, and the higher the image distortion rate is, that is, the lower the data availability and image quality are.

[0090] For example, the noise matrix of Laplace distribution generated according to the weighted feature matrix can be represented by the following formula:

[0091]

[0092] Wherein, F represents the noise matrix, ε, Pi n The above steps are explained, and will not be repeated here.

[0093] The embodiment of the application generates a noise matrix of Laplace distribution by weighting the feature matrix of the reference image, so that the intensity of the added noise can be freely adjusted according to the differential privacy protection requirements. Moreover, based on the weighted noise generation method of the feature matrix, the redundant disturbance caused by random noise is avoided, and the statistical characteristics of the noise are ensured to meet the requirements of differential privacy through Laplace distribution. The combination of the two makes a balance between protecting privacy and maintaining data availability, so that the restored light field data set can protect privacy and improve data availability.

[0094] For example, the noise matrix F is added to the frequency domain matrix of other sub-images in the same group.

[0095] b6, wavelet inverse transform is performed on the frequency domain matrix of all sub-images after adding noise to obtain a processed second pixel matrix;

[0096] b7, encoding the image number, the second pixel matrix and the group number to obtain one-dimensional encoding data of each sub-image after processing.

[0097] Specifically, the wavelet inverse transform is used to restore the frequency domain matrix after adding noise to a pixel matrix, and the pixel matrix is processed by one-dimensional dimension reduction, and the same encoding method as in steps S102 to S104 is used to obtain the processed key-value pair key-vblue as a one-dimensional array set after differential privacy processing.

[0098] This invention provides a method to obtain a frequency domain matrix by performing wavelet transform on sub-images within the same group, selecting the sub-image at the center as a reference to generate a noise matrix, adding noise to other sub-images, and finally performing an inverse transform. This method ensures that sub-images within the same group maintain relatively consistent noise characteristics, avoiding the destruction of spatial correlation caused by inconsistent noise among sub-images. At the same time, the wavelet transform enables noise addition in the frequency domain, ensuring the visual quality of the image. It also maintains the spatial correlation of the light field image while protecting privacy, thereby improving the usability of the processed light field image dataset in practical applications.

[0099] Step S106: Based on the one-dimensional encoded data of each processed sub-image, rearrange each sub-image according to the lens array of the light field camera to obtain a light field image dataset with differential privacy protection.

[0100] In one specific embodiment, reference may be made to Figure 4 Step S106 includes:

[0101] c1 reads the chromaticity components (UV) of each sub-image from the preset path;

[0102] c2, restores the processed one-dimensional encoded data to the luminance component Y;

[0103] c3 merges the restored luminance component Y with the corresponding chrominance component UV to obtain a sub-image in YUV format.

[0104] c4 arranges all YUV format sub-images according to the lens array of the light field camera to obtain a differential privacy-preserving light field image dataset.

[0105] For example, the chromaticity components UV of the corresponding group number Gi and sub-image number Dj are read in batches from the preset storage path, and the processed luminance components (second pixel matrix) are read from the one-dimensional encoded data generated in step S105; the luminance components Y and chromaticity components UV of the same group number Gi and number Dj are matched; according to the YUV format, the luminance matrix Y and chromaticity matrix UV are merged pixel by pixel to generate a complete YUV format sub-image; according to the group number Gi and number Dj, the position of each sub-image in the two-dimensional array is determined based on the arrangement rules of the lens array, and the sub-images of the same group are arranged in order of position to stitch together a complete light field image.

[0106] The embodiment of the present application can restore the light field image protected by differential privacy by recombining the processed luminance component Y and the original stored chrominance component UV and arranging according to the lens array, adopts the component separation processing and recombination mode, avoids the image distortion caused by unnecessary processing of chrominance information, and further guarantees the spatial structure integrity of the light field image by arranging according to the lens array. Thus, the visual quality and spatial information of the image are maintained to the greatest extent while protecting privacy, thereby improving the usability of the processed light field image data set.

[0107] In actual application, the processed one-dimensional coded data generated in step S105 can be read, the group number and the number of the sub-image are extracted from the key, and the processed second pixel matrix is obtained from the value; the sub-image is classified into the corresponding light field image data group based on the group number Gi, and is arranged into a two-dimensional array according to the number Dj and the arrangement rule (row priority or column priority) of the lens array; for a 3*3 array, D1 corresponds to the upper left corner, and D9 corresponds to the lower right corner; the one-dimensional pixel data of each sub-image is restored into two-dimensional image data, the restored two-dimensional image data is combined into a multi-view data set of the light field image according to the layout of the lens array, and the rearranged light field image data set is stored in a preset format and re-uploaded to hdfs or local storage for subsequent use; if subsequent operations such as refocusing or three-dimensional modeling are required, the light field image data set protected by differential privacy is obtained from hdfs or local storage based on a preset interface.

[0108] The embodiment of the present application groups the light field image according to the shooting time and the lens array, numbers and extracts the pixel of the sub-image in each group, then performs one-dimensional coding and differential privacy processing, and finally rearranges according to the lens array, so that the spatial correlation between the sub-images is maintained while protecting the privacy of the light field image. Because the grouping processing mode is adopted, the problem of loss of position information caused by differential privacy processing of independent images is avoided, and the integrity of the light field image array information is ensured through one-dimensional coding and rearrangement, thereby solving the problem that the image data with spatial correlation cannot be effectively protected in the prior art, realizing the availability of the light field image data set while protecting privacy, and enabling subsequent refocusing, three-dimensional imaging and other applications to correctly restore the light field.

[0109] In the embodiment, a light field image data set processing system based on differential privacy is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0110] The embodiment provides a light field image dataset processing system based on differential privacy, as shown in the figure, comprising a data uploading module and a MapReduce module. Figure 5 The data uploading module 501 is used for grouping light field image datasets according to shooting time sequences and lens arrays of the light field cameras to obtain a plurality of light field image data groups and group numbers, wherein each light field image data group contains a plurality of sub-images shot by the light field camera at the same time; the image numbers of all the sub-images in each light field image data group are numbered to obtain a first pixel matrix; the image numbers, the first pixel matrix and the group numbers are encoded to obtain one-dimensional encoding data of each sub-image and are stored.

[0111] The MapReduce module 502 comprises a Map unit 5021 and a Reduce unit 5022.

[0112] The Map unit 5021 is used for processing the one-dimensional encoding data based on a differential privacy processing algorithm to obtain one-dimensional encoding data of each sub-image after processing.

[0113] The Reduce unit 5022 is used for rearranging each sub-image according to the lens array of the light field camera based on the one-dimensional encoding data of each sub-image after processing to obtain a light field image dataset protected by differential privacy.

[0114] It should be noted that, in actual application, MapReduce is a distributed computing framework for processing large-scale datasets, and is suitable for scenarios requiring large-scale parallel computing. The core thereof comprises two stages: a Map stage and a Reduce stage, and hdfs (Hadoop Distributed File System) is a basic storage system in the MapReduce framework.

[0115] The Map stage is responsible for decomposing input data into a plurality of small tasks (i.e., processing the one-dimensional encoding data), and performing parallel execution by different nodes in a distributed system, and the Reduce stage processes the grouped intermediate results, and performs summarization, merging or further calculation, and finally outputs the results (i.e., processing the one-dimensional encoding data to obtain one-dimensional encoding data of each sub-image after processing).

[0116] In the application process, the embodiment of the present application can refer to

[0117] The embodiment of the present application can refer to Figure 6Light field image array -> grouping (get group1 to groupN) -> segmentation (get multiple sub-images) -> (one-dimensional processing, expressed in the form of key-value) -> input MapReduce module Map unit respectively based on key-value to calculate the characteristics, after inverse transformation to get the processed key-value and output -> Reduce unit re-encoding as an image matrix, and restore the light field image array.

[0118] In some optional embodiments, the data uploading module 501 specifically comprises: an image association unit, configured to associate the image number, the first pixel matrix, and the group number according to the lens array to obtain sub-image metadata of each sub-image after association and store the sub-image metadata; wherein the sub-image metadata is in the form of key-value pair, the key is path information composed of at least the image number and the group number, and the value is the first pixel matrix.

[0119] In some optional embodiments, the Map unit 5021 specifically comprises:

[0120] a metadata aggregation sub-unit, configured to aggregate the sub-image metadata of the same group based on the group number;

[0121] a pixel matrix reading sub-unit, configured to read the first pixel matrix of each sub-image in the sub-image metadata of the same group based on the path information;

[0122] a wavelet transform sub-unit, configured to perform wavelet transform on the first pixel matrix of all sub-images in the same group to obtain a frequency domain matrix corresponding to each sub-image;

[0123] a feature matrix extraction sub-unit, configured to select a sub-image at a center position of each group as a reference image, and take the frequency domain matrix of the reference image as a feature matrix;

[0124] a noise adding sub-unit, configured to generate a noise matrix based on the feature matrix of the reference image, and add the noise matrix to the frequency domain matrix of other sub-images in the same group which are not the reference image;

[0125] a wavelet inverse transform sub-unit, configured to perform wavelet inverse transform on the frequency domain matrix of all sub-images after adding the noise to obtain a processed second pixel matrix;

[0126] an encoding generation sub-unit, configured to encode the image number, the second pixel matrix, and the group number to obtain one-dimensional encoded data of each sub-image after processing.

[0127] In some optional embodiments, the key in the sub-image metadata is path information composed of at least the image number, the image encoding mode, and the group number, and if the image encoding mode is YUV format, the data uploading module 501 specifically comprises:

[0128] a luminance coding unit, configured to code each sub-image in YUV format to obtain a luminance component Y and a chrominance component UV;

[0129] a matrix generating unit, configured to perform pixel extraction on the luminance component Y to obtain a first pixel matrix, and store the chrominance component UV according to the path information to a preset path.

[0130] In some optional embodiments, the Reduce unit 5022 comprises:

[0131] a chrominance component reading sub-unit, configured to read the chrominance component UV corresponding to each sub-image from the preset path;

[0132] a luminance component restoring sub-unit, configured to restore the processed one-dimensional coded data to the luminance component Y;

[0133] a YUV merging sub-unit, configured to merge the restored luminance component Y and the corresponding chrominance component UV to obtain a sub-image in YUV format;

[0134] an image arranging sub-unit, configured to arrange all the sub-images in YUV format according to the lens array of the light field camera to obtain a light field image data set protected by differential privacy.

[0135] In some optional embodiments, the data uploading module 501 specifically comprises:

[0136] a weighting processing unit, configured to perform weighting processing on the feature matrix of the reference image based on a preset differential privacy protection strength parameter;

[0137] a noise matrix generating unit, configured to generate a noise matrix with Laplace distribution according to the weighted feature matrix.

[0138] Further function descriptions of the above-mentioned various modules and units are the same as those of the above-mentioned corresponding embodiments, and will not be repeated here.

[0139] The light field image data set processing system based on differential privacy in the embodiment is presented in the form of functional units, and the units herein refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories executing one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0140] The embodiment of the application further provides a computer device having the light field image data set processing system based on differential privacy shown in the above Figure 5 .

[0141] Please refer to Figure 7 .Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0142] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0143] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0144] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0146] The computer device also comprises a communication interface 30 for communication of the computer device with other devices or communication networks.

[0147] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code originally stored in a remote storage medium or non-transitory machine readable storage medium and downloaded to a local storage medium and stored in the local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0148] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be invoked or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of computer program instructions executed by a computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0149] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the defined scope.

Claims

1. A method for processing light field image datasets based on differential privacy, characterized in that, The method includes: The light field image dataset is grouped according to the shooting time order and the lens array of the light field camera to which it belongs, resulting in multiple light field image data groups and group numbers. Each light field image data group contains multiple sub-images captured by the light field camera at the same time. The image number is obtained by numbering all sub-images in each light field image data group; The first pixel matrix is ​​obtained by extracting pixels from all sub-images in each light field image data group; The image number, the first pixel matrix, and the group number are encoded to obtain one-dimensional encoded data for each sub-image and then stored. The one-dimensional encoded data is processed based on the differential privacy processing algorithm to obtain the one-dimensional encoded data of each sub-image after processing; Based on the one-dimensional encoded data of each processed sub-image, each sub-image is rearranged according to the lens array of the light field camera to obtain a differential privacy-preserving light field image dataset.

2. The method according to claim 1, characterized in that, The process of encoding the image number, the first pixel matrix, and the group number to obtain and store the one-dimensional encoded data of each sub-image specifically includes: The image number, the first pixel matrix, and the group number are associated according to the lens array to obtain and store the sub-image metadata of each associated sub-image; The sub-image metadata is in the form of key-value pairs, where the key is path information consisting of at least the image number and group number, and the value is the first pixel matrix.

3. The method according to claim 2, characterized in that, The differential privacy processing algorithm is used to process the one-dimensional encoded data to obtain the processed one-dimensional encoded data for each sub-image, specifically including: Aggregate the metadata of sub-images in the same group based on the group number; Based on the path information, read the first pixel matrix of each sub-image in the metadata of the sub-images in the same group; Perform wavelet transform on the first pixel matrix of all sub-images within the same group to obtain the frequency domain matrix corresponding to each sub-image; The sub-image at the center position of each group is selected as the reference image, and the frequency domain matrix of the reference image is used as the feature matrix; A noise matrix is ​​generated based on the feature matrix of the reference image, and the noise matrix is ​​added to the frequency domain matrix of other sub-images of non-reference images in the same group; Perform inverse wavelet transform on the frequency domain matrix of all sub-images after adding noise to obtain the processed second pixel matrix; The image number, the second pixel matrix, and the group number are encoded to obtain the one-dimensional encoded data of each sub-image after processing.

4. The method according to claim 2, characterized in that, The key in the sub-image metadata is path information consisting of at least image number, image encoding method, and group number; if the image encoding method is YUV format, then the step of extracting pixels from all sub-images in each light field image data group to obtain the first pixel matrix specifically includes: Each sub-image is encoded in YUV format to obtain the luminance component Y and the chrominance component UV. The luminance component Y is extracted into pixels to obtain a first pixel matrix, and the chrominance components UV are stored in a preset path according to the path information.

5. The method according to claim 4, characterized in that, The one-dimensional encoded data based on each processed sub-image is rearranged according to the lens array of the light field camera to obtain a differential privacy-preserving light field image dataset, specifically including: Read the chromaticity components (UV) of each sub-image from the preset path; The processed one-dimensional encoded data is restored to the luminance component Y; The restored luminance component Y is merged with the corresponding chrominance components UV to obtain a sub-image in YUV format. Arrange all YUV format sub-images according to the lens array of the light field camera to obtain a differential privacy-preserving light field image dataset.

6. The method according to claim 3, characterized in that, The generation of the noise matrix based on the feature matrix of the reference image specifically includes: The feature matrix of the reference image is weighted based on the preset differential privacy protection strength parameters; Generate a Laplace-distributed noise matrix based on the weighted feature matrix.

7. A light field image dataset processing system based on differential privacy, characterized in that, The system includes a data upload module and a MapReduce module; The data upload module is used to group the light field image dataset according to the shooting time order and the lens array of the light field camera to obtain multiple light field image data groups and group numbers. Each light field image data group contains multiple sub-images captured by the light field camera at the same time. All sub-images in each light field image data group are numbered to obtain image numbers. Pixels of all sub-images in each light field image data group are extracted to obtain a first pixel matrix. The image numbers, the first pixel matrix, and the group number are encoded to obtain one-dimensional encoded data for each sub-image and stored. The MapReduce module includes a Map unit and a Reduce unit; The Map unit is used to process one-dimensional encoded data based on the differential privacy processing algorithm to obtain one-dimensional encoded data of each sub-image after processing. The Reduce unit is used to rearrange each sub-image according to the lens array of the light field camera based on the one-dimensional encoded data of each processed sub-image to obtain a differential privacy-preserving light field image dataset.

8. A computer device, characterized in that, The computer device includes a memory and a processor; A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the light field image dataset processing method based on differential privacy as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the differential privacy-based light field image dataset processing method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the differential privacy-based light field image dataset processing method according to any one of claims 1-6.

Images