An adaptive contrast enhancement method and processing system based on light field sub-aperture image
Through an adaptive contrast enhancement processing system based on the optical field sub-aperture image, the AI acceleration module and DDR3 storage unit are used to solve the problem of insufficient complex light conditions and contrast in traditional technology, real-time adaptive enhancement and efficient processing of images are achieved. For a variety of image processing requirements.
Patent Information
- Application Number
- CN202410710315.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-06-03
AI Technical Summary
Traditional image acquisition and display technology has limitations in complex light conditions and in image processing with insufficient contrast, and it is necessary to improve the performance of image processing algorithms and hardware devices.
An adaptive contrast enhancement processing system based on optical field sub-aperture image is designed, including an image acquisition device, an optical field image processing device and a display device. The original light is used by a DDR3 storage unit and an AI acceleration module to use the preprocessing and algorithm processing unit to deal with the original light. The field data is adaptively enhanced in light field contrast, recovers and compresses the sub-aperture image, and is transmitted and displayed over the network.
Real-time adaptive contrast enhancement of images is achieved, image quality and processing efficiency are improved, and it is suitable for scenarios with high real-time requirements, significantly improving processing speed and user experience.
Smart Images

Figure CN118537275B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an adaptive contrast enhancement method and processing system based on light field sub-aperture images. Background Art
[0002] In real life, image acquisition and display technology has been widely used in various fields, such as medical imaging, industrial detection, security monitoring, etc. However, in some application scenarios, traditional image acquisition and display technology has some limitations, such as image acquisition under complex lighting conditions and image display with insufficient contrast, which requires further improvement of the performance of image processing algorithms and hardware equipment. Summary of the invention
[0003] The present invention provides an adaptive contrast enhancement method and processing system based on light field sub-aperture images, which solves the problem of improving image processing algorithms and hardware equipment. The technical solution is as follows:
[0004] An adaptive contrast enhancement processing system based on light field sub-aperture images comprises a camera acquisition device, a light field image processing device and a light field image display device connected in sequence, wherein the light field image processing device is also connected to a DDR3 storage unit and an AI acceleration module; the light field image processing device comprises a network receiving unit, a preprocessing unit, an algorithm processing unit and a network transmission unit connected in sequence; the network receiving unit receives original light field data from the camera acquisition device; the preprocessing unit preprocesses the original light field data to obtain a full focus map and a depth map; the algorithm processing unit performs adaptive light field contrast enhancement processing on the input full focus map, restores a sub-aperture image according to the enhanced full focus map and depth map, and compresses the sub-aperture image; the network transmission unit transmits the compressed sub-aperture image to the light field image display device.
[0005] The video acquisition device includes an optical system module, an image sensor module, and an image information transmission end connected in sequence. The optical system module 11 is composed of a main lens, a microlens array, a color filter array, and a sensor array, and is used to obtain the position, direction, and color information of the light field; the image sensor module is used to record the position, direction, and color information of the acquired light field, and generate original light field data; the image information transmission end is used to save the original light field data generated by the image sensor module and transmit it to the light field image processing module.
[0006] The DDR3 storage unit is used to store the original light field data received by the network receiving unit; after the AI acceleration module receives the image data transmitted from the preprocessing unit or the adaptive algorithm processing unit, it analyzes and infers the transmitted image data to realize external hardware acceleration and realize real-time image processing.
[0007] According to the above method of the adaptive contrast enhancement processing system based on light field sub-aperture image, the following steps are included:
[0008] Step S1: The camera acquisition device obtains and saves the original light field data, and transmits it to the light field image processing device;
[0009] Step S2: After the network receiving unit of the light field image processing device receives the original light field data, the pre-processing unit processes the original light field data to obtain a full focus map and a depth map, and then transmits the data to the algorithm processing unit to perform real-time adaptive light field contrast enhancement algorithm processing on the full focus map;
[0010] The algorithm processing unit restores the sub-aperture image for display through the enhanced full focus image and depth image, compresses and encodes the sub-aperture image, and transmits it to the light field image display device for display through the network transmission unit.
[0011] In step S2, the specific method of light field image processing is:
[0012] Step S101, preprocessing the received raw light field data by a preprocessing unit to obtain a full focus map and a depth map;
[0013] Step S102, performing adaptive contrast enhancement on the full focus image obtained after preprocessing by an algorithm processing unit;
[0014] Step S103, the algorithm processing unit restores the sub-aperture image according to the enhanced full focus image and the depth image, and compresses and encodes the sub-aperture image;
[0015] Step S104: the network transmission unit transmits the compressed sub-aperture image to the light field image display device.
[0016] In step S101, decoding the collected raw light field data includes the following steps: reading the raw light field data, setting decoding options and selecting an appropriate white image according to the camera model data information, decoding the lens image using the white image to obtain light field image data, and then generating a full focus map and a depth map through the obtained light field image data.
[0017] The algorithm processing unit performs an operation of adaptively enhancing contrast of the light field image, comprising the following steps:
[0018] Step S201, obtaining a full focus image;
[0019] Step S202, converting the RGB image to a grayscale image;
[0020] Step S203, using two parameters gain1 and gain2 to adjust the mapping curve of the grayscale histogram equalization algorithm to enhance the contrast of the image;
[0021] Step S204: restore the sub-aperture image through the full focus image and the depth image.
[0022] Furthermore, in step S203, the specific method of adjusting the mapping curve of the grayscale histogram equalization algorithm includes the following steps:
[0023] S401: Probability density function of grayscale image, used to describe the frequency of occurrence of each grayscale level in the image, n k is the input image pixel value r k The frequency of occurrence, n is the total number of pixels in the image, k is the grayscale value range of the image k=1,2,...,N, N is the maximum grayscale value, the specific formula is as follows:
[0024] ;
[0025] S402: Cumulative density function, used to describe the cumulative distribution of gray levels in the entire gray range, P(r i ) is the frequency of occurrence of the i-th gray level i=0,1,2...,k, k is the gray value range of the image k=1,2,...,N, N is the maximum gray value, the specific formula is as follows:
[0026] ;
[0027] S403: Histogram equalization mapping, using the cumulative density function to map the light field image to X0 to X n-1 In the range of [X 0, X n-1 ] is the grayscale value range of the input image, X0 is the minimum value of the grayscale range, X n-1 is the maximum value of the grayscale range, C(X) represents the cumulative distribution function when the grayscale value is X, and the specific formula of the mapping curve is as follows:
[0028] ;
[0029] S404: Adjust the mapping curve by setting two parameters gain1 and gain2 to coordinate the mapping curve, where gain1 is used to adjust the brighter part of the image, gain2 is used to adjust the darker part, T' is the adjusted mapping curve, T0 is the original gray value corresponding curve, T1 is the HE mapping curve, t is 0 or 1, which is used to adjust the dark part and the bright part respectively, and gain(t) is a value between 0 and 1. The specific formula is as follows:
[0030] .
[0031] Furthermore, in step S2, the real-time adaptive light field contrast enhancement algorithm processing is implemented by training the network, and the steps of the training network processing are as follows:
[0032] Step S301, generating a light field image data set;
[0033] Step S302, network architecture design, the input layer is the feature information of the image histogram value and brightness, the hidden layer uses a multi-layer perceptron, the output layer outputs adaptive parameters gain1 and gain2, and ReLU is used as the activation function;
[0034] Step S303, using the sum of square error function as the loss function;
[0035] Step S304: the test error of the adaptive parameter value obtained through MLP network training is less than 0.2.
[0036] Furthermore, in step S203, contrast enhancement is performed on the image, and the steps of using the training network are as follows:
[0037] S501: Create a data set and extract features from the data set. During training, the input features include brightness information extracted from the full-focus image, including 16 histogram values (L0, ..., L 15 ) and the maximum, minimum and average brightness (L 16 , L 17 , L 18 ), L k is the k-th normalized histogram value, indicating the relative frequency of the k-th gray level; P k represents the kth histogram value, that is, the number of pixels in the image at the kth grayscale level; the denominator is the sum of the histogram values of all 16 grayscale levels, which is used for normalization, k=0,1,...,15, and the calculation formula is as follows:
[0038] ;
[0039] Where P k is the number of pixels at the kth gray level; k represents a natural number, and i is a natural number;
[0040] S502: The feature information of the above 19 groups of images is used as the input layer, and the output layer is the adaptive parameters gain1 and gain2, where gain1 is used to adjust the brighter part of the image, and gain2 is used to adjust the darker part. The output value of the output layer is given by the following formula:
[0041] ;
[0042] Among them, w is the weight, x represents the input feature value, namely L0, L1, ..., L 18 , ℎ k represents the sum of the output weights of the k neurons in the output layer, x jk represents the eigenvalue associated with the jth feature in the input layer and the kth neuron in the output layer; w jk represents the weight connecting the jth input feature of the input layer and the kth neuron of the output layer; j is an index variable used to traverse all input features in the input layer, ranging from 0 to M; M is the number of feature values; k is an index variable used to traverse all neurons in the output layer; h k is the weighted input sum of the kth neuron in the output layer; a k is the activation value of the kth neuron in the output layer;
[0043] The g(·) is an activation function, and the rectified linear activation unit (ReLU) is used as the activation function. The formula is as follows:
[0044] ;
[0045] The loss function E uses the error square sum function, and the formula is as follows:
[0046] ;
[0047] Where N is the total number of samples; during training, it represents the number of samples used to calculate the loss function; k is an index variable used to traverse all samples, ranging from 0 to N-1; y k represents the output value of the kth sample predicted by the model, and t is the true value of the kth sample;
[0048] S503: Using 19 sets of data representing image feature information as input and gain1 and gain2 as output, the light field image is trained through an MLP network to establish a relationship between the input and the output.
[0049] Compared with the prior art, the present invention has the following beneficial effects: an adaptive contrast enhancement algorithm based on light field sub-aperture images is proposed, and real-time adaptive contrast enhancement of images is achieved through full focus maps and depth maps, image details are retained, and contrast is improved, thereby having higher quality and efficiency in image processing; hardware acceleration processing is performed using the Atlas 200 AI chip platform, which can significantly improve image processing efficiency and speed, and is particularly suitable for scenes with high real-time requirements, making the processing speed faster and more efficient; the present invention constructs a complete image processing flow, including image acquisition, processing, display and other links, which can meet various image processing needs, and the light field image display device can display the processed image results in real time, and users can observe the image processing effect in time, thereby improving user experience and operating efficiency, and having a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a structural schematic diagram of the adaptive contrast enhancement processing system based on light field sub-aperture images;
[0051] Figure 2 is a schematic diagram of the optical system module;
[0052] Figure 3 is a schematic flow chart of the light field image adaptive contrast enhancement method;
[0053] Figure 4 is a schematic diagram of the operating state of the light field image adaptive contrast enhancement method;
[0054] Figure 5 is a training network structure of the light field image adaptive contrast enhancement method;
[0055] Figure 6 is a flow chart of steps of the display device;
[0056] The meaning of the serial numbers in the figure are: 1-photographic acquisition device; 11-optical system module; 12-image sensor module; 13-image information transmission end; 2-light field image processing device; 21-network receiving unit; 22-preprocessing unit; 23-algorithm processing unit; 24-network transmission unit; 25-DDR3 storage unit; 26-AI acceleration module; 3-light field image display device; 31-network receiving unit; 32 display unit. DETAILED DESCRIPTION
[0057] like Figure 1As shown, the adaptive contrast enhancement processing system based on light field sub-aperture images includes a camera acquisition device 1, a light field image processing device 2 and a light field image display device 3 connected in sequence, and the light field image processing device 2 is also connected to a DDR3 storage unit 25 and an AI acceleration module 26. Among them:
[0058] The camera acquisition device 1 is used to acquire and record the original light field data and transmit it to the next level;
[0059] The light field image processing device 2, the DDR3 storage unit 25 and the AI acceleration module 26 are used to process the original light field data in real time to obtain a sub-aperture image that can be used for a display device;
[0060] The light field image display device 3 is used to display the processed sub-aperture image.
[0061] The video acquisition device 1 comprises an optical system module 11, an image sensor module 12, and an image information transmission terminal 13 which are connected in sequence.
[0062] The optical system module 11 is composed of a main lens, a microlens array, a color filter array and a sensor array to obtain the position, direction and color information of the light field;
[0063] The image sensor module 12 is used to record the position, direction and color information of the acquired light field and generate raw light field data;
[0064] The image information transmission terminal 13 is used to store the original light field data generated by the image sensor module and transmit it to the light field image processing module 2 .
[0065] like Figure 2 As shown, the main structure of the optical system module 11 includes a sensor array, a microlens array and a main lens. The microlens array is an array composed of many tiny lenses. Each microlens corresponds to a macro pixel on the sensor array. Each macro pixel corresponds to a position sampling of the light field, and each point in the macro pixel corresponds to a direction sampling of the light field at the position. All macro pixels together constitute the sampling of the light field at each point on the lens aperture and each microlens position.
[0066] The light field image processing device 2 is used to receive the original light field data transmitted by the camera acquisition device 1 and transmit the data to the light field image display device 3 after processing, and includes a network receiving unit 21, a preprocessing unit 22, an algorithm processing unit 23 and a network transmission unit 24 connected in sequence. Among them:
[0067] The network receiving unit 21 receives the original light field data from the photographic acquisition device 1;
[0068] The preprocessing unit 22 preprocesses the original light field data to obtain a full focus map and a depth map, accelerates the processing and calculation process by calling the AI acceleration module 26, and transmits the processed full focus map to the algorithm processing unit 23 for the next step of adaptive light field contrast enhancement processing;
[0069] The algorithm processing unit 23, after receiving the full focus map data transmitted from the preprocessing unit 22, performs adaptive light field contrast enhancement processing on the transmitted full focus map, restores the sub-aperture image according to the enhanced full focus map and depth map, and compresses the sub-aperture image. The processing and calculation process is accelerated by calling the AI acceleration module 26, thereby realizing real-time adaptive light field contrast enhancement based on the full focus map and depth map;
[0070] The network transmission unit 24 transmits the compressed sub-aperture image to the light field image display device 3 .
[0071] Furthermore, the DDR3 storage unit 25 is used to store the original light field data received by the network receiving unit 21, and the preprocessing unit 22 reads the original light field data from the DDR3 storage unit 25 and then performs preprocessing;
[0072] The AI acceleration module 26 adopts the Atlas 200 AI acceleration module, which is connected to the light field image processing device 2 through the PCIE bus. When the AI acceleration module 26 receives the image data transmitted from the preprocessing unit 22 or the adaptive algorithm processing unit 23, it analyzes and infers the image data transmitted, and realizes external hardware acceleration to realize real-time image processing. The Atlas 200 AI chip platform has the characteristics of high performance and low power consumption, can effectively support the processing requirements of light field images, and provide stable and fast image processing capabilities.
[0073] The adaptive contrast enhancement method based on light field sub-aperture image comprises the following steps:
[0074] Step S1: the camera acquisition device 1 obtains and saves the original light field data, and transmits it to the light field image processing device 2;
[0075] Step S2: After the network receiving unit 21 of the light field image processing device 2 receives the original light field data, the pre-processing unit 22 processes the original light field data to obtain a full focus map and a depth map, and then transmits the full focus map to the algorithm processing unit 23 for real-time adaptive light field contrast enhancement algorithm processing. The algorithm processing unit 23 restores the sub-aperture image for display through the enhanced full focus map and depth map, compresses and encodes the sub-aperture image, and transmits it to the light field image display device 3 through the network transmission unit 24 for display.
[0076] In step S1, the specific method for collecting the original light field data is:
[0077] The video acquisition device 1 records light information from different directions by adding a microlens array in front of the camera sensor, stores the collected light field information in an external storage card of the device, and transmits the original light field data to the light field image processing device 2 through a communication module.
[0078] like Figure 3 As shown, further, in step S2, the specific method of light field image processing is:
[0079] Step S101, preprocessing the received raw light field data by the preprocessing unit 22 to obtain a full focus map and a depth map;
[0080] Step S102, performing adaptive contrast enhancement on the full focus image obtained after preprocessing by the algorithm processing unit 23;
[0081] Step S103, the algorithm processing unit 23 restores the sub-aperture image according to the enhanced full focus image and depth image, and compresses and encodes the sub-aperture image to improve transmission efficiency;
[0082] Step S104 : the network transmission unit 24 transmits the compressed sub-aperture image to the light field image display device 3 .
[0083] The decoding of the collected original light field data in step S101 includes the following steps:
[0084] Read the original light field data, set the decoding options and select the appropriate white image according to the camera model data information, use the white image to decode the lens image, and obtain the light field image data. The decoding process includes steps such as de-mosaicing, color correction, and gamma correction. Then generate the full focus map and depth map through the obtained light field image data.
[0085] like Figure 4 As shown, the algorithm processing unit 23 performs an adaptive contrast enhancement operation on the light field image, including the following steps:
[0086] Step S201, obtaining a full focus image;
[0087] Step S202, converting the RGB image to a grayscale image;
[0088] Step S203, using two parameters gain1 and gain2 to adjust the mapping curve of the grayscale histogram equalization algorithm to enhance the contrast of the image;
[0089] Step S204: restore the sub-aperture image through the full focus image and the depth image.
[0090] Furthermore, in step S203, the specific method of adjusting the mapping curve of the grayscale histogram equalization algorithm includes the following steps:
[0091] S401: Probability density function of grayscale image, describing the frequency of occurrence of each grayscale level in the image. k is the input image pixel value r k The frequency of occurrence, n is the total number of pixels in the image, k is the grayscale value range of the image k=1,2,...,N, N is the maximum grayscale value, the specific formula is as follows:
[0092] ;
[0093] S402: Cumulative density function, calculated based on probability density function, used to describe the cumulative distribution of gray levels in the entire gray range, where P(r i ) is the frequency of occurrence of the i-th gray level i=0,1,2...,k, k is the gray value range of the image k=1,2,...,N, N is the maximum gray value, the specific formula is as follows:
[0094] ;
[0095] S403: Histogram Equalization (HE) mapping process, using the cumulative density function to map the light field image to X0 to X n-1 In the range of [X 0, X n-1 ] is the grayscale value range of the input image, X0 is the minimum value of the grayscale range, X n-1 is the maximum value of the grayscale range, C(X) represents the cumulative distribution function when the grayscale value is X, and the specific formula of the mapping curve is as follows:
[0096] ;
[0097] S404: Adjust the mapping curve by setting two parameters gain1 and gain2 to coordinate the mapping curve, where gain1 is used to adjust the brighter part of the image, gain2 is used to adjust the darker part, T' is the adjusted mapping curve, T0 is the original gray value corresponding curve, T1 is the HE mapping curve, t is 0 or 1, which is used to adjust the dark part and the bright part respectively, and gain(t) is a value between 0 and 1. The specific formula is as follows:
[0098] ;
[0099] The design framework of the training network for adaptive contrast enhancement of light field images. The processing steps are as follows:
[0100] Step S301, generating a light field image data set;
[0101] Step S302, network architecture design, the input layer is the feature information of the image histogram value and brightness, the hidden layer uses a multi-layer perceptron, the output layer outputs adaptive parameters gain1 and gain2, and ReLU is used as the activation function;
[0102] Step S303, using the sum of square error function as the loss function;
[0103] Step S304: the test error of the adaptive parameter value obtained through MLP network training is less than 0.2.
[0104] like Figure 5 As shown, in step S203, the image is contrast enhanced, and the steps of using the corresponding training network structure are as follows:
[0105] S501: Create a data set and extract features from the data set. During training, the input features include brightness information extracted from the full-focus image, including 16 histogram values (L0, ..., L 15 ) and the maximum, minimum and average brightness (L 16 , L 17 , L 18 ), L k is the k-th normalized histogram value, indicating the relative frequency of the k-th gray level; P k represents the kth histogram value, that is, the number of pixels in the image at the kth grayscale level; the denominator is the sum of the histogram values of all 16 grayscale levels, which is used for normalization, k=0,1,...,15, and the calculation formula is as follows:
[0106] ;
[0107] Where P k is the number of pixels at the kth gray level; k represents a natural number, and i is a natural number.
[0108] S502: The feature information of the above 19 groups of images is used as the input layer, and the output layer is the adaptive parameters gain1 and gain2, where gain1 is used to adjust the brighter part of the image, and gain2 is used to adjust the darker part. The output value of the output layer is given by the following formula:
[0109] ;
[0110] Among them, w is the weight, x represents the input feature value, namely L0, L1, ..., L 18 , ℎ k represents the sum of the output weights of the k neurons in the output layer, x jk: The feature value associated with the jth feature in the input layer and the kth neuron in the output layer; w jk : The weight connecting the jth input feature of the input layer and the kth neuron of the output layer; j is an index variable used to traverse all input features in the input layer, ranging from 0 to M, where M is the number of feature values; k is an index variable used to traverse all neurons in the output layer; h k is the weighted input sum of the kth neuron in the output layer; a k is the activation value of the kth neuron in the output layer.
[0111] g(·) is the activation function, and the rectified linear activation unit (ReLU) is used as the activation function. The formula is as follows:
[0112] ;
[0113] The loss function E uses the error square sum function, and the formula is as follows:
[0114] ;
[0115] Where N is the total number of samples. During training, it indicates the number of samples used to calculate the loss function. k is an index variable used to traverse all samples. The value range is from 0 to N-1. k The output value of the kth sample predicted by the model. t is the true value (target value) of the kth sample.
[0116] S503: Using 19 sets of data representing image feature information as input and gain1 and gain2 as output, the light field image is trained through an MLP network (Multilayer Perceptron) to establish a relationship between the input and the output.
[0117] The light field image display device 3 comprises a network receiving unit 31 and a display unit 32. The network receiving unit 31 receives the compressed sub-aperture image transmitted by the light field image processing module 2, and transmits it to the display unit 32 through a signal line for decoding and display.
[0118] like Figure 6 As shown, it is a specific flow chart of the light field image display device 3:
[0119] Step S601, the light field image display device 3 receives the compressed sub-aperture image transmitted from the light field image processing device 2 through the network receiving unit 31;
[0120] Step S602: the display unit 32 decodes the compressed sub-aperture image and displays it on the display.
[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An adaptive contrast enhancement processing system based on light field sub-aperture images, characterized in that: It comprises a camera acquisition device, a light field image processing device and a light field image display device connected in sequence, wherein the light field image processing device is also connected to a DDR3 storage unit and an AI acceleration module; the light field image processing device comprises a network receiving unit, a preprocessing unit, an algorithm processing unit and a network transmission unit connected in sequence; the network receiving unit receives the original light field data from the camera acquisition device; the preprocessing unit preprocesses the original light field data to obtain a full focus map and a depth map; the algorithm processing unit performs adaptive light field contrast enhancement processing on the input full focus map, restores a sub-aperture image according to the enhanced full focus map and depth map, and compresses the sub-aperture image; The network transmission unit transmits the compressed sub-aperture image to the light field image display device; The algorithm processing unit performs an operation of adaptively enhancing contrast of the light field image, comprising the following steps: Step S201, obtaining a full focus image; Step S202, converting the RGB image to a grayscale image; Step S203, using two parameters gain1 and gain2 to adjust the mapping curve of the grayscale histogram equalization algorithm to enhance the contrast of the image; wherein gain1 is used to adjust the brighter part of the image, and gain2 is used to adjust the darker part; Step S204: restore the sub-aperture image through the full focus image and the depth image.
2. The adaptive contrast enhancement processing system based on light field sub-aperture images according to claim 1, characterized in that: The video acquisition device comprises an optical system module, an image sensor module, and an image information transmission end connected in sequence, wherein the optical system module is composed of a main lens, a microlens array, a color filter array, and a sensor array, and is used to obtain the position, direction, and color information of the light field; The image sensor module is used to record the position, direction and color information of the acquired light field and generate original light field data; the image information transmission end is used to store the original light field data generated by the image sensor module and transmit it to the light field image processing module.
3. The adaptive contrast enhancement processing system based on light field sub-aperture images according to claim 1, characterized in that: The DDR3 storage unit is used to store the original light field data received by the network receiving unit; after the AI acceleration module receives the image data transmitted from the preprocessing unit or the adaptive algorithm processing unit, it analyzes and infers the transmitted image data to realize external hardware acceleration and realize real-time image processing.
4. The adaptive contrast enhancement method based on light field sub-aperture images implemented by the adaptive contrast enhancement processing system based on light field sub-aperture images according to any one of claims 1 to 3 comprises the following steps: Step S1: The camera acquisition device obtains and saves the original light field data, and transmits it to the light field image processing device; Step S2: After the network receiving unit of the light field image processing device receives the original light field data, the pre-processing unit processes the original light field data to obtain a full focus map and a depth map, and then transmits the data to the algorithm processing unit to perform real-time adaptive light field contrast enhancement algorithm processing on the full focus map; The algorithm processing unit restores the sub-aperture image for display through the enhanced full focus image and depth image, compresses and encodes the sub-aperture image, and transmits it to the light field image display device for display through the network transmission unit.
5. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 4, characterized in that: In step S2, the specific method of light field image processing is: Step S101, preprocessing the received raw light field data by a preprocessing unit to obtain a full focus map and a depth map; Step S102, performing adaptive contrast enhancement on the full focus image obtained after preprocessing by an algorithm processing unit; Step S103, the algorithm processing unit restores the sub-aperture image according to the enhanced full focus image and the depth image, and compresses and encodes the sub-aperture image; Step S104: the network transmission unit transmits the compressed sub-aperture image to the light field image display device.
6. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 5, characterized in that: In step S101, decoding the collected raw light field data includes the following steps: reading the raw light field data, setting decoding options and selecting an appropriate white image according to the camera model data information, decoding the lens image using the white image to obtain light field image data, and then generating a full focus map and a depth map through the obtained light field image data.
7. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 4, characterized in that: The algorithm processing unit performs an operation of adaptively enhancing contrast of the light field image, comprising the following steps: Step S201, obtaining a full focus image; Step S202, converting the RGB image to a grayscale image; Step S203, using two parameters gain1 and gain2 to adjust the mapping curve of the grayscale histogram equalization algorithm to enhance the contrast of the image; Step S204: restore the sub-aperture image through the full focus image and the depth image.
8. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 7, characterized in that: In step S203, the specific method of adjusting the mapping curve of the grayscale histogram equalization algorithm includes the following steps: S401: Probability density function of grayscale image, used to describe the frequency of occurrence of each grayscale level in the image, n k is the input image pixel value r k The frequency of occurrence, n is the total number of pixels in the image, k is the grayscale value range of the image k=1,2,...,N, N is the maximum grayscale value, the specific formula is as follows: ; S402: Cumulative density function, used to describe the cumulative distribution of gray levels in the entire gray range, P(r i ) is the frequency of occurrence of the i-th gray level i=0,1,2...,k, k is the gray value range of the image k=1,2,...,N, N is the maximum gray value, the specific formula is as follows: ; S403: Histogram equalization mapping, using the cumulative density function to map the light field image to X0 to X n-1 In the range of [X 0, X n-1 ] is the grayscale value range of the input image, X0 is the minimum value of the grayscale range, X n-1 is the maximum value of the grayscale range, C(X) represents the cumulative distribution function when the grayscale value is X, and the specific formula of the mapping curve is as follows: ; S404: Adjust the mapping curve by setting two parameters gain1 and gain2 to coordinate the mapping curve, where gain1 is used to adjust the brighter part of the image, gain2 is used to adjust the darker part, T' is the adjusted mapping curve, T0 is the original gray value corresponding curve, T1 is the HE mapping curve, t is 0 or 1, which is used to adjust the dark part and the bright part respectively, and gain(t) is a value between 0 and 1. The specific formula is as follows: 。 9. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 4, characterized in that: In step S2, the real-time adaptive light field contrast enhancement algorithm processing is implemented by training the network, and the steps of the training network processing are as follows: Step S301, generating a light field image data set; Step S302, network architecture design, the input layer is the feature information of the image histogram value and brightness, the hidden layer uses a multi-layer perceptron, the output layer outputs adaptive parameters gain1 and gain2, and ReLU is used as the activation function; Step S303, using the sum of square error function as the loss function; Step S304: the test error of the adaptive parameter value obtained through MLP network training is less than 0.
2.
10. The method for adaptive contrast enhancement based on light field sub-aperture images according to claim 7, characterized in that: Step S203, contrast enhancement of the image is performed, and the steps of using the training network are as follows: S501: Create a data set and extract features from the data set. During training, the input features include brightness information extracted from the full-focus image, including 16 histogram values (L0, ..., L 15 ) and the maximum, minimum and average brightness (L 16 , L 17 , L 18 ), L k is the k-th normalized histogram value, indicating the relative frequency of the k-th gray level; P k represents the kth histogram value, that is, the number of pixels in the image at the kth grayscale level; the denominator is the sum of the histogram values of all 16 grayscale levels, which is used for normalization, k=0,1,...,15, and the calculation formula is as follows: ; Where P k is the number of pixels at the kth gray level; k represents a natural number, and i is a natural number; S502: The feature information of the 19 groups of images in the all-focus image is used as the input layer, and the output layer is the adaptive parameters gain1 and gain2, where gain1 is used to adjust the brighter part of the image, and gain2 is used to adjust the darker part; the output value of the output layer is given by the following formula: ; Among them, w is the weight, x represents the input feature value, namely L0, L1, ..., L 18 , ℎ k represents the sum of the output weights of the k neurons in the output layer, x jk represents the feature value associated with the jth feature in the input layer and the kth neuron in the output layer; w jk represents the weight connecting the jth input feature of the input layer and the kth neuron of the output layer; j is an index variable used to traverse all input features in the input layer, ranging from 0 to M; M is the number of feature values; k is an index variable used to traverse all neurons in the output layer; h k is the weighted input sum of the kth neuron in the output layer; a k is the activation value of the kth neuron in the output layer; The g(·) is an activation function, and the rectified linear activation unit (ReLU) is used as the activation function. The formula is as follows: ; The loss function E uses the error square sum function, and the formula is as follows: ; Where N is the total number of samples; during training, it represents the number of samples used to calculate the loss function; k is an index variable used to traverse all samples, ranging from 0 to N-1; y k represents the output value of the kth sample predicted by the model, and t is the true value of the kth sample; S503: Using 19 sets of data representing image feature information as input and gain1 and gain2 as output, the light field image is trained through an MLP network to establish a relationship between the input and the output.
