Water-air visible light communication image processing method based on CMOS camera

Abstract

The application discloses a water-air visible light communication image processing method based on a CMOS camera, and relates to the technical field of optical camera communication, and comprises the following steps: S1, reading n initial received images, carrying out gray scale processing on the images, and calculating average pixel values of R, G and B three channels and an average gray scale image of the gray scale processed images; S2, finding the column and the row where the center of the lamp is located in the image according to the sum of each row and each column of all channel average gray scale images; S3, estimating maximum reference values of R, G and B channels according to the average pixel values of the three channels; S4, estimating gamma indexes according to the average pixel values of the three channels; and S5, restoring other images according to the obtained average pixel values, maximum reference values and gamma indexes of the three channels. The application can improve the communication rate, can well cope with the color image distortion problem caused by complex environment, and can relieve the interference caused by the water wave for the decoding algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical camera communication, and particularly relates to a water-air visible light communication image processing method based on a CMOS camera. BACKGROUND

[0002] In recent years, visible light communication (VLC) as a short-range communication technology has attracted widespread attention due to its high bandwidth and low latency characteristics; it is even considered as one of the next wireless communication technologies (6G). VLC technology provides opportunities for indoor positioning, mobile payment and navigation applications. In most studies, VLC requires a photodiode as a dedicated receiver, which delays its commercialization. In order to weaken the dependence on dedicated devices, optical camera communication (OCC) as a special type of VLC emerges as the times require, which mainly uses charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) cameras as receivers. Commercial CMOS-based optical cameras are currently widely used in smartphones, which provides favorable conditions for the development of OCC technology.

[0003] To achieve faster communication rate, previous researchers improved the black and white LED light at the transmitting end to color RGB-LED light, which makes the communication framework naturally have the potential to achieve faster rate at the modulation end. The code shift keying (CSK) modulation scheme recommended by IEEE standard [IEEE Standard 802.15.7-2011; IEEE Standard for Local and Metropolitan Area Networks - Part 15.7: Short-Range Wireless Optical Communication Using Visible Light. IEEE: New York, NY, USA, 2011; pp. 1-309.] is currently widely used in optical camera communication based on RGB-LED light. In previous studies, there are several similar schemes that are also based on the CSK modulation scheme to build an optical camera communication system. In [Hu, P.; Pathak, P. H.; Feng, X.; Fu, H.; Mohapatra, P. Colorbars: Increasing data rate of led-to-camera communication using color shift keying. In Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies, Heidelberg, Germany, 1-4 December 2015; pp. 1-13.], the intensity of each channel of the RGB channel of the transmitting end LED light is modulated to mix the colors of the three channels to generate 8 colors, and the corresponding data of the 8 colors is decoded at the decoding end to recover the binary data of each channel of the RGB channel. However, this scheme has a high requirement for the consistency of the data color, and the color of each symbol needs to be accurately adjusted, which makes the scheme not ideal in dealing with the color deviation and wave problem of the received image in the water-air optical communication environment.

[0004] In [47-kbit / s RGB-LED-based optical camera communication based on 2D-CNN and XOR-based data loss compensation.], an optical camera communication decoding framework using a two-dimensional convolutional neural network is proposed based on an RGB-LED and CSK modulation scheme. At the same time, the scheme designs a special data packet construction form, divides a single data packet into several parts, and uses the proposed XOR mechanism to recover the data. There is also a patent named "a rolling shutter camera image processing method based on visible light communication" (publication number: CN115719359A), which uses black and white LED lights to modulate the signal of the transmitting end, and extracts individual bright and dark symbol stripes by detecting the boundary, and then fits the symbol gray value by threshold method for decoding; but this scheme uses black and white LED lights as the transmitting end, and under the same symbol width condition at the receiving end, a single symbol only contains one bit of data, which means that the communication rate it can achieve is much lower than the optical camera communication framework using RGB-LED lights as the transmitting end.

[0005] In summary, the above-mentioned related schemes do not consider the image distortion challenges encountered when the system is applied in complex environment scenarios, and the effect is not ideal when dealing with complex environment application scenarios such as water-air. SUMMARY

[0006] The purpose of the present application is to provide a water-air visible light communication image processing method based on CMOS camera, which not only can improve the communication rate, but also can well deal with the color image distortion problem brought by complex environment, and alleviate the interference brought by water waves for decoding algorithm.

[0007] To achieve the above-mentioned purpose, the present application provides the following scheme:

[0008] A water-air visible light communication image processing method based on a CMOS camera, comprising the following steps:

[0009] S1, read the initial received n images, perform grayscale processing on the images, and calculate the average pixel values of R, G and B channels and the average grayscale image of the grayscale processed images;

[0010] S2, find the column and row where the center of the lamp is located in the image according to the sum of each row and column of all channel average grayscale images;

[0011] S3, estimate the maximum reference value of R, G and B channels according to the average pixel values of the three channels;

[0012] S4, estimate the gamma index according to the average pixel values of the three channels;

[0013] S5, restoring other images according to the average pixel value of the three channels, the maximum reference value and the gamma index.

[0014] Further, the S1 specifically comprises the following steps:

[0015] S1.1, reading the initial received n images, splitting each image into R, G, B three channels, and accumulating the pixel value of each channel to obtain the accumulated pixel of each channel;

[0016] S1.2, according to the accumulated pixel value of each channel, dividing by the number of pictures n to obtain the average pixel value of the three channels;

[0017] S1.3, converting all the n pictures read in S1.1 into gray scale images, and accumulating the pixel value to obtain the accumulated pixel value;

[0018] S1.4, according to the accumulated pixel value in S1.3, dividing by the number of pictures n to obtain the average gray scale image.

[0019] Further, the S2 specifically comprises the following steps:

[0020] S2.1, summing each column of the average gray scale image to obtain a vector X, which is used to represent the sum of the pixel value of each column of the average gray scale image, determining the column with the maximum sum of pixel value to obtain the column where the center of the lamp is located in the image;

[0021] S2.2, summing each row of the average gray scale image to obtain a vector Y, which is used to represent the sum of the pixel value of each row of the average gray scale image, determining the row with the maximum sum of pixel value to obtain the row where the center of the lamp is located in the image.

[0022] Further, the S3 specifically comprises the following steps:

[0023] S3.1, for the read n images, taking the column where the center of the lamp is located, splitting into R, G, B three channels, and dividing the pixel value of each channel by the average pixel value of the corresponding channel in S1.2 to normalize the images;

[0024] S3.2, for the three channels, the following operations are performed respectively: all columns are spliced into a one-dimensional vector, each pixel of the one-dimensional vector is traversed, and the maximum element in the 15 pixels around each pixel is found by comparing the 15 pixels; The maximum element and its corresponding index are recorded;

[0025] S3.3, arranging the elements with the maximum pixel value obtained in step S3.2 in descending order, finding the maximum pixel value among them, and taking 20% of the maximum pixel value as the maximum reference value of the corresponding channel.

[0026] Further, the S4 specifically comprises the following steps:

[0027] S4.1, taking the one-dimensional vector obtained in S3.2, calculating the mean value, and dividing it by the maximum reference value of the corresponding channel obtained in S3.3 to obtain the normalized average pixel value;

[0028] S4.2, calculating the three channels as follows: gamma=log(0.5) / log(mean_ref), wherein mean_ref is the normalized average pixel value in S4.1.

[0029] Further, the S5 specifically comprises the following steps:

[0030] S5.1, taking the column where the center of the lamp is located in the image to be repaired, and splitting it into R, G, and B channels, and dividing the pixel value of each channel by the corresponding channel average pixel value;

[0031] S5.2, dividing all pixel values in the column by the maximum reference value of the corresponding channel;

[0032] S5.3, performing exponential operation on the value obtained in S5.2 and the gamma obtained in S4, i.e. new pixel value=old pixel value^gamma index, to obtain the corrected column;

[0033] S5.4, copying the corrected column m times in the column direction, m being the number of columns of the original image, and combining the images of the three channels to obtain the corrected new image.

[0034] Further, the method is to modulate the signal intensity of the RGB channel by using RGB-LED as the transmitting end, and to obtain the initial received n images by using the CMOS camera to collect.

[0035] According to the specific embodiments provided by the application, the following technical effects are disclosed: the water-air visible light communication image processing method based on the CMOS camera provided by the application uses RGB color LED lamp, compared with the past scheme of using black and white LED lamp to build optical camera communication framework, the application makes the optical camera communication framework can realize faster communication rate.

[0036] In optical camera communication applications, the image processing method of this invention is a more robust image processing method, capable of handling complex environments that previous solutions could not handle, effectively addressing color shift issues in water-air environments, and mitigating interference from waves in water on the decoding algorithm.

[0037] In optical camera communication applications in water-air environments, this invention does not require the deployment of additional receiving equipment. It only requires software adjustments to effectively address color image distortion caused by complex environments without increasing deployment costs. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram illustrating the principle of the water-air visible light communication image processing method based on a CMOS camera according to the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] The purpose of this invention is to provide a water-air visible light communication image processing method based on a CMOS camera, which can not only improve the communication rate, but also effectively deal with the color image distortion problem caused by complex environments and alleviate the interference of waves in the water on the decoding algorithm.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, in this invention, the transmitting end utilizes an FPGA and uses CSK for modulation, where one symbol contains the color of the RGB channels, totaling 3 bits of data. A CMOS-based camera is used to receive the light signal. Signal modulation is performed using RGB-LEDs, theoretically achieving a data rate three times that of a black-and-white LED-based frame. This invention can address the image distortion problem of optical camera communication frames in complex environments such as water and air.

[0044] In the embodiment, the transmitting end uses FPGA to modulate signals, drives RGB-LED to send light signals by using IO port of FPGA, and the light signals are transmitted through water and air medium; the receiving end uses a camera based on CMOS to shoot videos to capture light signals, and then sends the captured color stripe images as input to the image processing algorithm of the application, and high-quality images are obtained after processing.

[0045] The water-air visible light communication image processing method based on a CMOS camera provided by the embodiment of the application comprises the following steps:

[0046] S1, reading n initial received images, performing grayscale processing on the images, and calculating the average pixel values of R, G and B channels and the average grayscale image of the grayscale processed images;

[0047] S2, finding the column and row where the center of the lamp is located in the image according to the sum of each row and column of the average grayscale image of all channels;

[0048] S3, estimating the maximum reference value of R, G and B channels according to the average pixel values of the three channels;

[0049] S4, estimating the gamma index according to the average pixel values of the three channels; the gamma index is used to measure the degree of nonlinearity of the color captured by the camera and the real color of the object, and Gamma correction can better restore the real color of the object.

[0050] S5, restoring other images according to the obtained average pixel values, maximum reference value and gamma index of the three channels.

[0051] In a further embodiment, S1 specifically comprises the following steps:

[0052] S1.1, reading n initial received images, and splitting each image into R, G and B channels, and accumulating the pixel values of each channel to obtain the accumulated pixels of each channel;

[0053] S1.2, dividing the obtained accumulated pixel values of each channel by the number n of pictures to obtain the average pixel values of the three channels;

[0054] S1.3, converting all the n pictures read in S1.1 into grayscale images, and accumulating the pixel values to obtain the accumulated pixel values;

[0055] S1.4, dividing the accumulated pixel values in S1.3 by the number n of pictures to obtain the average grayscale image.

[0056] In a further embodiment, S2 specifically comprises the following steps:

[0057] S2.1, summing each column of the average gray image to obtain a vector X, which is used to represent the sum of pixel values of each column of the average gray image, determining the column with the largest sum of pixel values to obtain the column in which the center of the lamp is located in the image;

[0058] S2.2, summing each row of the average gray image to obtain a vector Y, which is used to represent the sum of pixel values of each row of the average gray image, determining the row with the largest sum of pixel values to obtain the row in which the center of the lamp is located in the image.

[0059] In further embodiments, the S3 specifically comprises the following steps:

[0060] S3.1, for each of the n images read, taking a column in which the center of the lamp is located, splitting into R, G, B three channels, dividing the pixel value of each channel by the average pixel value of the corresponding channel, and performing normalization processing on the images;

[0061] S3.2, for each of the three channels, performing the following operations: splicing all columns into a one-dimensional vector, traversing each pixel of the one-dimensional vector, comparing 15 pixels around each pixel, and finding the pixel with the maximum value in the 15 pixels; recording the pixel with the maximum value and its corresponding index;

[0062] S3.3, arranging the pixel value maximum elements obtained in step S3.2 in descending order, finding the largest one of the pixel values, and taking 20% of the largest one of the pixel values as the maximum reference value of the corresponding channel.

[0063] In further embodiments, the S4 specifically comprises the following steps:

[0064] S4.1, taking the one-dimensional vector spliced in S3.2, calculating the mean value, and dividing the mean value by the maximum reference value of the corresponding channel obtained in S3.3 to obtain the normalized average pixel value;

[0065] S4.2, for each of the three channels, calculating the following: gamma = log(0.5) / log(mean_ref), wherein mean_ref is the normalized average pixel value in S4.1.

[0066] In further embodiments, the S5 specifically comprises the following steps:

[0067] S5.1, taking a column in which the center of the lamp is located in the image to be repaired, splitting into R, G, B three channels, and dividing the pixel value of each channel by the average pixel value of the corresponding channel;

[0068] S5.2, dividing all pixel values of the column by the maximum reference value of the corresponding channel;

[0069] S5.3, the value obtained in S5.2 is subjected to exponential operation with the gamma obtained in S4, to represent the new pixel value, to represent the old pixel value, and gamma represents the gamma index, then a = b γ , so as to obtain the corrected column;

[0070] S5.4, the corrected column is copied m times in the column direction, m is the column number of the original image, and the images of the three channels are combined to obtain the corrected new image.

[0071] In a further embodiment, the method is to modulate the signal intensity of the RGB channel by taking RGB-LED as the emission end, and to obtain the initial received n images by CMOS camera acquisition.

[0072] In summary, the water-air visible light communication image processing method based on the CMOS camera provided by the application uses RGB color LED lamps, compared with the past scheme of using black and white LED lamps to build an optical camera communication framework, the application enables the optical camera communication framework to achieve faster communication rate.

[0073] In the remaining technical features in this embodiment, those skilled in the art can flexibly select them to meet different specific actual needs according to actual conditions. However, it is obvious to those skilled in the art that these specific details are not necessarily used to implement the application.

[0074] Changes and variations made by those skilled in the art without departing from the spirit and scope of the application shall be within the protection scope of the appended claims of the application. In the above description, a large number of specific details are set forth in order to provide a thorough understanding of the application. However, it is obvious to those skilled in the art that the application does not have to be implemented with these specific details. In other examples, in order not to obscure the application, well-known technologies such as specific construction details, operating conditions and other technical conditions are not specifically described.

[0075] The principles and implementation modes of the application are described by applying specific examples in this paper, and the above description of the examples is only used to help understand the method of the application and its core idea; at the same time, for those skilled in the art, according to the idea of the application, there will be changes in specific implementation modes and application scope. In summary, the content of this specification should not be understood as a limitation of the application.

Claims

1. A method for image processing of water-air visible light communication based on CMOS camera, characterized in that, The method comprises the following steps: S1, reading the initial received n images, the image is grayed, the average pixel value of R, G, B three channels and the average gray image of the grayed image are calculated; S2, according to the sum of each row and column of the average gray image of all channels, find the column and row where the center of the lamp is located in the image; S3, estimate the maximum reference value of R, G, B channel according to the average pixel value of three channels; S4, estimate the gamma index according to the average pixel value of three channels; S5, restore other images according to the average pixel value, maximum reference value and gamma index of three channels obtained; S5 specifically comprises the following steps: S5.1, take the center of the lamp in a column of the image to be repaired, split into R, G, B three channels, and divide the pixel value of each channel by the corresponding channel average pixel value; S5.2, divide all pixel values in the column by the maximum reference value of the corresponding channel; S5.3, the value obtained by S5.2 is subjected to exponential operation with the obtained gamma, that is, new pixel value = old pixel value^gamma index, to obtain the corrected column; S5.4, copy the corrected column in the column direction m times, m is the number of columns of the original image, and combine the images of the three channels to obtain the corrected new image.

2. The CMOS camera based water-air visible light communication image processing method according to claim 1, wherein, S1 specifically comprises the following steps: S1.1, read the initial received n images, split each image into R, G, B three channels, and accumulate the pixel value of each channel to obtain the accumulated pixel of each channel; S1.2, divide the accumulated pixel value of each channel by the number of pictures n to obtain the average pixel value of three channels; S1.3, convert all the n pictures read in S1.1 into gray images, and accumulate the pixel value to obtain the accumulated pixel value; S1.4, divide the accumulated pixel value in S1.3 by the number of pictures n to obtain the average gray image. 3.The water-air visible light communication image processing method based on CMOS camera according to claim 1, wherein, S2 specifically comprises the following steps: S2.1, sum each column of the average gray image to obtain a vector X, which is used to represent the sum of pixel values of each column of the average gray image, and determine the column with the maximum sum of pixel values to obtain the column where the center of the lamp is located in the image; S2.2, sum each row of the average gray image to obtain a vector Y, which is used to represent the sum of pixel values of each row; find the maximum value in the average gray image to obtain the row where the center of the lamp is located in the image. 4.The water-air visible light communication image processing method based on a CMOS camera according to claim 2, wherein, S3 specifically comprises the following steps: S3.1, take the center of the lamp in a column of each of the n images read, split into R, G, B three channels, and divide the pixel value of each channel by the average pixel value of the corresponding channel in S1.2 to normalize the images; S3.2, perform the following operations on the three channels respectively: combine all columns into a one-dimensional vector, traverse each pixel of the one-dimensional vector, compare the 15 pixels around each pixel, and find the element with the maximum pixel value in the 15 pixels; record the element with the maximum pixel value and its corresponding index; S3.3, arranging the elements with the maximum pixel value obtained in step S3.2 in descending order, finding the maximum pixel value among them, and taking 20% of the maximum pixel value as the maximum reference value of the corresponding channel.

5. The CMOS camera-based water-air visible light communication image processing method of claim 4, wherein, The S4 specifically comprises the following steps: S4.1, taking the one-dimensional vector obtained in S3.2, calculating the mean value, and dividing by the maximum reference value of the corresponding channel obtained in S3.3 to obtain the normalized average pixel value; S4.2, calculating the three channels as follows: gamma = log(0.5) / log(mean_ref), wherein mean_ref is the normalized average pixel value in S4.

1. 6.The water-air visible light communication image processing method based on CMOS camera according to claim 1, wherein, The method is to modulate the signal intensity of the RGB channel by taking RGB-LED as the transmitting end, and to obtain the initial received n images by CMOS camera.

Images