Optical camera communication-oriented GAN-based stripe image processing method

Abstract

The invention belongs to the field of visible light communication, and particularly discloses an optical camera communication-oriented GAN-based stripe image processing method, which comprises the following steps of: synthesizing a stripe image data set; migrating samples of the fringe image data set to a target domain; training a generative adversarial network (GAN), and obtaining an image reconstruction model and a stripe extraction model; the image reconstruction model is constructed based on a U-NET network, and an attention module SimAM is adopted; the stripe extraction model is constructed by cascading an input layer, a plurality of middle layers and an output layer, the middle layers are constructed by cascading a multi-scale module and a pooling layer, the multi-scale module comprises a plurality of CondConv networks, and residual connection is configured between the input and the output of the multi-scale module. Through the lightweight design of the image reconstruction model and the lightweight design of the stripe extraction model, it can be ensured that the model obtained for generative adversarial network training can be suitable for mobile terminal deployment.

Description

Technical Field

[0001] The present application belongs to the field of visible light communication, and more specifically, relates to a GAN-based fringe image processing method for optical camera communication. Background Art

[0002] Visible light communication is a technology that uses visible light bands for communication. It uses the characteristics of visible light for data transmission and transmits information by modulating the brightness or color of light.

[0003] Visible light communication (VLC) uses a light source, such as an LED or laser, to transmit light signals. The receiving end uses a camera to receive and decode the light signals, thereby achieving data transmission. In VLC scenarios, when a light source emits a light signal, it flickers at a certain frequency, imperceptible to the naked eye. When the light signal is transmitted to the receiving end, the receiving end uses the light sensor in the camera to capture the image at a certain frame rate. Due to the difference between the flickering frequency of the LED and the frame rate of the camera, the captured image appears striped, with alternating light and dark stripes appearing in the image. The spacing and brightness variations of these stripes correspond to the flickering frequency and brightness of the LED. Therefore, by extracting the stripes in the image, the stripe sequence can be obtained, and the light signal can be decoded.

[0004] For stripe images, there are generally two tasks: stripe image reconstruction and stripe extraction. The stripe image reconstruction task is used to remove stripes from the image and restore the background image, while the stripe extraction task is used to extract the stripes from the image. Image reconstruction and stripe extraction are mutually exclusive tasks. Excessive black stripes in an image facilitate stripe extraction but hinder image reconstruction. Conversely, fewer black stripes facilitate image reconstruction but hinder stripe extraction.

[0005] A Generative Adversarial Network (GAN) consists of two neural networks: a generator and a discriminator. The generator is responsible for generating data samples, while the discriminator is responsible for determining whether the generated data samples are real or fake. GANs work by training the generator and the discriminator in an adversarial relationship, enabling the generator to produce sufficiently realistic data samples while the discriminator can accurately distinguish between real and generated data. During training, the generator continuously generates data samples, while the discriminator continuously learns to better distinguish between real and generated data, until the generator's data samples become indistinguishable to the discriminator.

[0006] In related technologies, considering that image reconstruction and stripe extraction are a pair of mutually adversarial (contradictory) tasks, the use of stripe image datasets and GANs to generate adversarial between the two contradictory tasks can effectively balance the image reconstruction accuracy and stripe extraction accuracy, and obtain image reconstruction models and stripe extraction models suitable for visible light communication scenarios.

[0007] However, the image reconstruction model and fringe extraction model used in related technologies are relatively complex and difficult to apply to mobile deployment. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the purpose of this application is to lightweight the image reconstruction model and the stripe extraction model so that the model is suitable for mobile terminal deployment.

[0009] To achieve the above objectives, in a first aspect, the present application provides a GAN-based fringe image processing method for optical camera communication, the method comprising: Synthetic stripe image dataset; Based on real stripe images and stripe image datasets, the samples of the stripe image dataset are transferred to the target domain through CycleGAN. The target domain is the domain where the real stripe images are located. Based on the stripe image dataset after migration to the target domain, a generative adversarial network (GAN) is trained to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is used to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution (CondConv) network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolution CondConv networks. Different conditional convolution CondConv networks in the multi-scale module use different convolution kernels. Residual connections are configured between the input and output of the multi-scale module, and the output layer is constructed based on an activation function.

[0010] It can be understood that the lightweight design of the above-mentioned image reconstruction model and the lightweight design of the stripe extraction model can ensure that the model obtained by training the generative adversarial network is suitable for mobile deployment.

[0011] Optionally, after acquiring the image reconstruction model and the fringe extraction model, the method further includes: acquiring a fringe image of the surrounding environment through a camera, wherein a light source in the surrounding environment flashes under the control of a visible light communication device; The fringe image is input into an image reconstruction model to obtain a reconstructed image output by the image reconstruction model; and / or the fringe image is input into a fringe extraction model to obtain a fringe sequence output by the fringe extraction model.

[0012] It is understandable that the extracted fringe sequence can be conveniently used to decode information in visible light communication, and the image reconstruction result can be used for tasks such as environmental perception and target recognition and detection.

[0013] In a possible implementation, the method further includes: in the process of constructing the U-NET network, constructing an upsampling layer (or upsampling module) based on the conditional convolution CondConv network and the pixel reorganization PixelShuffle module.

[0014] It is understandable that, unlike deconvolution, PixelShuffle overcomes the checkerboard problem of deconvolution. It can be said that PixelShuffle is the best performing upsampling technology among the current general upsampling technologies. Conv+PixelShuffle is also a common combination method in the field of image super-resolution. Feature maps often need to go through Conv to expand the number of channels before PixelShuffle. However, traditional Conv+PixelShuffle will increase the complexity. This application uses the combination of CondConv+PixelShuffle as the upsampling layer of the model, and uses CondConv to replace Conv. CondConv can significantly reduce the complexity of the model while ensuring performance. Therefore, in order to make the generator model (image reconstruction model) more lightweight, this application adopts CondConv+PixelShuffle to achieve a deconvolution effect. Experiments have shown that this implementation method can effectively reduce the complexity of the generator model while ensuring that performance will not be reduced. It provides a guarantee for mobile deployment.

[0015] In a possible implementation, the multiple conditional convolution CondConv networks are specifically three conditional convolution CondConv networks, and the convolution kernels used by the three conditional convolution CondConv networks are respectively a 3x3 convolution kernel, a 5x5 convolution kernel, and a 7x7 convolution kernel.

[0016] In one possible implementation, the pooling layer of the stripe extraction model implements pooling based on the conditional convolution CondConv network.

[0017] It is understandable that in the pooling stage, a (2,1) convolution kernel and a CondConv with a step size of (2,1) are used to implement the pooling operation. The traditional AvgPool or MaxPool is not used here because it is found through experiments that the convolution-based pooling operation has more flexibility, can better aggregate features, and reduce feature loss. In addition, this application uses ConCat for residual operations, which means that the number of channel features increases after each layer is calculated. Therefore, when a convolution is used as pooling, the number of output channels can be controlled for aggregation to further aggregate the features after the residual and reduce the number of channels.

[0018] In a possible implementation, the stripe image dataset includes a stripe image sample X, a stripe sequence label S corresponding to the stripe image sample X, and a stripe-free image label Y; Training the generative adversarial network, including: continuously calculating loss values and optimizing model parameters based on pixel loss function, adversarial loss function, multi-scale SSIM loss function, and discriminator loss function until the training stop condition is met; A pixel loss function is used to determine the pixel loss based on the streak-free image label Y and the reconstruction result, where the reconstruction result is determined based on the output of the image reconstruction model by inputting the streak image sample X into the image reconstruction model; an adversarial loss function for determining an adversarial loss based on a first stripe extraction result, wherein the first stripe extraction result is input into a stripe extraction model and determined based on an output of the stripe extraction model; A multi-scale SSIM loss function is used to determine the multi-scale SSIM loss based on the stripe-free image label Y and the reconstruction result; The discriminator loss function is used to determine a first discriminant loss based on the second stripe extraction result; determine a second discriminant loss based on the third stripe extraction result and the stripe sequence label S; determine a comprehensive discriminant loss by weighted summation based on the first discriminant loss and the second discriminant loss; the second stripe extraction result is determined by inputting the stripe-free image label Y into the stripe extraction model and based on the output of the stripe extraction model; the third stripe extraction result is determined by inputting the stripe image sample X into the stripe extraction model and based on the output of the stripe extraction model.

[0019] The stripe sequence label, also called the true stripe sequence, is used to characterize the stripe sequence contained in the stripe image sample.

[0020] The stripe-free image label, which can also be called the true stripe-free image, represents the environmental background obtained after removing the stripe sequence from the stripe image sample.

[0021] Optimize model parameters, including adjusting model parameters through backpropagation based on the loss values corresponding to each loss function.

[0022] The above-mentioned training stop condition can be that the loss value is less than a preset loss threshold, or the maximum number of iterative training rounds is reached.

[0023] In a possible implementation, the stripe image dataset includes a stripe image sample X, a stripe sequence label S corresponding to the stripe image sample X, and a stripe-free image label Y; The above synthetic stripe image dataset includes: Based on the Hypersim dataset, stripes are added to each original image sample in the Hypersim dataset to obtain stripe image samples. The original image samples are used as stripe-free image labels, and the added stripes are used as stripe sequence labels.

[0024] It can be understood that by adding stripes based on the Hypersim dataset, it is possible to efficiently generate stripe image samples that are close to real-world optical camera communications, effectively improving the efficiency of constructing the stripe image dataset while ensuring sample quality.

[0025] In one possible implementation, the adding of stripes to each original image sample in the Hypersim dataset includes adding stripes using the following formula: ; in, represents a fringe image sample, Represents the depth data corresponding to the original image sample, Represents a random light source emission point, Indicates the added stripes, Represents the diffuse light intensity of the original image sample that illuminates the surface of the object. Represents the diffuse reflectance of the object surface in the original image sample, represents the residual term corresponding to the original image sample, Represents the adjustment factor of the diffuse lighting component, Represents the residual coefficient.

[0026] In a second aspect, the present application provides a GAN-based fringe image processing device for optical camera communication, comprising: Dataset acquisition module, used to synthesize stripe image dataset; The data migration module is used to migrate the samples of the stripe image dataset to the target domain based on the real stripe image and the stripe image dataset through CycleGAN. The target domain is the domain where the real stripe image is located; A training module is used to train a generative adversarial network (GAN) based on the stripe image dataset after migration to the target domain, and obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is used to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution CondConv network. The intermediate layer is constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolution CondConv networks. Different conditional convolution CondConv networks in the multi-scale module use different convolution kernels. A residual connection is configured between the input and output of the multi-scale module, and the output layer is constructed based on the activation function.

[0027] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0030] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies: (1) In order to make the generator model (image reconstruction model) lightweight, this application uses the attention module SimAM for one or more residual connections of the U-NET network, which can effectively reduce the number of network layers of the U-NET network, thereby reducing the complexity of the generator model and providing a guarantee for mobile terminal deployment.

[0031] (2) To make the discriminator (stripe extraction model) lightweight, this application combines pyramid convolution with CondConv. Compared with the existing Transformer architecture (such as the Vit framework), it can effectively reduce the complexity of the discriminator model while ensuring that the performance is not degraded. This provides guarantees for mobile terminal deployment and high-speed communication decoding.

[0032] (3) This application uses CondConv+PixelShuffle to achieve a deconvolution effect. Experiments have shown that this implementation can effectively reduce the complexity of the generator model while ensuring that the performance is not degraded. This provides a guarantee for mobile deployment.

[0033] (4) By adding stripes based on the Hypersim dataset, we can efficiently generate stripe image samples that are close to the real ones for optical camera communication, and effectively improve the efficiency of constructing the stripe image dataset while ensuring the quality of the samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 1 is a flow chart of a GAN-based fringe image processing method for optical camera communication provided in an embodiment of the present application; Figure 2 is a schematic diagram of adding stripes to image samples in a Hypersim dataset to obtain stripe image samples, provided by an embodiment of the present application; Figure 3 This is an overview diagram of the GANOCCAS network structure provided by the embodiment of the present application; Figure 4 This is a structural diagram of a generator provided in an embodiment of the present application; Figure 5 is a structural diagram of a discriminator provided in an embodiment of the present application; Figure 6 1 is a schematic structural diagram of a GAN-based fringe image processing device for optical camera communication provided in an embodiment of the present application; Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to facilitate a clearer understanding of the various embodiments of the present application, some relevant background knowledge is first introduced as follows.

[0036] (1) Description of OCC (reflective optical camera communication); CMOS sensors use a rolling shutter mechanism, which uses a line-by-line exposure method. This rolling shutter effect can be exploited to increase the data rate of visible light imaging communication. At each moment, the CMOS image sensor exposes only one row. After all rows are exposed, the data from all rows captured at different times is combined to form an image. This line-by-line exposure method results in the appearance of alternating light and dark stripes (also known as a light stripe code or stripe sequence) when capturing rapidly flashing LED lamps. The faster the lamp flashes, the greater the number of stripes. By decoding these alternating light and dark stripes in the image, several bits of information can be transmitted, enabling visible light imaging communication. The lamp flashing frequency must be less than the CMOS line scanning frequency, otherwise data loss will occur. When the LED flashing frequency exceeds the line scanning frequency, the CMOS sensor only captures the average light intensity emitted by the LED and cannot detect the light and dark stripes within the LED pixel area. The exposure time and ISO sensitivity of the CMOS image sensor affect the acquisition of the LED-ID light stripe code.

[0037] (2) Signal modulation based on PWM; Pulse Width Modulation (PWM), abbreviated as PWM, is a very effective technology that uses the digital output of a microprocessor to control analog circuits. It is widely used in many fields from measurement, communication to power control and conversion.

[0038] PWM frequency refers to the number of times a signal changes from high to low and back to high in one second (one cycle). This means the number of PWM cycles in one second. This unit is expressed in Hz. This is expressed in units of Hz: 50Hz or 100Hz. If the frequency is 50Hz, meaning one cycle is 20ms, there are 50 PWM cycles in one second.

[0039] PWM modulates LED lights. The PWM frequency will affect the LED flickering frequency. Generally speaking, frequencies above 100Hz are not detectable by the human eye.

[0040] Duty cycle: It is the ratio of the high level time to the total cycle time within a pulse cycle.

[0041] Four different duty cycles can be used to represent different signals in one cycle.

[0042] (3) Communication rate calculation; PWM modulation can be used to transmit information using four different duty cycles. The communication rate is related to the PWM frequency. Assume the PWM frequency is Phz, which means it changes P times in 1 second.

[0043] Assume that the PWM duty cycle is used to represent different symbols (for example, 4 duty cycles represent 2-bit binary data): Symbol rate: One symbol is transmitted per PWM cycle, so the symbol rate is P symbols / second (Baud).

[0044] Modulation order M: 4 duty cycles. The calculation formula for the communication rate (bit rate) is: ; ; Therefore, using four duty cycles for signal modulation, the communication rate is 2P bps. P is the PWM frequency. For 256 bps, P = 128 Hz. For 128 bps, P = 64 Hz. For 64 bps, P = 32 Hz.

[0045] In OCC communication, the image forms stripes of different widths due to different duty cycles within the PWM cycle, thereby representing different information. For an image with a width of W, the number of video frames captured per second is S, which means that W frames are generated per second. S pixel width, and assuming that the PWM frequency at this time is P, that is, these P changes are reflected in W S pixel, so we can know the frequencies P and W The relationship of S is as follows, one cycle of PWM results in a pixel width of c_pixel: c_pixel P=W S; When c_pixel=4,W=256,S=2,P=128hz, the communication rate is 256bps; When c_pixel=4,W=256,S=30, P=128hz, the communication rate is 3840bps.

[0046] In GANOCCAS, S=2, which means that 2 images are taken per second and the image width is 256. 256, various rates and corresponding parameters are calculated as shown in Table 1.

[0047] Table 1 Rate and corresponding parameters

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] Throughout the specification and claims of this application, the terms "first" and "second" are used to distinguish between different objects, rather than to describe a specific order of objects. For example, a first stripe extraction result and a second stripe extraction result are used to distinguish between different stripe extraction results, rather than to describe a specific order of stripe extraction results.

[0050] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0051] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0052] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0053] Figure 1 is a flow chart of a GAN-based fringe image processing method for optical camera communication provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101, S102 and S103.

[0054] Step S101, synthesizing a stripe image dataset; Step S102: Based on a real stripe image (which can be obtained by shooting in a scenario facing optical camera communication) and a stripe image dataset, samples of the stripe image dataset are migrated to a target domain using CycleGAN, where the target domain is the domain where the real stripe image is located. Step S103: training a generative adversarial network (GAN) based on the stripe image dataset after migration to the target domain to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is used to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a CondConv convolutional network. The intermediate layer is constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple CondConv convolutional networks. Different CondConv convolutional networks in the multi-scale module use different convolution kernels. Residual connections are configured between the input and output of the multi-scale module, and the output layer is constructed based on an activation function.

[0055] Specifically, the samples in the synthetic stripe image dataset differ in style from real stripe images, necessitating style conversion (style transfer) of these samples. CycleGAN (Cycle-Consistent Generative Adversarial Networks) is a deep learning model designed to achieve image-to-image translation without the need for paired training data. In the absence of paired training data, CycleGAN can be trained using a source domain image set (a dataset constructed from real stripe images) and a target domain image set (i.e., a synthetic stripe image dataset). CycleGAN learns the complex mapping relationship between the source domain (the domain where the synthetic stripe image dataset samples reside) and the target domain (the domain where the real stripe images reside). Once CycleGAN has learned this mapping relationship, the generator of CycleGAN can transfer samples from the stripe image dataset to the target domain. After transfer to the target domain, the samples in the stripe image dataset are closer to real stripe images, ensuring the training effectiveness of image reconstruction and stripe extraction models.

[0056] SimAM (Simple Attention Module) is a simple, parameter-free attention module for convolutional neural networks (CNNs). SimAM can further improve the expressiveness of the model with almost no increase in computational complexity. This improved expressiveness can reduce the number of upsampling and downsampling modules in the U-NET network. Experiments have shown that the number of upsampling and downsampling modules can be reduced to less than four without compromising performance.

[0057] Therefore, in order to make the generator model (image reconstruction model) lightweight, this application uses the attention module SimAM for one or more residual connections of the U-NET network, which can effectively reduce the number of network layers of the U-NET network, thereby reducing the complexity of the generator model and providing a guarantee for mobile deployment.

[0058] The discriminator (stripe extraction model) is used for stripe extraction. This model is based on the CondConv design and uses the multi-scale pyramid convolution architecture to extract multi-scale features. It also combines residual connections to further retain more information. During the continuous feature extraction process, the network continuously compresses and aggregates height information, thereby continuously reducing computational complexity and ultimately outputting a stripe sequence.

[0059] Therefore, to make the discriminator (stripe extraction model) lightweight, this application combines pyramid convolution with CondConv. Compared to existing Transformer architectures (such as the Vit framework), this effectively reduces the complexity of the discriminator model while ensuring no performance degradation. This ensures mobile deployment and high-speed communication decoding.

[0060] It can be understood that the lightweight design of the above-mentioned image reconstruction model and the lightweight design of the stripe extraction model can ensure that the model obtained by training the generative adversarial network is suitable for mobile deployment.

[0061] The following is an illustrative example of the GAN-based fringe image processing method for optical camera communication provided by this application.

[0062] Example 1: The following is an example of data set synthesis.

[0063] A dataset for training optical communication is synthesized based on the Hypersim dataset. Hypersim is a dataset of real 3D scenes that can provide high-resolution textures and dynamic lighting for overall indoor scene understanding.

[0064] Hypersim provides a simple lighting model as follows: ; Color is the final calculated color value of the object surface. Color is composed of the optical components finally received by a camera. In actual data, it is represented in HDR format and is used to represent the color of the point in the rendered image. Diffuse_reflectance refers to the diffuse reflectance of the object surface. It is a property that describes how evenly an object reflects light in all directions. Diffuse reflectance is usually related to the material properties of the object. For example, a white wall will have a higher diffuse reflectance, while a black object will have a lower diffuse reflectance. Diffuse_illumination refers to the intensity of the diffuse light that hits the surface of an object. It depends on the intensity of the light source, the distance between the object and the light source, and the direction of the light source. This value represents the intensity of the incident light on the surface of the object.

[0065] diffuse_reflectance The diffuse_illumination product calculates how much light is reflected from the surface due to diffuse reflections. This is the main component of the surface color, especially when there is no strong specular reflection.

[0066] Residual is a residual term used to capture lighting effects that do not follow the diffuse reflection model, such as specular reflection, refraction, ambient occlusion, subsurface scattering, etc. The residual term can include any additional lighting effects that are not taken into account in the diffuse reflection model.

[0067] In general, the formula provided by Hypersim is a simplified lighting model, which assumes that the color of the object surface is mainly determined by diffuse lighting and some additional residual lighting effects.

[0068] In the field of OCC (reflective optical camera communication), turning off the OCC light source causes the image to appear as dark stripes, while turning on the OCC light source causes the image to appear as bright stripes. The interweaving of light and dark creates communication stripes of varying widths, and data synthesis is performed based on the Hypersim optical model.

[0069] Figure 2 FIG is a schematic diagram of adding stripes to image samples in a Hypersim dataset to obtain stripe image samples provided by an embodiment of the present application, such as Figure 2 As shown, Is represents a binary sequence used for communication, usually 0 represents a dark stripe and 1 represents a light stripe. represents the expanded mask image, and D represents the depth data corresponding to the image. In OCC communication, the intensity of the light source follows the inverse square law with respect to distance. The farther away, the weaker the reflected light, and therefore, the less noticeable the stripes.

[0070] ; For the above formula, is the stripe signal after the depth map weight, is the original optical fringe signal (or called the real fringe sequence S), represents the square of the distance in the depth map, and r is a random OCC light source emission point. This formula describes the relationship between the change of OCC signal and distance.

[0071] Then Added to the diffuse lighting component, the superimposed light component is obtained , which represents the superposition of the OCC light source and the diffuse reflection light source in real space, where the OCC light source contains a coded signal.

[0072] Next, get ,in, Represents diffuse reflectivity, which describes the reflectivity of different objects in real scenes. By multiplying it with Optical components after different diffuse reflectivity.

[0073] Finally and Add together to get image, It is a non-diffuse residual used to describe ambient light and global illumination, etc.

[0074] The formula for the above process is: ; in is the adjustment factor of the diffuse lighting component, is the residual coefficient, in particular, when = =0, it means that there is no light source other than OCC in the image. At this time, the stripes will not appear transparent, but completely black, and the light in the environment is only provided by the OCC light source.

[0075] The resulting Xocc is an optical component containing the optical signal encoding. After tone mapping (using the Hypersim mapping algorithm), it is converted into an RGB image. Xocc can be called a streak image sample. Each image in the Hypersim dataset is used as a true streak-free image Y.

[0076] Example 2: The following is an example of data style conversion.

[0077] Because the synthetic dataset was generated using 3D scene data, its style still differs from the real stripe dataset. This step uses CycleGAN to perform style conversion on this constructed dataset. CycleGAN (Cycle-Consistent Generative Adversarial Network) is an unsupervised image conversion model based on generative adversarial networks (GANs). Its core concept is to achieve mutual conversion between images in two different domains without paired training data, for example, turning a horse into a zebra, a photograph into a painting, or day into night.

[0078] For each generated stripe image Xocc and a randomly taken real-style stripe image Xocc(real), CycleGAN is used for unsupervised learning to learn the style mapping from the synthetic stripe image to the real-style image, and the final synthetic stripe image dataset Xocc is obtained.

[0079] Example 3: The following is an example explanation of the model structure.

[0080] To design a lightweight model for inference and computation on mobile devices, we developed the GAN-based Image Reconstruction and Stripe Decoding Model (OCCGAN) based on CondConv. CondConv, or Conditionally Parameterized Convolutions, is a dynamic convolution method that allows the network to learn specific convolution kernel parameters for each input sample. This method is a plug-and-play module that can replace conventional convolutions in standard convolutional networks to increase model size and capacity while maintaining efficient inference capabilities.

[0081] Figure 3 This is an overview diagram of the GANOCCAS network structure provided by the embodiment of the present application, such as Figure 3 As shown in the figure, GANOCCAS (a reflective communication decoding model based on a generative adversarial network) consists of two parts: a generator for image reconstruction and a discriminator for extracting fringe signals. These two parts solve two different problems. To ensure mobile deployment and ensure appropriate accuracy and reconstruction quality, the model is designed based on the principles of lightweight and simplicity.

[0082] The generator model for image reconstruction draws on mainstream model architectures and uses a U-NET network. For the last three levels of residual connections, a SimAm attention module is applied to each residual to further enhance the model's representational capabilities. SimAM (Simple Attention Module) is a simple, parameter-free attention module for convolutional neural networks (CNNs). Its core concept is to design an energy function based on neuroscience theory, particularly spatial inhibition theory, to explore the importance of each neuron and infer the 3D attention weights of the feature map.

[0083] SimAm can further improve the expressiveness of the model without increasing any computational complexity, and is suitable for most computer vision tasks.

[0084] In addition, the combination of CondConv+PixelShuffle is used as the upsampling layer of the model. Unlike deconvolution, PixelShuffle overcomes the problem of checkerboard generation in deconvolution. It can be said that PixelShuffle is the best performing upsampling technology currently used. Conv+PixelShuffle is also a common combination in the field of image super-resolution. However, traditional Conv+PixelShuffle will increase the complexity. Feature maps often need to go through Conv to expand the number of channels before PixelShuffle. This application uses CondConv to replace Conv. While ensuring performance, CondConv can significantly reduce the complexity of the model. In order to make the generator model more lightweight, CondConv+PixelShuffle is adopted to achieve a deconvolution effect. Experiments have shown that this method can effectively reduce the complexity of the generator model while ensuring unchanged performance.

[0085] For the discriminator, in addition to helping determine whether the image generated by the generator has completely removed the stripes, the discriminator also needs to continuously learn how to identify real stripes. In the field of OCC communication, the speed of extracting signal stripes will directly affect the overall communication quality. Therefore, it is important to design a lightweight and fast stripe extraction model. After referring to the relevant experience of WinkLink (a reflective visible light communication decoding model), it is learned that Winklink can be understood as a two-stage model, that is, first reconstructing the image and then using the reconstructed image to decode the signal. This design method makes the model well adaptable to various reconstruction scenarios and has good generalization performance. However, this method means that the image needs to be reconstructed once when decoding the signal. However, image reconstruction is a task with relatively high computational complexity. This method of reconstructing the image first and then extracting the stripes seriously affects the final model performance, which in turn affects the decoding rate and communication quality. How to design a fast and accurate stripe decoding model has become an urgent problem to be solved in the OCC field.

[0086] This application did not choose to use mainstream frameworks in the industry, such as Transformers and Mamba, for its design. This is because modules and frameworks such as Vit (Vision Transformer) often have high time complexity and are difficult to deploy on mobile devices, even though Vit has been proven to achieve good results in most computer vision tasks.

[0087] The discriminator is used for stripe extraction and is implemented based on CondConv. A multi-scale convolution and pooling layer is designed, and residual connections are introduced. By continuously compressing the height information, an image is continuously compressed to become a stripe sequence. Among them, the multi-scale convolution layer is implemented based on CondConv as a convolution module. Inspired by pyramid convolution, the two are combined to further reduce the complexity of the model.

[0088] In the pooling stage, a (2,1) convolution kernel and a CondConv with a stride of (2,1) are used to implement the pooling operation. The traditional AvgPool or MaxPool is not used here because experiments have shown that the convolution-based pooling operation has more flexibility, can better aggregate features, and reduce feature loss. ConCat is used for residual operations, which means that the number of channel features will increase after each layer is calculated. Therefore, when a convolution is used as pooling, the number of output channels can be controlled for aggregation to further aggregate the features after the residual and reduce the number of channels.

[0089] Example 4: The following is an example explanation of the generator structure.

[0090] U-Net is a lightweight and simple image reconstruction architecture that is widely used in image reconstruction. Figure 4 This is a diagram of the generator structure provided by the embodiment of the present application, such as Figure 4 As shown in the figure, a lightweight image reconstruction model suitable for mobile deployment is designed based on the U-Net architecture. By reducing the number of U-Net network layers, constructing residual connections, adding parameter-free attention modules, and improving the downsampling module, an efficient and lightweight network structure is achieved. The following is an expanded description of the following contents in the generator structure: (1) image generation unit; (2) residual connections and SimAm attention fusion.

[0091] (1) Image generation unit; The image generation unit consists of downsampling and upsampling modules. The downsampling module is responsible for feature extraction and serves as the Encoder component of the U-Net network. The upsampling module is responsible for restoring features back to the image. Residual connections are used to transfer features between the upsampled and downsampled features. During this residual connection, the SimAm parameter-free attention module is used for feature fusion and enhancement. The final image is output through the image reconstruction layer, which includes a Tanh activation function to control the image pixels between (-1, 1). Finally, for convenience of representation, the (-1, 1) result is further scaled to (0, 1) for output.

[0092] For the input sample represents a striped image, represents the output stripe-free image, where , Where B is the sample batch size, W is the image width, H is the image height, and 3 is the number of image channels. It is an RGB format image. represents the i-th up-sampling and down-sampling module, Represents a rebuild module. Represents the output feature map of the i-th module. Represents the input image X.

[0093] Likewise, Conv stands for convolutional module and CondConv stands for conditional convolution. represents the instance normalization function, and PixelShuffle represents the pixel reorganization module. Represents the residual connection from the downsampling module in the i-th step. PixelShuffle is an efficient upsampling technique for image super-resolution reconstruction. Its core idea is to improve image resolution by rearranging the channel dimension of the feature map to the spatial dimension.

[0094] For M: ; ; ; ; in, represents the mean, represents the standard deviation, is a small constant. When i is equal to the last module number 9, the final value of Y is: .

[0095] (2) Residual connection and SimAm attention fusion; For residual connections ,in Represents the SimAm attention module: ; in It is the SimAm attention module. SimAM is a lightweight, parameter-free attention module based on neuroscience theory. It generates attention weights by calculating the local self-similarity of feature maps, thereby improving the performance of convolutional neural networks.

[0096] When i=9, Represents the original input image. After the image is processed by the convolution layer, when the SNR image is low, there is basically no stripe signal in the image. At this time, the image quality is high, so the model can directly use the original input to ensure the model quality.

[0097] It can be understood that this application designs a lightweight generator model suitable for mobile deployment, and proposes a model structure with only 4 layers of downsampling and 4 layers of upsampling. At the same time, by introducing CondConv combined with PixelShuffle technology to complete the upsampling part, while ensuring the effect, the number of parameters is greatly reduced.

[0098] Compared with the existing model, the improvement is: the number of model layers can be controlled within 9 layers, which significantly reduces the number of model layers; at the same time, in the upsampling part, the transposed convolution is replaced by CondConv+PixelShuffle to implement the upsampling process, which significantly reduces the number of parameters; the original input After the residual connection with the output image, it passes through the image reconstruction layer ( Figure 4 ), thereby ensuring that the model can directly utilize the original input information in low SNR images. This design enables the model to directly utilize the real structural information of the original input when processing images with severe noise interference, avoiding detail loss or over-smoothing caused by intermediate processing links. It significantly improves the model's ability to reconstruct low-quality images and enhances the preservation and recovery of image details. It can still output high-quality and highly restored images under low SNR conditions, effectively improving the processing effect of low SNR images.

[0099] Example 5: The following is an example explanation of the discriminator structure.

[0100] Figure 5 This is a diagram of the discriminator structure provided by the embodiment of the present application, such as Figure 5 As shown in the figure, for the stripe extraction network, the design is based on CondConv (conditional convolution), and a multi-scale pyramid convolution network structure is designed, which greatly reduces the number of parameters while improving the performance.

[0101] Multi-scale pyramid convolution uses 3x3, 5x5, and 7x7 kernels. Input features are evenly divided into three groups. Multi-scale information is extracted through convolution at each of the three scales, and then reassembled through concatenation. Each convolution is replaced by CondConv. Convolutions at different levels use different "groups" parameters to group input and output channels (grouping input and output channels, performing a separate convolution operation on each group, and then concatenating the results). For 3x3, 5x5, and 7x7 kernels, groups equals 1, 2, and 4, respectively. This design significantly reduces computational complexity, ensuring robustness for mobile deployment and high-speed communication decoding.

[0102] The output of each multi-scale module is skipped and connected to the input features to further retain more information. After each multi-scale module, the height is pooled and compressed to further reduce the amount of subsequent computation. The aggregated stripe information is compressed until the height becomes 1. At this point, the image features are compressed into a stripe sequence with a height of 1 and the same width as the original input, thereby achieving stripe extraction.

[0103] Input sample represents a striped image, represents the output stripe sequence, where , , where B is the batch size, W is the image width, and H is the image height.

[0104] The input image first passes through the CondConv convolution network with a convolution kernel size of (2,1) and a step size of (2,1) to extract 48 channel features, and then passes through 5 residual network blocks to extract stripe features. Represents the output feature map of the i-th module, where i ranges from 1 to 7. CondConv3 is a 3x3 convolution kernel, CondConv5 is a 5x5 convolution kernel, CondConv7 is a 7x7 convolution kernel, and CondConv21 is a 2x1 convolution kernel. BN is a batch normalization module, and Relu and Sigmoid are activation functions. Each CondConv contains a BN module and a Relu activation function.

[0105] When i=1: ; When i>1 and i<7: ; ; When i=7, the final output is S: ; For PyConv, the input feature map passes through three convolution kernels of different scales, each convolution kernel outputs 16 dimensions, and then the three outputs are spliced to obtain a 48-dimensional output.

[0106] The final output is scaled to (0, 1) by the Sigmoid function to represent the stripe sequence, where 0 represents stripes and 1 represents no stripes.

[0107] It can be understood that this application designs a model for stripe extraction, which is based on CondConv design and uses the multi-scale pyramid convolution architecture to extract multi-scale features. At the same time, it combines residual connections to further retain more information. In the process of continuous feature extraction, the network continuously compresses and aggregates height information, thereby continuously reducing the computational complexity and finally outputting a stripe sequence.

[0108] Compared with the existing technology, the improvement is as follows: ordinary convolution is replaced by CondConv convolution, and the original maximum pooling is replaced by CondConv with a convolution kernel of (2x1) and a stride of (2,1). While further aggregating features and reducing the number of channels, it can also continuously compress the height like maximum pooling, which not only reduces the model complexity but also improves the model performance.

[0109] Example 6: The following is an example explanation of the loss function.

[0110] Loss is the deviation between the calculated result and the true value (i.e., the expected result) during the learning or training process. Its purpose is to guide subsequent training or learning in the right direction. The loss function specifically defines how to calculate this deviation and is closely linked to the model's structure and final results. represents the total loss function, Represents multi-scale pixel loss, considering that the result of image restoration should be as close as possible to the original Figure 1 The pixel reconstruction loss The purpose is to make the output closer to the original image at the pixel level. is the weight coefficient for generating adversarial loss. is the weight coefficient of MSSSIM loss. The total loss function is as follows: ; .

[0111] (1) Pixel loss; For image reconstruction tasks, the output of the generator G should be as consistent as possible with the real image. Therefore, a multi-scale pixel loss is introduced as the loss function, and the L1 distance is used instead of the L2 distance because the L1 distance encourages the reduction of blur, which can make the image generation clearer. Formula 23 is the L1 loss distance. Represents a true stripe-free image. The pixel loss function is as follows: ; in, represents the input striped image, represents a true stripe-free image, Express expectations.

[0112] (2) Fighting losses; Adversarial loss defines the loss in the adversarial learning process. Adversarial learning is to encourage the input image to generate an image without stripes as much as possible, while making the discriminator learn how to extract stripe sequences from striped images. Represents the input striped image. The adversarial loss function is as follows: .

[0113] (3) Multi-scale SSIM loss; The multi-scale SSIM loss improves upon the SSIM (Structural Similarity Index) by considering structural similarity at multiple scales to more comprehensively assess image similarity. In the multi-scale SSIM loss, the image is downsampled (or upsampled) multiple times to obtain images of different scales. The SSIM index is then calculated at each scale, and the final loss value is obtained by taking a weighted average of the SSIM indices at each scale.

[0114] The advantage of the multi-scale SSIM loss is that it can better capture the structural information of images at different scales, thereby more accurately evaluating the similarity between images. This loss function is often used in tasks such as image generation and super-resolution reconstruction. It helps the model better learn the structural features of images and generate more realistic and clear images.

[0115] In deep learning, multi-scale SSIM loss is often used as part of the loss function to guide the training of the model so that it can generate images with better structural similarity. The value of is to maximize the SSIM value of the generated image G(X) and the true stripe-free image Y. The multi-scale SSIM loss function is as follows: ; (4) Discriminator loss; In addition to the loss used to train the generator, the discriminator is also responsible for learning the Learning to generate realistic stripe sequences Therefore, during the cross-training process, the discriminator should also learn the ability to distinguish between the presence and absence of stripes in order to achieve stripe signal decoding. Used to guide the discriminator to learn how to Learning to generate realistic stripe sequences The discriminator loss function is as follows: ; in, Used to make the discriminator learn how to identify real stripe-free images , The discriminator learns how to Learning to generate realistic stripe sequences .

[0116] Example 7: The following is an example explanation of the experimental process and model complexity.

[0117] (1) Experimental process Use the method in Example 1 above to synthesize 10 scenes in the training set, namely "attic", "bathroom", "corridors", "kitchen", "leisure hall", "living room", "lobby", "office", "staircasecorridor", and "study". The synthesis is performed at three rates: 64bps, 128bps, and 256bps. There are 500 images at each rate and 50 images for each scene, totaling 1500 images. The SNR range is from 0 to -25.

[0118] At the same time, 10 different scenes were synthesized as the test set for testing. The test set scenes do not overlap with the training set at all. They are "art galleries", "bedroom", "corporate building", "living room", "lounge", "museum", "office", "office building", "public lobby", and "venue". Although the scene names of "office" and "living room" are the same, the scene contents are completely different. The test set is synthesized at three rates: 64bps, 128bps, and 256bps. Each scene has 50 images, for a total of 1500 images. The SNR range is (0 to -25).

[0119] The training machine configuration used is as follows: GPU:RTX 3080x2 (20GB) 1; CPU: 12 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz.

[0120] The model of this application uses the above-mentioned training set for model training, and uses the GAN framework to cross-train the generator and discriminator. At the same time, under the same environment, it uses the test set for evaluation. The evaluation content includes image reconstruction effect and bit error rate, among which image reconstruction includes SSIM index and image scene classification index. This application trains the ResNet classification model on the original image of the test set, and then uses the trained Resnet model to evaluate the striped image, the image reconstructed by this model, and the WinkLink reconstructed image, so as to evaluate the effect of the image reconstruction of this application.

[0121] For WinkLink, this application uses the same data set and evaluation method for training and evaluation in the same environment and with the same configuration, and compares it with the model of this application.

[0122] For the calculation of model complexity, the experimental results of this application were obtained by calculating under the same machine configuration, and the CPU and GPU speeds were measured and compared under exactly the same environment.

[0123] (3) Model complexity is shown in Table 2.

[0124] Table 2 Model complexity comparison table

[0125] The following describes the GAN-based stripe image processing device for optical camera communication provided in this application. The GAN-based stripe image processing device for optical camera communication described below and the GAN-based stripe image processing method for optical camera communication described above can be referenced to each other.

[0126] Figure 6 is a structural diagram of a GAN-based fringe image processing device for optical camera communication provided in an embodiment of the present application, such as Figure 6 As shown, the device includes: a data set acquisition module 10, a data migration module 20 and a training module 30. Among them: A data set acquisition module 10 is used to synthesize a stripe image data set; A data migration module 20 is used to migrate samples of the stripe image dataset to a target domain based on the real stripe image and the stripe image dataset through CycleGAN, where the target domain is the domain where the real stripe image is located; A training module 30 is configured to train a generative adversarial network (GAN) based on the stripe image dataset after migration to the target domain, and obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is configured to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is configured to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution CondConv network. The intermediate layer is constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolution CondConv networks. Different conditional convolution CondConv networks in the multi-scale module use different convolution kernels. A residual connection is configured between the input and output of the multi-scale module, and the output layer is constructed based on the activation function.

[0127] It is understandable that the detailed functional implementation of each of the above units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0128] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0129] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830, and a communication bus 840. The processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0130] In addition, the logic instructions in the aforementioned memory 830 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0131] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0132] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0133] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0134] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC.

[0135] The above embodiments can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially produce the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

[0136] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0137] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A GAN-based fringe image processing method for optical camera communication, characterized in that: include: Synthetic stripe image dataset; Based on real stripe images and stripe image datasets, the samples of the stripe image dataset are transferred to the target domain through CycleGAN. The target domain is the domain where the real stripe images are located. Based on the stripe image dataset after migration to the target domain, a generative adversarial network (GAN) is trained to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is used to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution CondConv network. The intermediate layer is constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolution CondConv networks. Different conditional convolution CondConv networks in the multi-scale module use different convolution kernels. A residual connection is configured between the input and output of the multi-scale module, and the output layer is constructed based on the activation function.

2. The GAN-based fringe image processing method for optical camera communication according to claim 1, characterized in that: Also includes: In the process of building the U-NET network, the upsampling layer is constructed based on the conditional convolution CondConv network and the pixel reorganization PixelShuffle module.

3. The GAN-based fringe image processing method for optical camera communication according to claim 1, characterized in that: The multiple conditional convolution CondConv networks are specifically three conditional convolution CondConv networks. The convolution kernels used in the three conditional convolution CondConv networks are The convolution kernel, The convolution kernel and The convolution kernel.

4. The GAN-based fringe image processing method for optical camera communication according to claim 1, characterized in that: The pooling layer of the stripe extraction model is based on the conditional convolution CondConv network to achieve pooling.

5. The GAN-based fringe image processing method for optical camera communication according to claim 1, characterized in that: The stripe image dataset includes stripe image samples, stripe sequence labels corresponding to the stripe image samples, and stripe-free image labels; The training of the generative adversarial network includes: continuously calculating loss values and optimizing model parameters based on a pixel loss function, an adversarial loss function, a multi-scale SSIM loss function, and a discriminator loss function until a training stop condition is met; a pixel loss function for determining a pixel loss based on a streak-free image label and a reconstruction result, wherein the reconstruction result is determined based on an output of the image reconstruction model by inputting the streak image sample into the image reconstruction model; an adversarial loss function for determining an adversarial loss based on a first stripe extraction result, wherein the first stripe extraction result is input into a stripe extraction model and determined based on an output of the stripe extraction model; Multi-scale SSIM loss function, used to determine the multi-scale SSIM loss based on the stripe-free image label and reconstruction results; The discriminator loss function is used to determine the first discriminant loss based on the second stripe extraction result; determine the second discriminant loss based on the third stripe extraction result and the stripe sequence label; determine the comprehensive discriminant loss by weighted summation based on the first discriminant loss and the second discriminant loss; the second stripe extraction result is input by inputting the stripe-free image label into the stripe extraction model and determining it based on the output of the stripe extraction model; the third stripe extraction result is input by inputting the stripe image sample into the stripe extraction model and determining it based on the output of the stripe extraction model.

6. The GAN-based fringe image processing method for optical camera communication according to any one of claims 1 to 5, characterized in that: The stripe image dataset includes stripe image samples, stripe sequence labels corresponding to the stripe image samples, and stripe-free image labels; The synthetic stripe image dataset includes: Based on the Hypersim dataset, stripes are added to each original image sample in the Hypersim dataset to obtain stripe image samples. The original image samples are used as stripe-free image labels, and the added stripes are used as stripe sequence labels.

7. The GAN-based fringe image processing method for optical camera communication according to claim 6, characterized in that: Adding stripes to each original image sample of the Hypersim dataset includes adding stripes by the following formula: ； in, represents a fringe image sample, Represents the depth data corresponding to the original image sample, Represents a random light source emission point, Indicates the added stripes, Represents the diffuse light intensity of the original image sample that illuminates the surface of the object. Represents the diffuse reflectance of the object surface in the original image sample, represents the residual term corresponding to the original image sample, Represents the adjustment factor of the diffuse lighting component, Represents the residual coefficient.

8. A GAN-based fringe image processing device for optical camera communication, characterized in that: include: Dataset acquisition module, used to synthesize stripe image dataset; The data migration module is used to migrate the samples of the stripe image dataset to the target domain based on the real stripe image and the stripe image dataset through CycleGAN. The target domain is the domain where the real stripe image is located; A training module is used to train a generative adversarial network (GAN) based on the stripe image dataset after migration to the target domain, and obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct the stripe image into a stripe-free image, and the stripe extraction model is used to extract stripes based on the stripe image. Among them, the image reconstruction model serves as the generator of GAN, and the stripe extraction model serves as the discriminator of GAN; The image reconstruction model is built based on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections; The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution CondConv network. The intermediate layer is constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolution CondConv networks. Different conditional convolution CondConv networks in the multi-scale module use different convolution kernels. A residual connection is configured between the input and output of the multi-scale module, and the output layer is constructed based on the activation function.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images