A GAN-based stripe image processing method for optical camera communication

Abstract

This application belongs to the field of visible light communication, specifically disclosing a GAN-based stripe image processing method for optical camera communication, including: synthesizing a stripe image dataset; transferring samples from the stripe image dataset to the target domain; training a generative adversarial network (GAN) to obtain an image reconstruction model and a stripe extraction model; the image reconstruction model is built based on the U-NET network and uses the attention module SimAM; the stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple CondConv networks, and residual connections are configured between the input and output of the multi-scale module. Through the lightweight design of the above-mentioned image reconstruction model and stripe extraction model, this application ensures that the model obtained by training the generative adversarial network is suitable for mobile deployment.

Description

Technical Field

[0001] This application belongs to the field of visible light communication, and more specifically, relates to a GAN-based stripe image processing method for optical camera communication. Background Technology

[0002] Visible light communication is a technology that uses the visible light band for communication. It utilizes the characteristics of visible light to transmit data, transmitting information by modulating the brightness or color of the light.

[0003] Visible light communication works by using light sources such as LEDs or lasers to emit light signals, which are then received and decoded by a camera at the receiving end, thus enabling data transmission. In visible light communication scenarios, the light source emits light signals that flicker at a certain frequency, imperceptible to the naked eye. When the light signal reaches the receiving end, the camera's light sensor captures an image at a specific frame rate. Because the flicker frequency of the LEDs differs from the camera's frame rate, stripes appear in the captured image—alternating bright and dark stripes. The spacing and brightness variations of these stripes correspond to the flicker frequency and brightness variations of the LEDs. By extracting these stripes from the image, a stripe sequence can be obtained, allowing for the decoding of the light signal.

[0004] For striped images, there are generally stripe image reconstruction tasks and stripe extraction tasks. Stripe image reconstruction is used to remove stripes from the image and restore the background image, while stripe extraction is used to extract the stripes from the image. Image reconstruction and stripe extraction are a pair of contradictory tasks. When there are too many black stripes in the image, it is beneficial for stripe extraction but detrimental to image reconstruction; conversely, fewer black stripes are beneficial for image reconstruction but detrimental to stripe extraction.

[0005] A Generative Adversarial Network (GAN) consists of two neural networks: a generator and a discriminator. The generator is responsible for generating data samples, while the discriminator is responsible for determining whether the generated data samples are real or fake. GANs work by training the generator and discriminator against each other, enabling the generator to generate sufficiently realistic data samples, while the discriminator can accurately distinguish between real and generated data. During training, the generator continuously generates data samples, and the discriminator continuously learns how to better distinguish between real and generated data, until the data samples generated by the generator can no longer be distinguished by the discriminator.

[0006] In related technologies, considering that image reconstruction and stripe extraction are a pair of opposing (contradictory) tasks, the use of stripe image datasets and GANs to generate adversarial tasks with contradictory properties can effectively balance the accuracy of image reconstruction and stripe extraction, and obtain image reconstruction models and stripe extraction models suitable for visible light communication scenarios.

[0007] However, the image reconstruction and stripe extraction models used in these technologies are quite complex and difficult to apply to mobile deployments. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this application is to create lightweight image reconstruction and stripe extraction models, making them suitable for mobile deployment.

[0009] To achieve the above objectives, in a first aspect, this application provides a GAN-based stripe image processing method for optical camera communication, the method comprising:

[0010] Synthetic stripe image dataset;

[0011] Based on real stripe images and stripe image datasets, CycleGAN is used to transfer samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located.

[0012] Based on the striped image dataset after transfer to the target domain, a generative adversarial network (GAN) is trained to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes from striped images.

[0013] In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN.

[0014] The image reconstruction model is built on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections.

[0015] The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolution (CondConv) network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolutional CondConv networks. Different conditional convolutional CondConv networks in the multi-scale module use different convolution kernels. Residual connections are configured between the input and output of the multi-scale module. The output layer is constructed based on an activation function.

[0016] It is understandable that the lightweight design of the image reconstruction model and the stripe extraction model described above can ensure that the model obtained by training on the generative adversarial network can be deployed on mobile devices.

[0017] Optionally, after acquiring the image reconstruction model and the stripe extraction model, the method further includes: acquiring a stripe image of the surrounding environment through a camera, wherein the light source in the surrounding environment flashes under the control of a visible light communication device;

[0018] Input the stripe image into the image reconstruction model to obtain the reconstructed image output by the image reconstruction model; and / or input the stripe image into the stripe extraction model to obtain the stripe sequence output by the stripe extraction model.

[0019] Understandably, the extracted stripe sequence can be readily used for information decoding in visible light communication. The image reconstruction results can then be used for tasks such as environmental perception, target recognition, and detection.

[0020] One possible implementation also includes: during the construction of the U-NET network, constructing an upsampling layer (or upsampling module) based on the conditional convolutional CondConv network and the pixel reorganization PixelShuffle module.

[0021] Understandably, unlike deconvolution, PixelShuffle overcomes the checkerboard pattern problem inherent in deconvolution, making it arguably the best-performing upsampling technique currently available. Conv+PixelShuffle is also a common combination in image super-resolution, where feature maps often need to be expanded by Conv before PixelShuffle, but traditional Conv+PixelShuffle increases complexity. This application utilizes a combination of CondConv+PixelShuffle as the model's upsampling layer, replacing Conv with CondConv. CondConv significantly reduces model complexity while maintaining performance. Therefore, to make the generator model (image reconstruction model) more lightweight, this application uses CondConv+PixelShuffle to achieve a deconvolution effect. Experiments demonstrate that this implementation effectively reduces generator model complexity without sacrificing performance, ensuring suitability for mobile deployment.

[0022] In one possible implementation, the multiple conditional convolutional CondConv networks specifically consist of three conditional convolutional CondConv networks, with the three conditional convolutional CondConv networks using 3x3, 5x5, and 7x7 convolutional kernels, respectively.

[0023] In one possible implementation, the pooling layer of the stripe extraction model is based on a conditional convolutional CondConv network.

[0024] Understandably, during the pooling stage, a (2,1) convolutional kernel and a (2,1) stride (CondConv) are used to perform the pooling operation. Traditional AvgPool or MaxPool are not used because experiments have shown that convolution-based pooling offers greater flexibility, better feature aggregation, and reduced feature loss. Furthermore, this application uses ConCat for residual operations, meaning that the number of channel features increases after each layer is computed. Therefore, when using a convolutional layer as pooling, the number of output channels can be controlled to further aggregate the residual features and reduce the number of channels.

[0025] In one possible implementation, the striped image dataset includes striped image samples X, striped sequence labels S corresponding to striped image samples X, and non-striped image labels Y.

[0026] Training the generative adversarial network includes: continuously calculating loss values ​​and optimizing model parameters based on pixel loss function, adversarial loss function, multi-scale SSIM loss function and discriminator loss function until the training stopping condition is met;

[0027] The pixel loss function is used to determine the pixel loss based on the stripeless image label Y and the reconstruction result, which is determined by inputting the stripe image sample X into the image reconstruction model and based on the output of the image reconstruction model.

[0028] The adversarial loss function is used to determine the adversarial loss based on the first fringe extraction result, which is determined by inputting the reconstruction result into the fringe extraction model and based on the output of the fringe extraction model.

[0029] A multi-scale SSIM loss function is used to determine the multi-scale SSIM loss based on the stripeless image label Y and the reconstruction result;

[0030] The discriminator loss function is used to determine the first discriminant loss based on the second stripe extraction result; the second discriminant loss based on the third stripe extraction result and the stripe sequence label S; and the comprehensive discriminant loss is determined by weighted summation of the first and second discriminant losses. The second stripe extraction result is determined by inputting the stripeless image label Y into the stripe extraction model and based on the output of the stripe extraction model. The third stripe extraction result is determined by inputting the stripe image sample X into the stripe extraction model and based on the output of the stripe extraction model.

[0031] Stripe sequence labels, also known as true stripe sequences, are used to characterize the stripe sequence contained in a stripe image sample.

[0032] Stripeless image labels, also known as true stripeless images, represent the environmental background obtained after removing the stripe sequence from a striped image sample.

[0033] Optimize model parameters, including adjusting model parameters through backpropagation based on the loss values ​​corresponding to each loss function.

[0034] The above-mentioned training stopping conditions can be either the loss value being less than a preset loss threshold, or reaching the maximum number of training iterations.

[0035] In one possible implementation, the striped image dataset includes striped image samples X, striped sequence labels S corresponding to striped image samples X, and non-striped image labels Y.

[0036] The aforementioned synthetic stripe image dataset includes:

[0037] Based on the Hypersim dataset, stripes are added to each original image sample in the Hypersim dataset to obtain striped image samples. The original image samples are used as labels for images without stripes, and the added stripes are used as labels for stripe sequences.

[0038] Understandably, by adding stripes to the Hypersim dataset, it is possible to efficiently generate striped image samples that closely resemble real-world optical camera communication, thereby effectively improving the efficiency of constructing striped image datasets while ensuring sample quality.

[0039] In one possible implementation, the above-mentioned addition of stripes to each original image sample in the Hypersim dataset includes adding stripes using the following formula:

[0040] ;

[0041] in, Represents a striped image sample. This represents the depth data corresponding to the original image sample. This represents a random light source emission point. Indicates the added stripes, This represents the intensity of diffuse reflected light illuminating the surface of an object in the original image sample. This represents the diffuse reflectance of the object's surface in the original image sample. This represents the residual term corresponding to the original image sample. The adjustment factor representing the diffuse light component. This represents the coefficient of the residual term.

[0042] Secondly, this application provides a GAN-based stripe image processing apparatus for optical camera communication, comprising:

[0043] The dataset acquisition module is used to synthesize stripe image datasets;

[0044] The data migration module is used to transfer samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located, based on real stripe images and stripe image datasets using CycleGAN.

[0045] The training module is used to train a generative adversarial network (GAN) based on the striped image dataset after transfer to the target domain, and to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes from striped images.

[0046] In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN.

[0047] The image reconstruction model is built on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections.

[0048] The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolutional CondConv network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolutional CondConv networks, each using different convolutional kernels. Residual connections are configured between the input and output of the multi-scale module. The output layer is constructed based on an activation function.

[0049] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0050] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0051] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0052] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0053] (1) In order to make the generator model (image reconstruction model) lightweight, this application adopts the attention module SimAM for one or more residual connections of the U-NET network, which can effectively reduce the number of network layers of the U-NET network, thereby reducing the complexity of the generator model and providing a guarantee for mobile deployment.

[0054] (2) To make the discriminator (stripes extraction model) lightweight, this application combines pyramid convolution with CondConv, which effectively reduces the complexity of the discriminator model compared to existing Transformer architectures (such as the Vit framework), while ensuring that performance is not reduced. This provides a guarantee for mobile deployment and high-speed communication decoding.

[0055] (3) This application adopts CondConv+PixelShuffle to achieve a deconvolution effect. Experiments have shown that this implementation can effectively reduce the complexity of the generator model while ensuring that the performance is not reduced. This provides a guarantee for mobile deployment.

[0056] (4) By adding stripes based on the Hypersim dataset, stripe image samples that are close to real optical camera communication can be generated efficiently, thereby effectively improving the efficiency of constructing stripe image datasets while ensuring sample quality. Attached Figure Description

[0057] Figure 1 This is a schematic flowchart of a GAN-based stripe image processing method for optical camera communication provided in an embodiment of this application.

[0058] Figure 2 This is a schematic diagram illustrating the process of adding stripes to image samples from the Hypersim dataset to obtain striped image samples, as provided in an embodiment of this application.

[0059] Figure 3 This is an overview diagram of the GANOCCAS network structure provided in the embodiments of this application;

[0060] Figure 4 This is a generator structure diagram provided in an embodiment of this application;

[0061] Figure 5 This is a structural diagram of the discriminator provided in the embodiments of this application;

[0062] Figure 6 This is a schematic diagram of the structure of a GAN-based stripe image processing device for optical camera communication provided in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] To facilitate a clearer understanding of the various embodiments of this application, some relevant background knowledge will be introduced as follows.

[0065] (1) OCC (Optical Communication with Reflective Cameras) description;

[0066] CMOS employs a rolling shutter mechanism, which utilizes the rolling shutter effect to achieve line-by-line exposure, thereby increasing the data rate of visible light imaging communication. At any given moment, the CMOS image sensor exposes only one row. After all rows of the image have been exposed, the data from all rows captured at different times are combined to form the image. This line-by-line exposure method results in alternating bright and dark stripes (or light stripes, stripe sequences) in the image when photographing rapidly flickering LED lights. The faster the light flickers, the more stripes there are. By decoding these alternating bright and dark stripes in the image, several bits of information can be transmitted, enabling visible light imaging communication. The flicker frequency of the light must be lower than the CMOS line scanning frequency; otherwise, data loss will occur. This is because when the LED flicker frequency is higher than the line scanning frequency, the CMOS sensor can only obtain the average light intensity emitted by the LED, and cannot obtain the bright and dark stripes in the LED pixel area. The exposure time and ISO of the CMOS image sensor affect the acquisition of the LED-ID light stripe.

[0067] (2) Signal modulation method based on PWM;

[0068] Pulse Width Modulation (PWM) is a highly effective technique that uses the digital output of a microprocessor to control analog circuits. It is widely used in many fields, from measurement and communication to power control and conversion.

[0069] PWM frequency refers to the number of times a signal transitions from a high level to a low level and back to a high level within one second (one cycle); that is, how many PWM cycles are there per second. The unit is Hz. It is represented as 50Hz or 100Hz. If the frequency is 50Hz, meaning one cycle is 20ms, then there are 50 PWM cycles per second.

[0070] The PWM modulation method for LED lights affects the LED flicker frequency. Generally speaking, frequencies above 100Hz are imperceptible to the human eye.

[0071] Duty cycle: The ratio of the high-level time to the total time of a pulse cycle.

[0072] Four different duty cycles can be used to represent different signals within one cycle.

[0073] (3) Communication rate calculation;

[0074] PWM modulation can be used to transmit information using four different duty cycles, and the communication rate is related to the PWM frequency. Assuming the PWM frequency is Hz, it changes P times in 1 second.

[0075] Suppose we use PWM duty cycles to represent different symbols (e.g., 4 duty cycles represent 2 bits of binary data):

[0076] Symbol rate: One symbol is transmitted per PWM cycle, so the symbol rate is P symbols per second (Baud).

[0077] Modulation order M: 4 duty cycles; the formula for calculating communication rate (bit rate) is:

[0078] ;

[0079] ;

[0080] Therefore, using four duty cycles for signal modulation, the communication rate is 2P bps, where P is the PWM frequency. For 256bps, P = 128 Hz. For 128 Hz, P = 64 Hz. For 64bps, P = 32 Hz.

[0081] In OCC communication, images form stripes of varying widths due to different duty cycles within the PWM cycle, thus representing different information. For an image with a width of W, the video captures S frames per second, meaning that W frames are generated per second. S is the width of a pixel, and assuming the PWM frequency is P, then these P changes are reflected in W. Within pixel S, the frequencies P and W can be determined. The relationship of S is as follows: one cycle of PWM results in a pixel width of c_pixel:

[0082] c_pixel P=W S;

[0083] When c_pixel=4, W=256, S=2, P=128hz, the communication rate is 256bps;

[0084] When c_pixel=4, W=256, S=30, P=128hz, the communication rate is 3840bps.

[0085] In GANOCCAS, S=2, which means assuming 2 images are captured per second, and the image width is 256. 256. The various rates and corresponding parameters were calculated and are shown in Table 1.

[0086] Table 1. Rate and Corresponding Parameters

[0087]

[0088] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0089] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of the objects. For example, "first stripe extraction result" and "second stripe extraction result," etc., are used to distinguish different stripe extraction results, not to describe a specific order of the stripe extraction results.

[0090] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0091] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0092] The embodiments of this application are described below with reference to the accompanying drawings.

[0093] Figure 1 This is a schematic flowchart of a GAN-based stripe image processing method for optical camera communication provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps S101, S102 and S103.

[0094] Step S101: Synthesize the stripe image dataset;

[0095] Step S102: Based on real stripe images (which can be acquired by shooting in scenarios oriented towards optical camera communication) and stripe image datasets, CycleGAN is used to transfer samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located.

[0096] Step S103: Based on the striped image dataset after transfer to the target domain, train a generative adversarial network (GAN) to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes based on striped images.

[0097] In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN.

[0098] The image reconstruction model is built on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections.

[0099] The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a CondConv convolutional network, the intermediate layers are constructed by cascading multi-scale modules and pooling layers, the multi-scale modules include multiple CondConv convolutional networks, different CondConv convolutional networks in the multi-scale modules use different convolutional kernels, residual connections are configured between the input and output of the multi-scale modules, and the output layer is constructed based on activation functions.

[0100] Specifically, the samples in the synthetic stripe image dataset have certain style differences from real stripe images, thus requiring style transfer. CycleGAN (Cycle-Consistent Generative Adversarial Networks) is a deep learning model used to achieve image-to-image transformation without requiring paired training data. Without paired training data, CycleGAN can be trained using a source domain image set (a dataset constructed from real stripe images) and a target domain image set (i.e., the synthetic stripe image dataset). CycleGAN learns the complex mapping relationship between the source domain (the domain containing the samples in the synthetic stripe image dataset) and the target domain (the domain containing the real stripe images). After learning this mapping relationship, the CycleGAN generator can transfer the samples from the stripe image dataset to the target domain. The samples in the stripe image dataset after transfer to the target domain are closer to real stripe images, ensuring the training effect of the image reconstruction model and the stripe extraction model.

[0101] SimAM (Simple Attention Module) is a simple, parameter-free attention module for convolutional neural networks (CNNs). SimAM can further enhance the expressive power of a model with almost no increase in computational complexity. As the expressive power of the model increases, the number of upsampling and downsampling modules in the U-NET network can be reduced. Experiments show that the number of upsampling / downsampling modules can be reduced to less than four without compromising performance.

[0102] Therefore, in order to make the generator model (image reconstruction model) lightweight, this application adopts the attention module SimAM for one or more residual connections of the U-NET network, which can effectively reduce the number of network layers of the U-NET network, thereby reducing the complexity of the generator model and providing a guarantee for mobile deployment.

[0103] The discriminator (stripes extraction model) is used for stripe extraction. This model is based on CondConv and uses a multi-scale pyramid convolution architecture to extract multi-scale features. It also incorporates residual connections to retain more information. During the continuous feature extraction process, the network continuously compresses and aggregates height information, thereby continuously reducing computational complexity and finally outputting a stripe sequence.

[0104] Therefore, to make the discriminator (stripes extraction model) lightweight, this application combines pyramid convolution with CondConv, which effectively reduces the complexity of the discriminator model compared to existing Transformer architectures (such as the Vit framework) while ensuring no performance degradation. This provides a guarantee for mobile deployment and high-speed communication decoding.

[0105] It is understandable that the lightweight design of the image reconstruction model and the stripe extraction model described above can ensure that the model obtained by training on the generative adversarial network can be deployed on mobile devices.

[0106] The following examples illustrate the GAN-based stripe image processing method for optical camera communication provided in this application.

[0107] Example 1: The following is an exemplary illustration of dataset synthesis.

[0108] This dataset is synthesized based on the Hypersim dataset for training in optical communication. Hypersim is a dataset that provides high-resolution textures and dynamic lighting for realistic 3D scenes, used for overall indoor scene understanding.

[0109] Hypersim provides a simple lighting model, as shown below:

[0110] ;

[0111] `color` is the final calculated color value of the object's surface. `color` is composed of optical components ultimately received by a camera, and in actual data, it is represented using HDR format to indicate the color of that point in the rendered image. `diffuse_reflectance` refers to the diffuse reflectance of the object's surface; it's a property describing how uniformly an object reflects light in all directions. Diffuse reflectance is usually related to the object's material properties; for example, a white wall will have a high diffuse reflectance, while a black object will have a low one. `diffuse_illumination` refers to the intensity of diffused light illuminating the object's surface. It depends on the intensity of the light source, the distance between the object and the light source, and the direction of the light source. This value represents the incident light intensity on the object's surface.

[0112] diffuse_reflectance The product `diffuse_illumination` calculates the intensity of light reflected from an object's surface due to diffuse reflection. This is a major component of an object's surface color, especially in the absence of strong specular reflection.

[0113] The residual is a term used to capture lighting effects that do not conform to the diffuse reflection model, such as specular reflection, refraction, ambient occlusion, and subsurface scattering. The residual can include any additional lighting effects not considered in the diffuse reflection model.

[0114] In summary, the formula provided by Hypersim is a simplified lighting model that assumes the color of an object's surface is primarily determined by diffuse lighting and some additional residual lighting effects.

[0115] In the field of OCC (Optical Camera Communication), turning off the OCC light source results in dark stripes in the image, while turning on the OCC light source results in bright stripes. The alternation of light and dark stripes creates communication stripes of varying widths, which are then synthesized based on the Hypersim optical model.

[0116] Figure 2 This is a schematic diagram illustrating the process of adding stripes to image samples from the Hypersim dataset to obtain striped image samples, as provided in the embodiments of this application. Figure 2 As shown, Is represents the binary sequence used for communication, typically 0 represents dark stripes and 1 represents bright stripes. The image represents the expanded Mask image, and D represents the depth data corresponding to the image. In OCC communication, the intensity of the light source follows an inverse square law with respect to distance. The farther away the light source, the weaker the reflected light becomes, and therefore, the less obvious the stripes are.

[0117] ;

[0118] For the above formula, It is the stripe signal after depth map weighting. It is the original optical fringe signal (or the real fringe sequence S). Let r represent the squared distance in the depth map, where r is a random OCC light source emission point. This formula describes the relationship between the OCC signal and distance.

[0119] Next Adding the diffuse reflection component to the light component yields the superimposed light component. In real space, this represents the superposition of an OCC light source and a diffuse light source, where the OCC light source contains encoded signals.

[0120] Next, get ,in, This represents diffuse reflectance, describing the reflectance of different objects in a real-world scene. Multiplying it by this value yields... Optical components after passing through different diffuse reflectances.

[0121] Finally and Adding them together gives image, It is a non-diffuse residual used to describe ambient light and global illumination, etc.

[0122] The formula for the above process is expressed as follows:

[0123] ;

[0124] in It is a modulating factor for the diffuse reflection component. It is the coefficient of the residual term, especially when = When =0, it means that there is no light source other than OCC in the image. At this time, the stripes will not be transparent, but will appear completely black, and the light in the environment is only provided by the OCC light source.

[0125] The resulting Xocc is an optical component containing the optical signal encoding. After tone mapping (using Hypersim's mapping algorithm), it becomes an RGB image. The Xocc can be referred to as a striped image sample. Each image in the Hypersim dataset serves as a real stripe-free image Y.

[0126] Example 2: The following is an exemplary illustration of data style transformation.

[0127] Since the synthetic dataset is generated using 3D scene data, its style still differs from the real stripe dataset. This step utilizes CycleGAN to perform style transfer on the constructed dataset. CycleGAN (Cycle-Consistent Generative Adversarial Network) is an unsupervised image transfer model based on Generative Adversarial Networks (GANs). Its core idea is to achieve mutual conversion between images from two different domains without paired training data, such as turning a horse into a zebra, a photograph into an oil painting, or daytime into nighttime.

[0128] For each generated stripe image Xocc, and a randomly captured real-style stripe image Xocc(real), CycleGAN is used for unsupervised learning to learn the style mapping from the synthetic stripe image to the real-style image, thus obtaining the final synthetic stripe image dataset Xocc.

[0129] Example 3: The following is an exemplary illustration of the model structure.

[0130] To design a lightweight model for inference and computation on mobile devices, a GAN-based image reconstruction-stripes decoding model (OCCGAN) was developed based on CondConv. CondConv, or Conditionally Parameterized Convolutions, is a dynamic convolution method that allows the network to learn specific convolution kernel parameters for each input sample. This approach is a plug-and-play module that can replace regular convolutions in standard convolutional networks to improve model size and capacity while maintaining efficient inference capabilities.

[0131] Figure 3 This is an overview diagram of the GANOCCAS network structure provided in the embodiments of this application, such as... Figure 3As shown, GANOCCAS (a reflective communication decoding model based on generative adversarial networks) consists of two parts: a generator for image reconstruction and a discriminator for extracting stripe signals. These two parts address two different problems. To suit mobile deployment and ensure corresponding accuracy and reconstruction results, the model was designed based on the principles of lightweight and simplification.

[0132] For the generator model used for image reconstruction, after drawing on mainstream model architectures, a U-NET network was used for construction. For the last three residual connections, a SimAM attention module was used on each residual to further enhance the model's representation ability. SimAM (Simple Attention Module) is a simple and parameter-free attention module for convolutional neural networks (CNNs). Its core idea is based on neuroscience theory, especially spatial inhibition theory, to design an energy function to explore the importance of each neuron and infer the 3D attention weights of the feature map accordingly.

[0133] SimAm can further enhance the expressive power of a model without increasing computational complexity, making it suitable for most computer vision tasks.

[0134] In addition, a combination of CondConv and PixelShuffle is used as the upsampling layer of the model. Unlike deconvolution, PixelShuffle overcomes the checkerboard pattern problem that deconvolution easily produces. It can be said that PixelShuffle is currently the best performing upsampling technique. Conv+PixelShuffle is also a common combination in the field of image super-resolution. However, traditional Conv+PixelShuffle increases complexity, as feature maps often need to be expanded by Conv to increase the number of channels before PixelShuffle. This application uses CondConv instead of Conv. CondConv can significantly reduce the complexity of the model while maintaining performance. To make the generator model more lightweight, CondConv+PixelShuffle is used to achieve a deconvolution effect. Experiments show that this method can effectively reduce the complexity of the generator model while maintaining performance.

[0135] For the discriminator, in addition to helping determine whether the image generated by the generator has completely removed the stripes, it also needs to learn how to recognize real stripes. In the field of OCC communication, the speed of signal stripe extraction directly relates to and affects the overall communication quality. Therefore, designing a lightweight and fast stripe extraction model is quite important. After referring to the relevant experience of WinkLink (a reflective visible light communication decoding model), it is understood that WinkLink can be understood as a two-stage model, that is, first reconstructing the image and then using the reconstructed image to decode the signal. This design allows the model to adapt well to various reconstruction scenarios and has good generalization performance. However, this means that the image needs to be reconstructed once when decoding the signal. However, image reconstruction is a relatively computationally complex task. This method of reconstructing the image first and then extracting stripes seriously affects the final model performance, and thus affects the decoding rate and communication quality. How to design a fast and accurate stripe decoding model has become an urgent problem to be solved in the OCC field.

[0136] This application did not choose to use mainstream frameworks in the industry, such as Transformers and Mamba, because Vit (Vision Transformer) and such modules and frameworks often have high time complexity and are difficult to deploy on mobile devices, even though Vit has been proven to achieve good results on most computer vision tasks.

[0137] The discriminator is used for stripe extraction and is implemented based on CondConv. A multi-scale convolution and pooling layer is designed, and residual connections are introduced. By continuously compressing the height information, the height information of an image is continuously compressed to become a stripe sequence. The multi-scale convolution layer is implemented based on CondConv as the convolution module. Inspired by pyramid convolution, the two are combined to further reduce the complexity of the model.

[0138] During the pooling stage, a (2,1) convolution kernel and a (2,1) stride CondConv are used to perform the pooling operation. The traditional AvgPool or MaxPool are not used here because experiments have shown that convolution-based pooling operations have more flexibility, can better aggregate features, and reduce feature loss. ConCat is used for residual operations, which means that the number of channel features increases after each layer is computed. Therefore, when using a convolution as pooling, the number of output channels can be controlled to further aggregate the residual features and reduce the number of channels.

[0139] Example 4: The following is an exemplary illustration of generator structure.

[0140] U-Net is a lightweight and simple image reconstruction architecture that is widely used in image reconstruction. Figure 4 This is a generator structure diagram provided in the embodiments of this application, such as... Figure 4 As shown, a lightweight image reconstruction model suitable for mobile deployment is designed based on the U-Net architecture. By reducing the number of U-Net network layers, constructing residual connections, adding a parameterless attention module, and improving the downsampling module, an efficient and lightweight network structure is achieved. The following is a detailed explanation of the following contents in the generator structure: (1) Image generation unit; (2) Residual connections and SimAm attention fusion.

[0141] (1) Image generation unit;

[0142] The image generation unit consists of downsampling and upsampling modules. The downsampling module is responsible for feature extraction and is the Encoder part of the U-Net network. The upsampling module is responsible for restoring the features back to the image. Residual connections are used to transfer features between the upsampled and downsampled features. In the residual connection process, the SimAm parameterless attention module is used for feature fusion and enhancement. The final generated image is output through an image reconstruction layer, which contains a Tanh activation function to control the image pixels between (-1,1). Finally, for ease of representation, the (-1,1) result is further scaled to (0,1) for the output.

[0143] For input samples Indicates a striped image. This represents the output stripeless image, where , Where B is the batch size, W is the image width, H is the image height, and 3 is the number of image channels. It is an RGB format image. This indicates the i-th upsampling / downsampling module. This indicates the module to be rebuilt. This represents the output feature map of the i-th module. This represents the input image X.

[0144] Similarly, Conv represents a convolutional module, and CondConv represents conditional convolution. This represents the instance normalization function, and PixelShuffle represents the pixel reorganization module. This represents the residual connection from the downsampling module at step i. PixelShuffle is an efficient upsampling technique for image super-resolution reconstruction. Its core idea is to improve image resolution by rearranging the channel dimension of the feature map to the spatial dimension.

[0145] For M:

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] in, This represents the mean. Indicates standard deviation, It is a relatively small constant. When i equals the last module number 9, the final value of Y is:

[0151] .

[0152] (2) Residual connections and SimAm attention fusion;

[0153] For residual connectivity ,in Indicates the SimAm attention module:

[0154] ;

[0155] in It is the SimAm attention module. SimAM is a lightweight, parameter-free attention module based on neuroscience theory. It generates attention weights by calculating the local self-similarity of feature maps, thereby improving the performance of convolutional neural networks.

[0156] When i=9 This represents the original input image. After processing the image through a convolutional layer, when the SNR of the image is low, there are basically no stripe signals in the image. At this time, the image quality is high, so the model can directly use the original input to ensure the quality of the model.

[0157] Understandably, this application designs a lightweight generator model suitable for mobile deployment, proposes a model structure with only 4 layers of downsampling and 4 layers of upsampling, and completes the upsampling part by introducing CondConv combined with PixelShuffle technology, which greatly reduces the number of parameters while ensuring the effect.

[0158] Compared to existing models, the improvements include: reducing the number of model layers to within 9; replacing transposed convolution with CondConv+PixelShuffle in the upsampling process, significantly reducing the number of parameters; and modifying the original input... After residual connection with the output image, it passes through the image reconstruction layer ( Figure 4 The design of the model (layer 9) ensures that the model can directly utilize the original input information in low SNR images. This design enables the model to directly utilize the real structural information of the original input when processing images with severe noise interference, avoiding detail loss or excessive smoothing caused by intermediate processing steps. This significantly improves the model's ability to reconstruct low-quality images, enhances the preservation and restoration of image details, and thus can still output high-quality, high-fidelity images under low SNR conditions, effectively improving the processing effect of low SNR images.

[0159] Example 5: The following is an exemplary illustration of the discriminator structure.

[0160] Figure 5 This is a structural diagram of the discriminator provided in the embodiments of this application, such as... Figure 5 As shown, the stripe extraction network is designed based on CondConv (conditional convolution), and a multi-scale pyramid convolutional network structure is designed to significantly reduce the number of parameters while improving performance.

[0161] Multi-scale pyramid convolution, using 3x3, 5x5, and 7x7 kernels, divides the input features into three groups, extracting multi-scale information through convolution at each of the three scales. The results are then concatenated using `CondConv`, with each convolution replaced by a `CondConv` parameter. Different levels of convolution use different `groups` parameters for channel grouping (grouping input and output channels, performing convolution on each group separately, and then concatenating the results). The 3x3, 5x5, and 7x7 kernels correspond to `groups` values ​​of 1, 2, and 4, respectively. This design significantly reduces computational complexity, ensuring compatibility with mobile deployments and high-speed communication decoding.

[0162] For the output of each multi-scale module, a residual connection (Skip Connection) is performed with the input features to further retain more information. At the same time, after each multi-scale module, the height is pooled and compressed to further reduce the subsequent computation. Simultaneously, the aggregated stripe information is compressed until the height becomes 1. At this point, the image features are compressed into a stripe sequence with a height of 1 and a width the same as the original input, thereby achieving stripe extraction.

[0163] Input Sample Indicates a striped image. This represents the output stripe sequence, where , Where B is the batch size, W is the image width, and H is the image height.

[0164] The input image first passes through a CondConv convolutional network with a kernel size of (2,1) and a stride of (2,1) to extract 48-channel features. Then, it passes through five residual networks (Block) to extract stripe features. Let represent the output feature map of the i-th module, where i takes values ​​in the range [1, 7]. Here, CondConv3 is a 3x3 convolutional kernel, CondConv5 is a 5x5 convolutional kernel, CondConv7 is a 7x7 convolutional kernel, CondConv21 is a 2x1 convolutional kernel, BN is the batch normalization module, and ReLU and Sigmoid are activation functions. Each CondConv module contains both a BN module and a ReLU activation function.

[0165] When i=1:

[0166] ;

[0167] When i>1 and i<7:

[0168] ;

[0169] ;

[0170] When i=7, the final output is S:

[0171] ;

[0172] For PyConv, the input feature map is processed by three convolutional kernels of different scales, each kernel outputting 16 dimensions, and then the three outputs are concatenated to obtain a 48-dimensional output.

[0173] The final output is scaled to the range (0,1) by the Sigmoid function to represent the stripe sequence, where 0 indicates stripes and 1 indicates no stripes.

[0174] Understandably, this application designs a model for stripe extraction. This model is based on CondConv and uses a multi-scale pyramid convolution architecture to extract multi-scale features. It also incorporates residual connections to retain more information. During the continuous feature extraction process, the network continuously compresses and aggregates height information, thereby continuously reducing computational complexity and finally outputting a stripe sequence.

[0175] Compared to existing technologies, the improvement is as follows: ordinary convolution is replaced with CondConv convolution, and the original max pooling is replaced with CondConv implementation with a kernel of (2x1) and a stride of (2,1). While further aggregating features and reducing the number of channels, it can also continuously compress the height like max pooling, which not only reduces the model complexity but also improves the model performance.

[0176] Example 6: The following is an exemplary illustration of the loss function.

[0177] Loss refers to the deviation between the calculated result and the true value (i.e., the expected result) during the learning or training process. Its purpose is to guide subsequent training or learning processes in the correct direction. The loss function specifically defines how this deviation is calculated, and it is closely related to the model structure and the final result. Represents the total loss function. This represents multi-scale pixel loss, considering that the result of image inpainting should be as close as possible to the original. Figure 1 As a result, pixel reconstruction loss This is to make the output closer to the original image at the pixel level. These are the weighting coefficients for generating adversarial losses. These are the weighting coefficients for the MSSSIM loss. The total loss function is as follows:

[0178] ;

[0179] .

[0180] (1) Pixel loss;

[0181] For image reconstruction tasks, the output of the generator G should be as consistent as possible with the real image. Therefore, multi-scale pixel loss is introduced as the loss function, using L1 distance instead of L2 distance because L1 distance encourages blur reduction and can make the generated image clearer. Equation 23 represents the L1 loss distance. This represents a true, stripe-free image. The pixel loss function is as follows:

[0182] ;

[0183] in, This represents the input striped image. Represents a true stripe-free image. It expresses expectation.

[0184] (2) Combating losses;

[0185] Adversarial loss is defined as the loss in the adversarial learning process. Adversarial learning aims to encourage the input image to generate an image without stripes, while simultaneously teaching the discriminator how to extract stripe sequences from a striped image. The input image is a striped image. The adversarial loss function is as follows:

[0186] .

[0187] (3) Multiscale SSIM loss;

[0188] Multi-scale SSIM loss is an improvement on SSIM (Structural Similarity Index), which more comprehensively evaluates the similarity between images by considering structural similarity at multiple scales. In multi-scale SSIM loss, the image is downsampled (or upsampled) multiple times to obtain images at different scales, and then the SSIM index at each scale is calculated. Finally, the SSIM indices at all scales are weighted and averaged to obtain the final loss value.

[0189] The advantage of multi-scale SSIM loss lies in its ability to better capture structural information of images at different scales, thus more accurately assessing the similarity between images. This loss function is commonly used in tasks such as image generation and super-resolution reconstruction, helping models to better learn the structural features of images and generate more realistic and clearer images.

[0190] In deep learning, multi-scale SSIM loss is often used as part of the loss function to guide the training of the model, enabling it to generate images with better structural similarity. This is achieved by minimizing... The value of is the SSIM value that maximizes the difference between the generated image G(X) and the real stripeless image Y. The multi-scale SSIM loss function is as follows:

[0191] ;

[0192] (4) Discriminator loss;

[0193] In addition to the loss function used to train the generator, the discriminator also undertakes the task of learning from the input image. Learning to generate realistic stripe sequences Therefore, during the cross-training process, the discriminator should also learn to distinguish between the presence and absence of stripes in order to decode the stripe signal. Used to guide the discriminator in learning how to interpret input images. Learning to generate realistic stripe sequences The weights are determined. The discriminator loss function is as follows:

[0194] ;

[0195] in, Used to enable the discriminator to learn how to recognize real, stripeless images. , This enables the discriminator to learn how to distinguish from the input image. Learning to generate realistic stripe sequences .

[0196] Example 7: The following provides an illustrative explanation of the experimental process and model complexity.

[0197] (1) Experimental procedure

[0198] Using the method described in Example 1 above, a training set of 10 scenes was synthesized: "attic", "bathroom", "corridors", "kitchen", "leisure hall", "living room", "lobby", "office", "staircasecorridor", and "study". The scenes were synthesized at three speeds: 64bps, 128bps, and 256bps. Each speed had 500 images, and each scene had 50 images, for a total of 1500 images. The SNR ranged from 0 to -25.

[0199] Simultaneously, 10 other different scenes were synthesized as a test set for testing. The test set scenes did not overlap with the training set at all. They were "art galleries", "bedroom", "corporate building", "living room", "lounge", "museum", "office", "office building", "public lobby", and "venue". Although "office" and "living room" had the same scene name, their scene content was completely different. The test set was synthesized at three rates: 64bps, 128bps, and 256bps. Each scene had 50 images, for a total of 1500 images. The SNR range was (0 to -25).

[0200] The training machine configuration used is as follows:

[0201] GPU: RTX 3080x2 (20GB) 1;

[0202] CPU: 12 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz.

[0203] The model in this application is trained using the aforementioned training set. The generator and discriminator are cross-trained using the GAN framework. Simultaneously, the model is evaluated using a test set under the same environment. The evaluation includes image reconstruction performance and bit error rate. Image reconstruction includes SSIM metrics and image scene classification metrics. This application trains a ResNet classification model on the original images in the test set. Then, the trained ResNet model is used to evaluate the striped image, the image reconstructed by this model, and the WinkLink reconstructed image, thereby evaluating the image reconstruction performance of this application.

[0204] For WinkLink, this application trains and evaluates the model using the exact same dataset and evaluation method under the same environment and configuration, and compares it with the model in this application.

[0205] For the calculation of model complexity, the experimental results of this application were obtained under the same machine configuration, and the CPU and GPU speeds were compared and measured in the same environment.

[0206] (3) The model complexity is shown in Table 2.

[0207] Table 2 Model Complexity Comparison Table

[0208]

[0209] The following describes the GAN-based stripe image processing apparatus for optical camera communication provided in this application. The GAN-based stripe image processing apparatus for optical camera communication described below can be referred to in correspondence with the GAN-based stripe image processing method for optical camera communication described above.

[0210] Figure 6 This is a schematic diagram of the structure of a GAN-based stripe image processing device for optical camera communication provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes: a dataset acquisition module 10, a data transfer module 20, and a training module 30. Wherein:

[0211] Data set acquisition module 10 is used to synthesize stripe image datasets;

[0212] The data migration module 20 is used to migrate samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located, based on real stripe images and stripe image datasets, using CycleGAN.

[0213] Training module 30 is used to train a generative adversarial network (GAN) based on a striped image dataset transferred to the target domain, and to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes based on striped images.

[0214] In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN.

[0215] The image reconstruction model is built on the U-NET network. During the construction process, the attention module SimAM is used for one or more residual connections.

[0216] The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolutional CondConv network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolutional CondConv networks, each using different convolutional kernels. Residual connections are configured between the input and output of the multi-scale module. The output layer is constructed based on an activation function.

[0217] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0218] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0219] Based on the methods in the above embodiments, this application provides an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0220] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0221] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0222] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0223] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0224] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0225] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0226] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0227] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A GAN-based stripe image processing method for optical camera communication, characterized in that, include: Synthetic stripe image dataset; Based on real stripe images and stripe image datasets, CycleGAN is used to transfer samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located. Based on the striped image dataset after transfer to the target domain, a generative adversarial network (GAN) is trained to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes from striped images. In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN. The image reconstruction model is built on the U-NET network. During the construction process, for one or more residual connections, the attention module SimAM is used to reduce the number of network layers in the U-NET network. The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolutional CondConv network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolutional CondConv networks. Different conditional convolutional CondConv networks in the multi-scale module use different convolutional kernels. Residual connections are configured between the input and output of the multi-scale module. The output layer is constructed based on activation functions. It also includes: during the construction of the U-NET network, an upsampling layer is built based on the conditional convolutional CondConv network and the pixel reorganization PixelShuffle module; The striped image dataset includes striped image samples, striped image sequence labels corresponding to the striped image samples, and labels for images without stripes; The synthetic stripe image dataset includes: Based on the Hypersim dataset, stripes are added to each original image sample in the Hypersim dataset to obtain striped image samples. The original image samples are used as labels for images without stripes, and the added stripes are used as labels for stripe sequences. The step of adding stripes to each original image sample in the Hypersim dataset includes adding stripes using the following formula: ； in, Represents a striped image sample. This represents the depth data corresponding to the original image sample. This represents a random light source emission point. Indicates the added stripes, This represents the intensity of diffuse reflected light illuminating the surface of an object in the original image sample. This represents the diffuse reflectance of the object's surface in the original image sample. This represents the residual term corresponding to the original image sample. The adjustment factor representing the diffuse light component. This represents the coefficient of the residual term.

2. The GAN-based stripe image processing method for optical camera communication according to claim 1, characterized in that, The multiple conditional convolutional CondConv network specifically consists of three conditional convolutional CondConv networks. The convolutional kernels used in the three conditional convolutional CondConv networks are as follows: convolution kernel, convolution kernel and The convolution kernel.

3. The GAN-based stripe image processing method for optical camera communication according to claim 1, characterized in that, The pooling layer of the stripe extraction model is implemented based on the conditional convolutional CondConv network.

4. The GAN-based stripe image processing method for optical camera communication according to claim 1, characterized in that, The striped image dataset includes striped image samples, striped image sequence labels corresponding to the striped image samples, and labels for images without stripes; The training of the generative adversarial network includes: continuously calculating loss values ​​and optimizing model parameters based on pixel loss function, adversarial loss function, multi-scale SSIM loss function and discriminator loss function until the training stopping condition is met; The pixel loss function is used to determine the pixel loss based on the stripeless image label and the reconstruction result. The reconstruction result is determined by inputting stripe image samples into the image reconstruction model and based on the output of the image reconstruction model. The adversarial loss function is used to determine the adversarial loss based on the first fringe extraction result, which is determined by inputting the reconstruction result into the fringe extraction model and based on the output of the fringe extraction model. A multi-scale SSIM loss function is used to determine the multi-scale SSIM loss based on the stripeless image label and reconstruction results; The discriminator loss function is used to determine the first discriminant loss based on the second stripe extraction result; the second discriminant loss based on the third stripe extraction result and the stripe sequence label; and the comprehensive discriminant loss based on the first and second discriminant losses through weighted summation. The second stripe extraction result is determined by inputting the stripeless image label into the stripe extraction model and based on the output of the stripe extraction model. The third stripe extraction result is determined by inputting the stripe image sample into the stripe extraction model and based on the output of the stripe extraction model.

5. A GAN-based stripe image processing device for optical camera communication, characterized in that, include: The dataset acquisition module is used to synthesize stripe image datasets; The data migration module is used to transfer samples from the stripe image dataset to the target domain, which is the domain where the real stripe images are located, based on real stripe images and stripe image datasets using CycleGAN. The training module is used to train a generative adversarial network (GAN) based on the striped image dataset after transfer to the target domain, and to obtain an image reconstruction model and a stripe extraction model. The image reconstruction model is used to reconstruct striped images into stripe-free images, and the stripe extraction model is used to extract stripes from striped images. In this model, the image reconstruction model serves as the generator of the GAN, and the stripe extraction model serves as the discriminator of the GAN. The image reconstruction model is built on the U-NET network. During the construction process, for one or more residual connections, the attention module SimAM is used to reduce the number of network layers in the U-NET network. The stripe extraction model is constructed by cascading an input layer, multiple intermediate layers, and an output layer. The input layer is a conditional convolutional CondConv network. The intermediate layers are constructed by cascading multi-scale modules and pooling layers. The multi-scale module includes multiple conditional convolutional CondConv networks. Different conditional convolutional CondConv networks in the multi-scale module use different convolutional kernels. Residual connections are configured between the input and output of the multi-scale module. The output layer is constructed based on activation functions. It also includes: during the construction of the U-NET network, an upsampling layer is built based on the conditional convolutional CondConv network and the pixel reorganization PixelShuffle module; The striped image dataset includes striped image samples, striped image sequence labels corresponding to the striped image samples, and labels for images without stripes; The synthetic stripe image dataset includes: Based on the Hypersim dataset, stripes are added to each original image sample in the Hypersim dataset to obtain striped image samples. The original image samples are used as labels for images without stripes, and the added stripes are used as labels for stripe sequences. The step of adding stripes to each original image sample in the Hypersim dataset includes adding stripes using the following formula: ； in, Represents a striped image sample. This represents the depth data corresponding to the original image sample. This represents a random light source emission point. Indicates the added stripes, This represents the intensity of diffuse reflected light illuminating the surface of an object in the original image sample. This represents the diffuse reflectance of the object's surface in the original image sample. This represents the residual term corresponding to the original image sample. The adjustment factor representing the diffuse light component. This represents the coefficient of the residual term.

6. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-4.

Images