Image data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses an image data processing method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a degraded image; inputting the degraded image into a target restoration model to obtain a clear image, the target restoration model being obtained by training an initial restoration model by using the first sample image, the initial restoration model comprising a background light extraction branch, a transmission light extraction branch and a target output branch, the target output branch being used for removing transmission noise of the first sample image, and the background light extraction branch being used for extracting the transmission noise of the second sample image; initial image data are generated, transmission noise of a second sample image determined by the background light data, the transmission light data and the initial image data is removed, a clear sample image is obtained, the clear sample image is used for adjusting model parameters of an initial restoration model, and a target restoration model is obtained. The technical problem that the removal accuracy of the transmission noise of the image is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for processing image data, a storage medium, and an electronic device. Background Art

[0002] In the process of image processing, supervised learning methods are often used to train image processing models, and then the image processing models are used to remove the transmission noise of the original collected images. Due to the difference between the sample images used in the training process and the real image acquisition environment, the generalization ability of the model is insufficient and the actual application effect is limited. Therefore, in actual application scenarios, the transmission noise removal effect of the original image is not good, and the image clarity improvement is limited.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing image data, a storage medium, and an electronic device to at least solve the technical problem of low accuracy in removing transmission noise of an image.

[0005] According to one aspect of an embodiment of the present application, a method for processing image data is provided, comprising: acquiring a degraded image, wherein the degraded image represents an image with transmission noise; inputting the degraded image into a target restoration model to obtain a clear image, wherein the target restoration model is obtained by training an initial restoration model using a first sample image, the initial restoration model comprises a background light extraction branch, a transmission light extraction branch and a target output branch, the background light extraction branch is used to extract background light data of the first sample image, the transmission light extraction branch is used to extract transmission light data of the first sample image, the target output branch is used to remove transmission noise of the first sample image, generate initial image data, and remove transmission noise of a second sample image determined by the background light data, the transmission light data and the initial image data to obtain a clear sample image, the clear sample image is used to adjust model parameters of the initial restoration model to obtain the target restoration model.

[0006] According to another aspect of an embodiment of the present application, there is also provided an image data processing device, comprising: an acquisition module, for acquiring a degraded image, wherein the degraded image represents an image with transmission noise; a processing module, for inputting the degraded image into a target restoration model to obtain a clear image, wherein the target restoration model is obtained by training an initial restoration model using a first sample image, the initial restoration model comprises a background light extraction branch, a transmission light extraction branch and a target output branch, the background light extraction branch is used to extract background light data of the first sample image, the transmission light extraction branch is used to extract transmission light data of the first sample image, the target output branch is used to remove transmission noise of the first sample image, generate initial image data, and remove transmission noise of a second sample image determined by the background light data, the transmission light data and the initial image data, to obtain a clear sample image, the clear sample image is used to adjust model parameters of the initial restoration model to obtain the target restoration model.

[0007] Optionally, the device is also used to: acquire the first sample image; input the first sample image into the initial restoration model to obtain the background light data, the transmitted light data and the initial image data, wherein the transmitted light data represents image data with noise added to the depth image data of the first sample image, the transmitted light data is used to indicate the light transmittance state of the first sample image, and the background light data is used to indicate the image brightness of the first sample image; generate a second sample image according to the background light data, the transmitted light data and the initial image data; use the target output branch to remove the transmission noise of the second sample image to obtain the clear sample image, wherein the target output branch includes multiple convolutional layers; calculate a training loss value based on the clear sample image, and when the training loss value is less than or equal to a loss threshold, determine the initial restoration model as the target restoration model.

[0008] Optionally, the device is used to generate a second sample image based on the background light data, the transmitted light data and the initial image data in the following manner: determine the product of the initial image data and the transmitted light data as first data; determine the difference between the transmitted light data and a constant 1 and the product of the background light data as second data; and determine the second sample image based on the first data and the second data.

[0009] Optionally, the device is also used to: use the depth extraction module in the transmitted light extraction branch to perform a depth information extraction operation on the first sample image to obtain depth image data, and determine initial transmitted light data based on the depth image data and the scattering coefficient; use the adapter module in the transmitted light extraction branch to add Gaussian noise to the initial transmitted light data to obtain the transmitted light data; and generate the second sample image based on the transmitted light data.

[0010] Optionally, the device is further used to: use the background light extraction branch to perform a Fourier transform operation on the first sample image to obtain the background light data, wherein the background light extraction branch includes multiple convolution layers; and generate the second sample image based on the background light data.

[0011] Optionally, the device is also used to: establish a target loss function based on the first sample image, the second sample image and the clear sample image; use the target loss function to determine the training loss value; and when the training loss value is less than or equal to a loss threshold, determine the initial restoration model as the target restoration model.

[0012] Optionally, the device is used to establish a target loss function based on the background light data, the transmitted light data and the initial image data in at least one of the following ways: establishing a first loss function based on the mean square error of the image data of the first sample image and the image data of the second sample image, wherein the target loss function includes the first loss function; establishing a second loss function based on the image data of the clear sample image and the mean square error of the image data obtained after performing a gradient stop operation on the initial image data, wherein the target loss function includes the second loss function; establishing a third loss function using partial derivatives of initial transmitted light data, wherein the initial transmitted light data is determined by depth image data and a scattering coefficient of the first sample image, and the depth image data is obtained by performing a depth information extraction operation on the first sample image by a depth extraction module in the transmitted light extraction branch, and the target loss function includes the third loss function.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for processing image data when running.

[0014] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method for processing image data.

[0015] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned method for processing image data through the computer program.

[0016] In an embodiment of the present application, a degraded image is first acquired, and then the degraded image is input into a target restoration model to obtain a clear image, and the target restoration model is obtained by performing unsupervised learning training on the initial restoration model. Specifically, the training process of the target restoration model first uses the first sample image to generate a second sample image, and removes the transmission noise from the second sample image to obtain a clear sample image. In this process, the generation of the second sample image combines the background light data, the transmission light data and the initial image data of the first sample image, wherein the background light data reflects the brightness of the image, and the transmission light data is obtained by adding Gaussian noise to the depth image data of the first sample image to simulate the light transmission in a real environment, and the initial image data is the data obtained by directly performing a convolution operation on the first sample image to preliminarily remove the transmission noise.

[0017] Then, by calculating the training loss value between the clear sample image and the first sample image, the parameters of the initial restoration model are optimized, so that the target restoration model can more accurately restore a clear image without transmission noise from the degraded image. That is, by designing a reasonable unsupervised training process and loss function, the purpose of improving the accuracy of transmission noise removal is achieved, thereby achieving the technical effect of efficiently and accurately removing image transmission noise without pairing clean images, thereby solving the technical problem of low transmission noise removal accuracy in the field of image enhancement.

[0018] In summary, the target restoration model in the embodiment of the present application significantly improves the accuracy of removing transmission noise, and provides more powerful technical support for applications such as foggy image defogging and underwater image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of an application environment of an optional method for processing image data according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of an optional method for processing image data according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of an optional method for processing image data according to an embodiment of the present application;

[0023] Figure 4 is a schematic structural diagram of an optional image data processing device according to an embodiment of the present application;

[0024] Figure 5 is a schematic structural diagram of an optional image data processing product according to an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The present application is described below in conjunction with embodiments:

[0029] According to one aspect of an embodiment of the present application, a method for processing image data is provided. Optionally, in this embodiment, the method for processing image data can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1 As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, wherein the wired network includes, a local area network, a metropolitan area network and a wide area network; the wireless network includes, Bluetooth, WIFI and other networks that implement wireless communication; the terminal device 103 may be a terminal configured with an application, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal and other computer devices; the above-mentioned server may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0030] Combination Figure 1 As shown, the above-mentioned method for processing image data can be executed by an electronic device, which can be a terminal device or a server. The above-mentioned method for processing image data can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0031] The above is only an example and is not specifically limited in this embodiment.

[0032] Optionally, as an optional implementation, as Figure 2 As shown, the method for processing the above image data includes:

[0033] S202, acquiring a degraded image, wherein the degraded image represents an image with transmission noise;

[0034] Optionally, in an embodiment of the present application, the above-mentioned degraded images include but are not limited to foggy monitoring images, underwater exploration images, endoscopic medical images or any other images acquired under harsh lighting conditions, that is, the quality of the degraded images is impaired due to the intervention of scattering media such as fog, water or dust.

[0035] It should be noted that the sources of the above-mentioned degraded images are diverse, and may be directly from a camera or sensor, or may be generated through post-processing or simulation. The specific acquisition method is not limited to a certain method. For example, it may be obtained through software simulation of scattering effects, adding artificial noise, or through a specific image degradation process. This application does not limit this.

[0036] It should also be noted that the embodiments of the present application do not limit the types of the above-mentioned degraded images, which can include images in natural environments as well as images of special scenes such as industry and medical treatment. As long as the images are affected by transmission noise, they can be used as input and denoised using the methods in the embodiments of the present application.

[0037] In addition, there are many possibilities for the size, format, and resolution of the degraded image, which are also not restricted as long as they can be accepted by the target restoration model.

[0038] S204, inputting the degraded image into a target restoration model to obtain a clear image, wherein the target restoration model is obtained by training an initial restoration model using a first sample image, the initial restoration model comprises a background light extraction branch, a transmitted light extraction branch and a target output branch, the background light extraction branch is used to extract background light data of the first sample image, the transmitted light extraction branch is used to extract transmitted light data of the first sample image, the target output branch is used to remove transmission noise of the first sample image, generate initial image data, and remove transmission noise of a second sample image determined by the background light data, the transmitted light data and the initial image data, to obtain a clear sample image, the clear sample image is used to adjust model parameters of the initial restoration model, to obtain a target restoration model.

[0039] Optionally, in the embodiment of the present application, the above-mentioned target restoration model refers to an optimized deep learning model that can effectively restore a clear image from an image with transmission noise, including but not limited to a model based on a convolutional neural network (CNN), a generative adversarial network (GAN) or a self-supervised learning framework. The ability to remove transmission noise is improved by unsupervised learning of the intrinsic features of the degraded image, especially the relationship between the transmission light data, the background light data and the initial image data.

[0040] It should be noted that the above-mentioned first sample image can come from different scenes, including but not limited to foggy environment, underwater environment, dusty environment or any other imaging conditions with scattering effects, or be generated through post-processing or simulation, for example, through software simulation of scattering effects, adding artificial noise or through a specific image degradation process. This application is not limited to this.

[0041] Furthermore, the above-mentioned background light extraction branch can be used to perform a background light extraction operation on the first sample image to obtain background light data, the transmitted light extraction branch can be used to perform a transmitted light extraction operation on the first sample image to obtain transmitted light data, and the target output branch can be used to perform a direct transmission noise removal operation on the first sample image to obtain initial image data.

[0042] Among them, the above-mentioned background light extraction branch and target output branch can be composed of simple convolutional layers, Norm and activation function Sigmoid, while the transmitted light extraction branch can include DepthModel (pre-trained large model, depth extraction module) and Adapter (adapter module), which can be implemented by open source models such as DINO and DepthAnything. The initial transmitted light data can be determined based on the depth image data of the first sample image and the scattering coefficient, and then the adapter module is used to perform data augmentation operations on the initial transmitted light data, for example, adding noise, etc., so that the augmentation operation of the transmitted light data can be more flexibly adapted to different scattering environments, thereby enhancing the robustness and generalization ability of the target restoration model.

[0043] That is, the background light extraction branch is used to directly output the scattered light data of the first sample image, while the transmitted light extraction branch, through the combination of the depth extraction module and the adapter, not only utilizes the depth information to estimate the transmitted light, but also increases the flexibility and adaptability of the target restoration model through data augmentation operations, so that it can effectively restore the image in various scattering environments.

[0044] Optionally, in an embodiment of the present application, the above-mentioned initial image data refers to image data extracted directly from the first sample image using the target data branch during the training stage of the initial restoration model, and image data from which part of the transmission noise has been removed through a simplified convolution operation. In other words, the above-mentioned initial image data refers to image data obtained after the initial restoration model makes a preliminary denoising attempt on the first sample image in the absence of depth information and background light data guidance, reflecting the target restoration model's preliminary estimate of the removal of image transmission noise in the early stage of unsupervised training.

[0045] Similarly, the embodiments of the present application do not limit the specific calculation method of the background light data, transmitted light data and initial image data involved in the generation process of the above-mentioned second sample image. For example, the extraction of background light data is based on the statistics of the local brightest pixels, the noise addition of transmitted light data follows a specific Gaussian distribution, and the generation of initial image data can adopt a combination of multiple convolutional layers and activation functions. The present application also does not limit this to ensure the diversity and adaptability of the method.

[0046] Specifically, the second sample image can be determined according to the following formula:

[0047] I''(x)=J(x)T'(x)+(1-T'(x))A(x);

[0048] Wherein, I''(x) represents the second sample image, J(x) represents the initial image data, T'(x) represents the transmitted light data, A(x) represents the background light data, and the second sample image represents a re-degraded image obtained according to the initial image data, the transmitted light data and the background light data. At the end of the model training, the re-degraded image and the first sample image meet the similarity condition, for example, the similarity between the re-degraded image and the first sample image is as high as 98%, or the mean absolute error or mean square error of the pixel values ​​between the re-degraded image and the first sample image is calculated to ensure that the two are numerically close to a preset threshold, or the structural similarity index, peak signal-to-noise ratio, feature vector similarity, etc. are used to evaluate the similarity between the two.

[0049] Then, after obtaining the second sample image, the second sample image is input into the target output branch again, so as to remove the transmission noise of the second sample image by using the target output branch, thereby obtaining a noise-free, high-definition clear sample image. The clear sample image represents the training result of the target restoration model under the unsupervised learning framework, which is not only visually closer to the real state of the original scene, that is, a clear image taken without the interference of scattering media, but also performs better in terms of image structural information, color fidelity and detail retention.

[0050] It should also be noted that, in the embodiment of the present application, multiple loss functions at different levels can be established for joint calculation, and the model learning can be constrained from different angles, so that the final trained model can not only remove the scattering effect, but also maintain the naturalness and detail integrity of the image. Specifically:

[0051] S1, heavy degradation loss, that is, the second sample image reconstructed according to the atmospheric degradation scattering equation should be consistent with the first sample degraded image, so as to design the first loss function L reson To implement this constraint, the formula is as follows:

[0052] ; Wherein, I(x) represents the first sample image input, and I''(x) represents the second sample image.

[0053] S2, image secondary augmentation, based on the first processed image J(x) (the above initial image data), the regenerated clean image J'(x) (the above clear sample image), this component should be consistent, so as to design the above second loss function L sim To achieve this constraint, in addition, considering the need to avoid falling into a degenerate solution, sg, the gradient stop operation, can be used. The formula is as follows:

[0054] ; Wherein, J(x) represents the above-mentioned initial image data, and J'(x) represents the above-mentioned clear sample image.

[0055] S3, according to the power operation law of Transmission-Map, taking into account the gradient consistency loss, the third loss function L is designed tmap To implement this constraint, the formula is as follows:

[0056] L tmap = ; wherein, T(x) represents the initial transmitted light data directly processed by the above target output branch, and the corresponding T'(x) represents the above transmitted light data.

[0057] That is, the scattering coefficient is a constant, and the partial derivative of the transmitted light data with respect to the depth image data is a constant. After the initial restoration model training is completed, its second-order derivative is 0.

[0058] Then, the target loss function is determined to be L according to the first loss function, the second loss function and the third loss function. total =L sim +L reson +L tmap .

[0059] Furthermore, when the target loss function satisfies the preset loss value output condition, the initial restoration model training is successful. The preset loss value output condition here refers to the value of the target loss function reaching or falling below a preset threshold or standard during the model training process, which usually means that the model training has converged and the model parameters can effectively process the input data to achieve the expected performance indicators, such as at least one of the loss function threshold, the loss function change rate, the validation set performance or the number of iterations or the training time.

[0060] In an exemplary embodiment, during the unsupervised training process of the initial restoration model, the initial restoration model receives a first sample image, which also comes from a foggy environment. The initial restoration model generates a second sample image based on the first sample image. This generation process can be regarded as a preliminary defogging attempt by the model on the first sample image. The generated second sample image retains a structure similar to that of the first sample image, but may have deficiencies or deviations in removing transmission noise.

[0061] Next, in order to obtain a clear sample image, the initial restoration model further processes the second sample image and inputs the second sample image into the target output branch again to remove the transmission noise therein. The second sample image is generated based on the background light data, the transmission light data and the initial image data. Specifically, the background light data reflects the atmospheric background brightness in the first sample image, and the transmission light data is simulated by adding specific noise to the depth image data of the first sample image, while the initial image data is obtained by removing part of the transmission noise from the first sample image through a preliminary convolution operation. This processing provides preliminary guidance and clues for subsequent more in-depth defogging operations.

[0062] Through the above processing, the initial restoration model can finally generate a clear sample image, namely the second sample image. The second sample image is visually closer to the real scene in a fog-free state, and has been significantly improved in terms of structural information, color accuracy and detail retention.

[0063] After obtaining the clear sample image of the second sample image, the training loss value of the initial restoration model is calculated using the clear sample image of the second sample image. This loss value reflects the gap between the model's defogging effect and the ideal effect. According to this feedback, its parameters are adjusted to gradually improve the denoising performance, and eventually evolve into a target restoration model. It can effectively restore clear images from degraded images in foggy days or other scattering effect scenes, greatly improving the efficiency and quality of image processing, and has significant application value in traffic monitoring, environmental monitoring, outdoor shooting and other fields.

[0064] In an exemplary embodiment, taking the unsupervised image defogging scenario based on deep learning as an example, first, the system obtains a degraded image from a surveillance camera in a foggy environment, sends the input degraded image I(x) to the target restoration model, and directly processes it by the target output branch to output the above-mentioned clear image, that is, the final defogging or underwater enhanced image.

[0065] In other words, in actual application, the target restoration model can retain only one of the above-mentioned target output branches, while the background light extraction branch and the transmitted light extraction branch in the initial restoration model are only used in the model training stage. That is, in actual application, the target restoration model only retains the target output branch and omits the background light extraction branch and the transmitted light extraction branch. At this time, the data extraction capabilities of the background light extraction branch and the transmitted light extraction branch have been internalized into the target output branch during the training stage of the initial restoration model. Therefore, there is no need to retain the background light extraction branch and the transmitted light extraction branch in the target training model, thereby simplifying the calculation process of actual applications, improving processing speed, and reducing resource requirements.

[0066] Specifically, in the training stage of the initial restoration model, the background light extraction branch and the transmitted light extraction branch are used to establish interactive feedback with the target output branch, so as to facilitate the learning and generalization of the initial restoration model on various foggy images. Through different paths and loss functions, the initial restoration model can learn how to deal with the fog effect in the image without explicit supervision signals, so that the fog effect can be effectively removed for unknown foggy images in practical applications. Once the initial restoration model training is completed, the target output branch can independently process the degraded image and restore the clear image without explicitly using the background light extraction branch and the transmitted light extraction branch.

[0067] Therefore, in practical applications, the target restoration model includes a target output branch, while the background light extraction branch and the transmitted light extraction branch are optional, which optimizes resource usage and reduces computing delays, making the target restoration model more suitable for real-time processing or running on resource-constrained devices.

[0068] In general, in the embodiment of the present application, in the process of training the initial restoration model to determine the target restoration model, through mechanisms such as secondary augmentation and consistency loss, the target restoration model learns how to effectively remove the scattering effect between a given degraded image and component estimation, that is, the target output branch and the background light extraction branch and the transmitted light extraction branch are coordinated and optimized, and a clear image can be directly output. That is, by designing a reasonable unsupervised training process and loss function, the purpose of improving the accuracy of transmission noise removal is achieved, thereby achieving the technical effect of efficiently and accurately removing image transmission noise without pairing clean images, thereby solving the technical problem of low transmission noise removal accuracy in the field of image enhancement.

[0069] In summary, the target restoration model in the embodiment of the present application significantly improves the accuracy of removing transmission noise, and provides more powerful technical support for applications such as foggy image defogging and underwater image enhancement.

[0070] As an optional scheme, the method also includes: acquiring the above-mentioned first sample image; inputting the above-mentioned first sample image into the above-mentioned initial restoration model to obtain the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data, wherein the above-mentioned transmitted light data represents image data with noise added to the depth image data of the above-mentioned first sample image, the above-mentioned transmitted light data is used to indicate the light transmittance state of the above-mentioned first sample image, and the above-mentioned background light data is used to indicate the image brightness of the above-mentioned first sample image; generating a second sample image according to the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data; using the above-mentioned target output branch to remove the transmission noise of the above-mentioned second sample image to obtain the above-mentioned clear sample image, wherein the above-mentioned target output branch includes multiple convolutional layers; calculating the training loss value based on the above-mentioned clear sample image, and when the above-mentioned training loss value is less than or equal to the loss threshold, determining the above-mentioned initial restoration model as the above-mentioned target restoration model.

[0071] It should be noted that there are many methods for extracting the above-mentioned background light data, which can be based on statistical analysis of local maximum brightness values, or can be achieved through frequency domain transformation, edge detection or specific image preprocessing algorithms, and this application does not limit this.

[0072] Similarly, the generation of transmitted light data is not limited to adding Gaussian noise to the depth image data, but can also use various types of noise, or even simulated data based on physical models, to better reflect the light transmission characteristics of the real atmosphere or underwater environment. The generation strategy of the above initial image data is also diverse. In addition to the convolution operation, residual learning, jump connection or a specific denoising algorithm can also be used, which is also not limited in this application.

[0073] In an exemplary embodiment, a first sample image (which may be multiple sample images) is obtained from a traffic monitoring camera in foggy weather, and the first sample image is blurred due to fog scattering. The first sample image is then input into an initial restoration model based on a deep learning architecture. First, the background light data, transmitted light data, and initial image data of the first sample image are obtained. Based on these data, the initial restoration model generates a second sample image, that is, an image that contains transmitted light noise but has been preliminarily processed. Next, the target output branch in the initial restoration model processes the second sample image. This branch consists of multiple convolutional layers and is specifically used to remove transmitted noise. Through this processing, the initial restoration model generates a clear sample image, which is visually closer to the original fog-free state, and the clarity and details are significantly improved.

[0074] Based on the clear sample images, the training loss value of the initial restoration model can be calculated, which reflects the gap between the model denoising effect and the ideal effect. If the training loss value is less than or equal to the pre-set loss threshold, that is, the model's defogging effect has reached or exceeded the set performance standard, then the current initial restoration model can be determined as the final target restoration model, indicating that the initial restoration model has been optimized through the unsupervised training process and can achieve high-quality defogging effects on unseen foggy images.

[0075] Through the embodiments of the present application, an unsupervised training method is adopted to achieve the technical effect of effectively removing transmission noise from degraded images, thereby achieving the purpose of improving image clarity and information extraction capabilities. This technical solution not only simplifies the preparation process of training data, but also improves the generalization ability of the target restoration model in complex scattering environments. It has a significant improvement effect on various applications in the field of image processing, such as traffic monitoring, underwater exploration, medical image analysis, etc.

[0076] As an optional scheme, the above-mentioned generating a second sample image based on the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data includes: determining the product of the above-mentioned initial image data and the above-mentioned transmitted light data as the first data; determining the difference between the above-mentioned transmitted light data and a constant 1 and the product of the above-mentioned background light data as the second data; and determining the above-mentioned second sample image based on the above-mentioned first data and the above-mentioned second data.

[0077] Optionally, in the embodiment of the present application, the first data refers to the result data obtained after performing a pixel-by-pixel multiplication operation on the initial image data and the transmitted light data, which reflects the degradation of each pixel in the image under the influence of the scattering medium, including but not limited to performing a point multiplication operation on the preliminarily processed image and the transmitted light data generated by simulating the scattering environment to obtain a preliminary estimate of the degraded image.

[0078] It should be noted that the constant 1 in the embodiment of the present application may refer to a standardized all-one matrix, which is used to perform difference calculation with the transmitted light data. The generated data reflects the transmittance information. This operation is multiplied with the background light data when generating the second data, simulating the fog-free image information in the atmospheric scattering equation. The present application does not limit this. The constant 1 can be a standardized matrix of any form, or even a specific numerical matrix that matches the image size, to adapt to the scattering model and calculation requirements of different scenes.

[0079] In an exemplary embodiment, the depth image data of the first sample image is first estimated using a depth extraction module, and noise is added to the depth image data through a data augmentation operation to obtain transmitted light data. The initial image data is then multiplied by the transmitted light data to obtain the first data. At the same time, the difference between the transmitted light data and the constant 1 is calculated, and the result is multiplied by the background light data to generate the second data. Finally, the sum of the first data and the second data is determined as the second sample image. This image retains the original image structure while adding a simulated degradation effect based on depth information and a scattering model, providing a rich set of transmitted noise samples for unsupervised training.

[0080] Through the embodiment of the present application, the technical effect of reliably separating scattered light and transmitted light information from degraded images is achieved by multiplying image data with transmitted light data, and the loss function calculation in the unsupervised learning framework is optimized, thereby improving the accuracy and robustness of the model in image defogging and underwater image enhancement tasks. By introducing prior knowledge of depth information and scattering models, the problem of lack of clean image pairing as a supervisory signal in unsupervised training is effectively solved, providing the model with more reasonable training objectives and constraints, thereby improving the performance and quality of image restoration in complex scattering environments.

[0081] As an optional solution, the above method also includes: using the depth extraction module in the above transmitted light extraction branch to perform a depth information extraction operation on the above first sample image to obtain depth image data, and determining initial transmitted light data based on the above depth image data and the scattering coefficient; using the adapter module in the above transmitted light extraction branch to add Gaussian noise to the above initial transmitted light data to obtain the above transmitted light data.

[0082] Optionally, in the embodiment of the present application, the above-mentioned depth information extraction operation refers to extracting depth image data related to the scene depth from the first sample image using a depth estimation model, including but not limited to using a pre-trained depth estimation network, a three-dimensional reconstruction algorithm, or a depth perception technology based on structured light. These methods can provide key depth prior information for subsequent transmission noise removal.

[0083] Further, the initial transmitted light data is determined using this formula: ;

[0084] Among them, d(x) is the depth image data, and beta is the scattering coefficient (also known as the scattering intensity).

[0085] After the initial transmitted light data is obtained, it can be input into the above-mentioned adapter module to implement a data augmentation operation on the initial transmitted light data, so as to add Gaussian noise to the initial transmitted light data to obtain the transmitted light data.

[0086] It should be noted that data augmentation operations are not limited to power operations, but may also include scale transformation, rotation, shearing, color adjustment, or the introduction of other types of noise to increase the diversity and complexity of training data, which is not limited in this application.

[0087] At the same time, the number and type of convolutional layers of the transmitted light extraction branch can also be adjusted according to the specific application scenarios and model design requirements, including but not limited to using a deeper network structure, introducing hole convolutions, or combining attention mechanisms to improve the model's feature extraction capabilities and generalization performance.

[0088] In an exemplary embodiment, a pre-trained depth estimation model is first used to perform a depth information extraction operation on the acquired first sample image to obtain initial depth image data. Subsequently, Gaussian noise is added to the initial depth image data to generate depth image data, aiming to simulate the depth uncertainty in a real scattering environment. Next, the transmitted light extraction branch is used to perform a data augmentation operation on the depth image data, and the transmitted light data is obtained through a power operation. This data augmentation operation can simulate the influence of the scattering medium on the propagation of light, and provide a basis for the subsequent generation of the second sample image. The second sample image is generated based on the transmitted light data, background light data, and initial image data. The degradation degree of the second sample image is controllable, and the characteristics of the transmitted noise are clear, providing high-quality training samples for model training.

[0089] Through the embodiments of the present application, the technical effect of accurately estimating the transmitted light data from the degraded image is achieved by using depth information extraction and data augmentation operations, and the loss calculation of the model in the unsupervised training process is optimized, thereby improving the model's defogging and underwater image enhancement performance. By using the depth information of the sample image and the image augmentation method, the robustness and accuracy of the target restoration model in processing complex scattering scenes are effectively enhanced, opening up a new path for improving image processing effects.

[0090] As an optional solution, the method also includes: using the above-mentioned background light extraction branch to perform a Fourier transform operation on the above-mentioned first sample image to obtain the above-mentioned background light data, wherein the above-mentioned background light extraction branch includes multiple convolution layers; and generating a second sample image based on the background light data.

[0091] Optionally, in the embodiment of the present application, the background light extraction branch refers to a specific component in the model, which is used to process the first sample image and extract background light data therefrom, that is, brightness information in the image that is not affected by the scattering medium. This includes but is not limited to using a convolutional neural network to perform frequency domain conversion operations, analyzing the brightness distribution of the image based on Fourier transform, thereby obtaining background light data for subsequent generation of the second sample image.

[0092] It should be noted that the method for obtaining background light data in this application is not limited to Fourier transform operations, but can also be obtained through other frequency domain or spatial domain analysis techniques, such as wavelet transform, Laplace transform or direct color space analysis. This application does not limit this and aims to cover all calculation methods that can effectively evaluate the overall brightness of the image.

[0093] In addition, the number and type of convolutional layers of the background light extraction branch can also be adjusted according to actual needs to optimize the model's estimation accuracy of background light data.

[0094] In an exemplary embodiment, the first sample image is first processed using a background light extraction branch in an initial restoration model. This branch includes multiple convolutional layers, which can analyze the frequency domain features of the image based on Fourier transform operations to obtain image brightness information. Background light data is obtained through this processing, and the background light data reflects the brightness distribution of the image without the influence of scattering media.

[0095] Then, after obtaining the background light data, a second sample image is generated by combining the transmitted light data and the depth information background to simulate the atmospheric scattering or underwater scattering effect, so as to provide degraded image samples for model training, so that the model can learn the skill of removing transmitted noise under unsupervised conditions.

[0096] Through the embodiments of the present application, a background light extraction branch is used to process images, thereby achieving the technical effect of accurately estimating background light data from images in foggy or underwater environments, providing key prior information for the unsupervised training process, and improving the efficiency and effect of image dehazing and underwater image enhancement model training, thereby effectively enhancing the robustness and accuracy of the target restoration model.

[0097] As an optional scheme, the method also includes: establishing a target loss function based on the first sample image, the second sample image and the clear sample image; using the target loss function to determine the training loss value; when the training loss value is less than or equal to the loss threshold, determining the initial restoration model as the target restoration model.

[0098] Optionally, in the embodiment of the present application, the above-mentioned target loss function refers to a quantitative index used to evaluate the performance of the initial restoration model when processing the scattering effect image. It is not limited to a single loss metric, including but not limited to mean square error (MSE), structural similarity index (SSIM), perceptual loss or adversarial loss, etc., and guides the optimization process of the model by comprehensively analyzing the difference between the clear sample image output by the model and the first sample image and the second sample image.

[0099] It should be noted that the calculation method of the training loss value can be flexibly changed in this application, not limited to the above-mentioned loss function combination, but also other forms of error metrics or performance indicators, such as peak signal-to-noise ratio (PSNR), Jaccard similarity coefficient, or loss functions customized for specific tasks. The setting of the loss threshold also depends on the specific application scenario and the expected image restoration quality. It can be an empirical value or the best value determined by cross-validation, and this application does not limit this.

[0100] In an exemplary embodiment, first, based on a first sample image collected in a foggy or underwater environment, a second sample image is generated by simulating the scattering effect. Subsequently, the second sample image is processed using the initial restoration model to obtain a preliminary clear sample image. Next, the system constructs a target loss function containing multiple loss components based on the differences between the first sample image, the second sample image, and the clear sample image, such as simultaneously considering MSE and SSIM, to comprehensively evaluate the restoration effect of the model. During the model training process, the model parameters are continuously optimized, and the current performance of the model is measured by calculating the value of the target loss function, i.e., the training loss value. When the training loss value is less than or equal to the pre-set loss threshold, it indicates that the model has reached the expected performance standard. At this time, the initial restoration model is formally determined as a target restoration model that can effectively process scattering effect images.

[0101] Through the embodiments of the present application, a multi-image-based loss function evaluation method is adopted to achieve the technical effect of accurately training the initial restoration model under unsupervised conditions, thereby achieving the purpose of optimizing model performance, effectively removing or reducing scattering effects, and improving image clarity and information extraction capabilities.

[0102] As an optional scheme, a target loss function is established based on background light data, transmitted light data and initial image data, including at least one of the following: a first loss function is established based on the mean square error of image data of the first sample image and image data of the second sample image, wherein the target loss function includes the first loss function; a second loss function is established based on the image data of the clear sample image and the mean square error of image data obtained after a gradient stop operation is performed on the initial image data, wherein the target loss function includes the second loss function; a third loss function is established using partial derivatives of the initial transmitted light data, wherein the initial transmitted light data is determined by the depth image data and the scattering coefficient of the first sample image, and the depth image data is obtained by performing a depth information extraction operation on the first sample image by the depth extraction module in the transmitted light extraction branch, and the target loss function includes the third loss function.

[0103] Optionally, in an embodiment of the present application, the above-mentioned target loss function refers to a set of functions used to quantify and guide the accuracy of image restoration during model training, including but not limited to a first loss function, a second loss function, and a third loss function, which work together to optimize the image descattering effect.

[0104] Specifically, a first image vector corresponding to the first sample image and a second image vector corresponding to the second sample image are obtained; a first loss function is established using the square of the L2 norm of the first image vector and the second image vector, wherein the target loss function includes the first loss function; a third image vector corresponding to the clear sample image is obtained, and a gradient stop operation is performed on the initial image data to obtain a fourth image vector; a second loss function is established using the square of the L2 norm of the third image vector and the fourth image vector, wherein the target loss function includes the second loss function.

[0105] It should be noted that the square of the L2 norm is used as part of the loss function to calculate the distance between image vectors, thereby measuring the consistency of the model prediction results with the target image. In the present application, this consistency assessment is not limited to pixel-level comparisons, but can also be extended to the feature vector level, such as using feature vectors generated by a pre-trained feature extraction network to capture higher-level image structure information. In addition, the use of gradient stop operations and partial derivative constraints is intended to avoid information leakage during training and to ensure the rationality of the relationship between transmitted light data and depth information. These operations can have multiple implementation paths in practical applications, and this application does not limit this.

[0106] In an exemplary embodiment, when processing the first sample image, the initial depth image data is first obtained using the depth information extraction operation, and then the data augmentation operation is performed using the transmitted light extraction branch in the initial restoration model to obtain the initial transmitted light data. This process involves performing a power operation on the initial depth image data to adapt to the mathematical characteristics of the scattering model. Next, the first sample image is converted into a first image vector, and the second sample image is converted into a second image vector. By calculating the square of the L2 norm of these two vectors, a first loss function is constructed to evaluate the pixel-level difference between the second sample image generated by the model and the original image. Similarly, a feature representation of the clear sample image, i.e., a third image vector, is obtained, and a gradient stop operation is performed on the initial image data to generate a fourth image vector. Based on the L2 norm square of the third image vector and the fourth image vector, a second loss function is constructed to supervise the consistency between the clear image output by the model and the image estimated by the depth information. Finally, the system uses the partial derivatives of the initial transmitted light data and the depth image data to establish a third loss function to ensure that the estimation of the transmitted light data is consistent with the changing trend of the depth information and avoid estimation bias during the training process.

[0107] Through the embodiments of the present application, a set of loss functions including L2 norm square, gradient stop operation and partial derivative loss is adopted to realize effective training and optimization of the image descattering model under unsupervised conditions, thereby achieving the purpose of improving the image defogging and underwater image enhancement effects while ensuring the generalization ability and robustness of the model. By rationally constructing multi-dimensional loss constraints, the problems of unclear goals and insufficient constraints in unsupervised learning are effectively solved, providing strong technical support for image processing in complex scattering environments.

[0108] In an exemplary embodiment, the image data processing method proposed in the present application can be applied to the field of video image processing technology, for example, foggy image enhancement and underwater image enhancement scenarios related to scattering effects. Among them, image scattering effect removal generally refers to image defogging and underwater image enhancement, which have a wide range of application needs in the industry, such as traffic monitoring information tracing, endoscopic surgery image processing, marine scientific exploration, underwater search and rescue and other scenarios. The mainstream method uses supervised learning to construct synthetic clean images and degraded images to train neural networks. However, due to the domain difference between synthetic images and real images, the neural network obtained by supervised training has poor generalization and is not effective in practical applications.

[0109] In other words, for the existing technology that requires a large amount of paired data to directly use supervised learning methods, it is often difficult to collect enhanced images corresponding to underwater or foggy images of real scenes, and there are domain differences between existing public paired data sets and real scenes. Directly migrating and using models trained based on public data will result in performance degradation.

[0110] Based on this, the depth prior information of the pre-trained large model is used in the embodiment of the present application. In order to more reasonably use the depth prior information, an Adapter (adapter, the above-mentioned transmitted light extraction branch) is designed to estimate the Transmission Map (the above-mentioned transmitted light data). The power operation relationship between the depth prior and the Transmission Map is used to design a scattering coefficient loss number. The characteristics of the degradation equation are used to design a reasonable secondary augmented degraded image, construct a priori constraints, and consistency losses to train the initial restoration model to obtain the target restoration model. Specifically, the training process of the initial restoration model includes but is not limited to:

[0111] S1, send the sample degraded image I(x) (the above-mentioned first sample image) into three networks to obtain the global background light A(x) (the above-mentioned background light data), transmission T(x) (the above-mentioned transmission light data), and clean image J(x) (the above-mentioned initial image data).

[0112] It should be noted that after the training is completed, in the actual use stage of the target restoration model, the re-degraded image obtained according to these three components (that is, the degraded image I'' (x) generated according to A(x), T(x), and J(x)) should be consistent with the original degraded image I(x). The atmospheric scattering equation used for image dehazing and underwater image enhancement is expressed as follows:

[0113] I''(x)=J(x)T'(x)+(1-T'(x))A(x);

[0114] d(x)=DepthModel(x);

[0115] T(x)=Adapter(e -d(x) );

[0116] Among them, DepthModel can be a pre-trained large model (used to obtain depth image information, which can be a component module in the above-mentioned target restoration model), which can be implemented using open source models such as DINO and DepthAnything. Its output is an estimated depth map. According to the principle of T(x), there is a power operation process here, and an Adapter can be designed to complete the fine-tuned T(x) output.

[0117] S2, secondary augmentation of the image, is performed based on the T(x) component estimated by the initial restoration model, mainly by modifying T(x):

[0118] ;

[0119] That is, a tiny random noise is added to the power exponent to achieve the purpose of enlarging the transmission image while retaining the original information. The noise obeys a Gaussian distribution with a mean of 0 and an equation of 0.01. At the same time, a new degraded image is reconstructed according to A(x) and J(x). (the second sample image above), the formula is as follows:

[0120] I''(x)=J(x)T'(x)+(1-T'(x))A(x);

[0121] Next, Feed into J-Net (target output branch above) to get a new clean image estimate (Clear sample image above).

[0122] Specifically, the loss function design of the initial restoration model may include but is not limited to:

[0123] S1, heavy degradation loss, that is, according to the atmospheric degradation scattering equation, the image reconstructed by the component should be consistent with the original degraded image. The above first loss function L is designed reson To implement this constraint, the formula is expressed as: ;

[0124] S2, image secondary augmentation, using the first processed image J(x) (the above initial image data), regenerated clean image J'(x) (the above clear sample image), this component should be consistent, and the above second loss function L is designed sim To achieve this constraint, in addition, considering the need to avoid falling into a degenerate solution, sg, the gradient stop operation, can be used. The formula is as follows: ;

[0125] S3, according to the power operation law of Transmission-Map, taking into account the gradient consistency loss, the third loss function L is designed tmap To implement this constraint, the formula is as follows:

[0126] L tmap = ;

[0127] That is, the scattering coefficient is a constant, and the partial derivative of the transmitted light data with respect to the depth image data is a constant. After the initial restoration model training is completed, its second-order derivative is 0.

[0128] Furthermore, the target loss function is determined according to the first loss function, the second loss function and the third loss function as follows:

[0129] L total =L sim +L reson +L tmap ;

[0130] It should also be noted that the embodiment of the present application takes into account the principle of the atmospheric scattering equation, and the equation is as follows:

[0131] ;

[0132] Among them, I(x) is a degraded image, that is, a foggy image or an underwater image, J(x) is a clear image, that is, the variable that the defogging or underwater enhancement algorithm ultimately needs to solve, T(x) is the Transmission-Map (the above-mentioned transmission light data), and A(x) is the global background light (the above-mentioned background light data). Specifically, T(x) is defined as follows:

[0133] ;

[0134] Among them, d(x) is the depth image data, beta is the scattering coefficient (ie, scattering intensity), specifically, Figure 3 is a schematic diagram of an optional method for processing image data according to an embodiment of the present application, Figure 3 The J-Net (target output branch) and A-Net (background light extraction branch) in the above can be composed of simple convolution layers, Norm and activation function Sigmoid. The number of layers can be flexibly set, for example, set to any layer from 3 to 5 layers, for solving the clean image J(x). The image A-Net is used to solve the global background light A(x), which is composed of a frequency domain adaptation network. Adapter is used to output the transmission light data (Transmission-Map), which can include multiple convolution layers with a convolution kernel of 1x1. This application does not limit the size of the convolution kernel, and this is just an example.

[0135] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0137] According to another aspect of the embodiment of the present application, there is also provided an image data processing device for implementing the above-mentioned image data processing method. Figure 4 As shown, the device comprises:

[0138] An acquisition module 402 is used to acquire a degraded image, wherein the degraded image represents an image with transmission noise;

[0139] The processing module 404 is used to input the degraded image into the target restoration model to obtain a clear image, wherein the target restoration model is obtained by training the initial restoration model using the first sample image, and the initial restoration model includes a background light extraction branch, a transmitted light extraction branch and a target output branch, the background light extraction branch is used to extract the background light data of the first sample image, the transmitted light extraction branch is used to extract the transmitted light data of the first sample image, the target output branch is used to remove the transmission noise of the first sample image, generate the initial image data, and remove the transmission noise of the second sample image determined by the background light data, the transmitted light data and the initial image data to obtain a clear sample image, and the clear sample image is used to adjust the model parameters of the initial restoration model to obtain the target restoration model.

[0140] As an optional solution, the above-mentioned device is also used to: obtain the above-mentioned first sample image; input the above-mentioned first sample image into the above-mentioned initial restoration model to obtain the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data, wherein the above-mentioned transmitted light data represents image data with noise added to the depth image data of the above-mentioned first sample image, the above-mentioned transmitted light data is used to indicate the light transmittance state of the above-mentioned first sample image, and the above-mentioned background light data is used to indicate the image brightness of the above-mentioned first sample image; generate a second sample image according to the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data; use the above-mentioned target output branch to remove the transmission noise of the above-mentioned second sample image to obtain the above-mentioned clear sample image, wherein the above-mentioned target output branch includes multiple convolutional layers; calculate the training loss value based on the above-mentioned clear sample image, and when the above-mentioned training loss value is less than or equal to the loss threshold, determine the above-mentioned initial restoration model as the above-mentioned target restoration model.

[0141] As an optional solution, the above-mentioned device is used to generate a second sample image based on the above-mentioned background light data, the above-mentioned transmitted light data and the above-mentioned initial image data in the following manner: the product of the above-mentioned initial image data and the above-mentioned transmitted light data is determined as the first data; the difference between the above-mentioned transmitted light data and a constant 1 and the product of the above-mentioned background light data are determined as the second data; and the above-mentioned second sample image is determined based on the above-mentioned first data and the above-mentioned second data.

[0142] As an optional solution, the above-mentioned device is also used to: use the depth extraction module in the above-mentioned transmitted light extraction branch to perform a depth information extraction operation on the above-mentioned first sample image to obtain depth image data, and determine initial transmitted light data based on the above-mentioned depth image data and the scattering coefficient; use the adapter module in the above-mentioned transmitted light extraction branch to add Gaussian noise to the above-mentioned initial transmitted light data to obtain the above-mentioned transmitted light data; and generate the above-mentioned second sample image based on the above-mentioned transmitted light data.

[0143] As an optional solution, the above-mentioned device is also used to: use the above-mentioned background light extraction branch to perform a Fourier transform operation on the above-mentioned first sample image to obtain the above-mentioned background light data, wherein the above-mentioned background light extraction branch includes multiple convolution layers; and generate the above-mentioned second sample image based on the above-mentioned background light data.

[0144] As an optional scheme, the above-mentioned device is also used to: establish a target loss function based on the above-mentioned first sample image, the above-mentioned second sample image and the above-mentioned clear sample image; use the above-mentioned target loss function to determine the above-mentioned training loss value; when the above-mentioned training loss value is less than or equal to the loss threshold, determine the above-mentioned initial restoration model as the above-mentioned target restoration model.

[0145] As an optional scheme, the above-mentioned device is used to establish a target loss function based on background light data, transmitted light data and initial image data in at least one of the following ways: establishing a first loss function based on the mean square error of image data of the first sample image and image data of the second sample image, wherein the target loss function includes the first loss function; establishing a second loss function based on the image data of the clear sample image and the mean square error of image data obtained after performing a gradient stop operation on the initial image data, wherein the target loss function includes the second loss function; establishing a third loss function using partial derivatives of the initial transmitted light data, wherein the above-mentioned initial transmitted light data is determined by the depth image data and the scattering coefficient of the above-mentioned first sample image, and the above-mentioned depth image data is obtained by performing a depth information extraction operation on the above-mentioned first sample image by the depth extraction module in the above-mentioned transmitted light extraction branch, and the above-mentioned target loss function includes the above-mentioned third loss function.

[0146] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0147] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0149] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0150] Figure 5 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0151] It should be noted that Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Figure 5 As shown, the computer system 500 includes a central processing unit 501 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 502 (ROM) or the program loaded from the storage part 508 to the random access memory 503 (RAM). Various programs and data required for system operation are also stored in the random access memory 503. The central processing unit 501, the read-only memory 502 and the random access memory 503 are connected to each other through a bus 504. The input / output interface 505 (Input / Output interface, i.e., I / O interface) is also connected to the bus 504.

[0153] The following components are connected to the input / output interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0154] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processor 501, various functions defined in the system of the present application are executed.

[0155] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processor 501, various functions provided by the embodiment of the present application are performed.

[0156] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned image data processing method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments through the computer program.

[0157] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0158] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0159] Alternatively, a person skilled in the art may understand that: Figure 6 The structure shown is for illustration only. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations shown.

[0160] Among them, the memory 602 can be used to store software programs and modules, such as program instructions / modules corresponding to the image data processing method and device in the embodiment of the present application. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, that is, realizing the above-mentioned image data processing method. The memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include a memory remotely located relative to the processor 604, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 602 can be specifically, but not limited to, used to store information such as degraded images, clear images, background light data, transmitted light data, and initial image data. As an example, such as Figure 6 As shown, the memory 602 may include but is not limited to the acquisition module 402 and the processing module 404 in the image data processing device. In addition, it may also include but is not limited to other module units in the image data processing device, which will not be repeated in this example.

[0161] Optionally, the transmission device 606 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 606 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0162] In addition, the electronic device further includes: a display 608 for displaying the degraded image and the clear image; and a connection bus 610 for connecting various module components in the electronic device.

[0163] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0164] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the image data processing method provided in various optional implementations of the above-mentioned image data processing.

[0165] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0166] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0167] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0168] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0169] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0173] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing image data, characterized in that: include: Acquire a degraded image, wherein the degraded image represents an image with transmission noise; The degraded image is input into a target restoration model to obtain a clear image, wherein the target restoration model is obtained by training an initial restoration model using a first sample image, and the initial restoration model includes a background light extraction branch, a transmitted light extraction branch, and a target output branch, the background light extraction branch is used to extract background light data of the first sample image, the transmitted light extraction branch is used to extract transmitted light data of the first sample image, the target output branch is used to remove transmission noise of the first sample image, generate initial image data, and remove transmission noise of a second sample image determined by the background light data, the transmitted light data, and the initial image data to obtain a clear sample image, and the clear sample image is used to adjust model parameters of the initial restoration model to obtain the target restoration model.

2. The method according to claim 1, characterized in that The method further comprises: Acquire the first sample image; Inputting the first sample image into the initial restoration model to obtain the background light data, the transmitted light data and the initial image data, wherein the transmitted light data represents image data with noise added to the depth image data of the first sample image, the transmitted light data is used to indicate the light transmission state of the first sample image, and the background light data is used to indicate the image brightness of the first sample image; generating a second sample image according to the background light data, the transmitted light data and the initial image data; Using the target output branch to remove transmission noise of the second sample image to obtain the clear sample image, wherein the target output branch includes a plurality of convolutional layers; A training loss value is calculated based on the clear sample image, and when the training loss value is less than or equal to a loss threshold, the initial restoration model is determined as the target restoration model.

3. The method according to claim 2, characterized in that The step of generating a second sample image according to the background light data, the transmitted light data and the initial image data comprises: determining the product of the initial image data and the transmitted light data as first data; Determine the product of the difference between the transmitted light data and the constant 1 and the background light data as the second data; The second sample image is determined according to the first data and the second data.

4. The method according to claim 1, characterized in that: The method further comprises: Using the depth extraction module in the transmitted light extraction branch to perform a depth information extraction operation on the first sample image to obtain depth image data, and determining initial transmitted light data according to the depth image data and the scattering coefficient; Adding Gaussian noise to the initial transmitted light data using an adapter module in the transmitted light extraction branch to obtain the transmitted light data; The second sample image is generated based on the transmitted light data.

5. The method according to claim 1, characterized in that The method further comprises: Using the background light extraction branch to perform a Fourier transform operation on the first sample image to obtain the background light data, wherein the background light extraction branch includes a plurality of convolutional layers; The second sample image is generated based on the background light data.

6. The method according to claim 1, characterized in that The method further comprises: Establishing a target loss function based on the first sample image, the second sample image and the clear sample image; Determine the training loss value using the objective loss function; When the training loss value is less than or equal to a loss threshold, the initial restoration model is determined as the target restoration model.

7. The method according to claim 6, characterized in that The establishing of a target loss function based on the background light data, the transmitted light data and the initial image data comprises at least one of the following: Establishing a first loss function based on the mean square error of the image data of the first sample image and the image data of the second sample image, wherein the target loss function includes the first loss function; Establishing a second loss function based on the image data of the clear sample image and the mean square error of the image data obtained after performing a gradient stop operation on the initial image data, wherein the target loss function includes the second loss function; A third loss function is established using partial derivatives of initial transmitted light data, wherein the initial transmitted light data is determined by depth image data and a scattering coefficient of the first sample image, the depth image data is obtained by performing a depth information extraction operation on the first sample image by a depth extraction module in the transmitted light extraction branch, and the target loss function includes the third loss function.

8. An image data processing device, characterized in that: include: An acquisition module, configured to acquire a degraded image, wherein the degraded image represents an image with transmission noise; A processing module is used to input the degraded image into a target restoration model to obtain a clear image, wherein the target restoration model is obtained by training an initial restoration model using a first sample image, and the initial restoration model includes a background light extraction branch, a transmitted light extraction branch and a target output branch, the background light extraction branch is used to extract background light data of the first sample image, the transmitted light extraction branch is used to extract transmitted light data of the first sample image, the target output branch is used to remove transmission noise of the first sample image, generate initial image data, and remove transmission noise of a second sample image determined by the background light data, the transmitted light data and the initial image data to obtain a clear sample image, and the clear sample image is used to adjust model parameters of the initial restoration model to obtain the target restoration model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images