Depth image quality enhancement model training method, system, equipment and medium
By introducing cross-modal contrast loss function and contrast learning method in deep image quality enhancement model training, the problem of poor model training efficiency, accuracy and adaptability is solved, and the adaptability and generalization ability of the model in practical applications is improved.
Patent Information
- Application Number
- CN202411032733.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
AI Technical Summary
The existing deep image quality enhancement model training has problems such as expensive and time-consuming data requirements, poor generalization capabilities of the model, large computing resources requirements, and complex evaluation standards, resulting in poor training efficiency, accuracy and adaptability.
Using cross-modal contrast loss function to fine-tune the preset model through comparative learning, the data pair set is constructed to optimize the generator network, so that the distance of the aligned depth-visible light samples in the latent feature space is as small as possible, and the distance of the non-paired sample is as possible. Large, improve the adaptability and generalization ability of the model.
By introducing a cross-modal comparison loss function, the model is more specialized and adaptable in specific tasks, improving the training convergence speed and model accuracy, enhancing the representation ability and accuracy of deep information, and improving the adaptation in practical applications Sex and generalization ability.
Smart Images

Figure CN118967487A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a method, system, device and medium for training a deep image quality enhancement model. Background Art
[0002] At present, the technology of human behavior understanding mainly focuses on identifying human activities through videos captured by traditional visible light cameras. With the advancement of imaging technology, the recent emergence of depth sensors has brought great advantages to the task of human behavior recognition. Compared with conventional color frames in human activity recognition, depth maps are more robust to changing lighting conditions, which is of great help to systems working in dark environments. More importantly, depth sensors protect human privacy compared to visible light cameras, allowing the system to be deployed in some sensitive scenarios such as toilets, national institutions, etc. Typical deep image quality enhancement models use convolutional neural networks (CNNs) as the infrastructure. CNNs can effectively extract features from images and process inputs of different resolutions and noise levels through stacked convolution and pooling operations. The key to training deep image enhancement models is to design a suitable loss function. Commonly used loss functions include mean square error (MSE), perceptual loss (such as VGG loss), adversarial loss, etc. These loss functions help the model optimize the generated depth images to make them visually closer to real high-quality images.
[0003] Although deep image quality enhancement models have made significant progress in improving the quality of deep images, they still have some major shortcomings:
[0004] Data requirements: Building high-quality deep image pair datasets is very expensive and time-consuming. It is difficult to obtain large-scale data pairs in real-world scenarios, and these data need to be carefully calibrated and processed to ensure their quality and accuracy.
[0005] Model generalization ability: The trained deep image quality enhancement model may have poor generalization ability for unseen scenes or noise types. The model may be overly dependent on a specific data distribution or noise pattern during training, resulting in poor performance in practical applications.
[0006] Computing resource requirements: Training deep image quality enhancement models usually requires a lot of computing resources, especially GPU acceleration and large memory support. This limits the application of the model in actual embedded devices or edge computing environments.
[0007] Complexity of evaluation criteria: Evaluating the performance of deep image quality enhancement models not only relies on quantitative indicators (such as PSNR, SSIM), but also needs to consider their actual effects in specific application scenarios. This increases the complexity of model design and optimization.
[0008] It can be seen that there is an urgent need for a deep image quality enhancement model training method that can improve model training efficiency, accuracy and adaptability. Summary of the invention
[0009] In view of this, the embodiments of the present disclosure provide a deep image quality enhancement model training method, system, device and medium, which at least partially solve the problems of poor efficiency, accuracy and adaptability of training models in the prior art.
[0010] In a first aspect, an embodiment of the present disclosure provides a deep image quality enhancement model training method, comprising:
[0011] Step 1: construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning;
[0012] Step 2: Input the data set into the preset training to obtain the target model.
[0013] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:
[0014] A cross-modal contrastive loss function corresponding to the preset model is constructed, and the preset model is fine-tuned through contrastive learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, while the distance between the unpaired samples is as large as possible.
[0015] According to a specific implementation of the embodiment of the present disclosure, the expression of the cross-modal contrast loss function is:
[0016] contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)
[0017] Among them, sim is the similarity metric, τ is the temperature hyperparameter of contrastive learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.
[0018] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:
[0019] Step 2.1, taking the depth-visible light samples in the data pair set as input, start training the generator network of the preset model. During the training process, the generator optimizes the cross-modal contrast loss function through back propagation;
[0020] Step 2.2, during the training process, the similarity between the predicted image generated by the generator network and the true label of the corresponding input data is evaluated according to a preset period to obtain an evaluation result;
[0021] Step 2.3: According to the evaluation results, the cross-modal contrast loss function is adjusted and optimized until it meets the preset requirements to obtain the target model.
[0022] According to a specific implementation of the embodiment of the present disclosure, before step 1, the method further includes:
[0023] Use depth sensing to obtain a high-resolution depth image corresponding to the sample object, and use a visible light camera to obtain a visible light image corresponding to the sample object;
[0024] Reducing the resolution of the high-resolution depth image through a preset operation to obtain a low-resolution depth image;
[0025] The low-resolution depth image is temporally and spatially aligned with the corresponding visible light image and preprocessed to obtain a data pair set.
[0026] According to a specific implementation of the embodiment of the present disclosure, the step of preprocessing to obtain a data pair set includes:
[0027] The aligned low-resolution depth image and visible light image are rotated, scaled and affine transformed to obtain new low-resolution depth image and visible light image, and the aligned low-resolution depth image and visible light image are combined to form a data pair set.
[0028] According to a specific implementation manner of the embodiment of the present disclosure, the preset operation is any one of bilinear interpolation or Gaussian pyramid.
[0029] In a second aspect, the present disclosure provides a deep image quality enhancement model training system, including:
[0030] A construction module is used to construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning;
[0031] The training module is used to input the data set into the preset preset for training to obtain the target model.
[0032] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0033] at least one processor; and,
[0034] a memory communicatively connected to the at least one processor; wherein,
[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep image quality enhancement model training method in the aforementioned first aspect or any implementation of the first aspect.
[0036] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the deep image quality enhancement model training method in the aforementioned first aspect or any implementation of the first aspect.
[0037] In the fifth aspect, the embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the deep image quality enhancement model training method in the aforementioned first aspect or any implementation of the first aspect.
[0038] The deep image quality enhancement model training scheme in the embodiment of the present disclosure includes: step 1, constructing a cross-modal contrast loss function corresponding to a preset model, and fine-tuning the preset model through contrast learning; step 2, inputting the data pair set into the preset model for training to obtain the target model.
[0039] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, on the basis of the traditional loss function, the introduction of a cross-modal contrast loss function can help the model better learn the relationship between depth information and other related modalities, so as to improve the representation ability and accuracy of depth information; by fine-tuning the preset model through contrastive learning, it can be made more specialized and adaptable in specific tasks, thereby accelerating the training convergence speed and improving the model accuracy; the data pair set consists of low-quality depth images and corresponding high-quality depth images. Such a data set helps the model learn how to recover high-quality depth information from low-quality depth images, thereby improving the adaptability and generalization ability of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 A flowchart of a deep image quality enhancement model training method provided by an embodiment of the present disclosure;
[0042] Figure 2 A schematic diagram of the structure of a deep image quality enhancement model training system provided by an embodiment of the present disclosure;
[0043] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0044] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0045] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0046] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0047] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0048] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0049] The present disclosure provides a method for training a deep image quality enhancement model, which can be applied in an image processing process.
[0050] See also Figure 1 , is a flow chart of a method for training a deep image quality enhancement model provided by an embodiment of the present disclosure. Figure 1As shown, the method mainly comprises the following steps:
[0051] Step 1: construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning;
[0052] Furthermore, the step 1 specifically includes:
[0053] A cross-modal contrastive loss function corresponding to the preset model is constructed, and the preset model is fine-tuned through contrastive learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, while the distance between the unpaired samples is as large as possible.
[0054] Furthermore, the expression of the cross-modal contrast loss function is:
[0055] contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)
[0056] Among them, sim is the similarity metric, τ is the temperature hyperparameter of contrastive learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.
[0057] Furthermore, before step 1, the method further includes:
[0058] Use depth sensing to obtain a high-resolution depth image corresponding to the sample object, and use a visible light camera to obtain a visible light image corresponding to the sample object;
[0059] Reducing the resolution of the high-resolution depth image through a preset operation to obtain a low-resolution depth image;
[0060] The low-resolution depth image is temporally and spatially aligned with the corresponding visible light image and preprocessed to obtain a data pair set.
[0061] Furthermore, the step of preprocessing to obtain a data pair set includes:
[0062] The aligned low-resolution depth image and visible light image are rotated, scaled and affine transformed to obtain new low-resolution depth image and visible light image, and the aligned low-resolution depth image and visible light image are combined to form a data pair set.
[0063] Furthermore, the preset operation is any one of bilinear interpolation or Gaussian pyramid.
[0064] During the specific implementation, 1. Data collection stage:
[0065] Equipment and Setup:
[0066] Use a depth sensor (such as Intel Real Sense) and a visible light camera (such as an RGB camera).
[0067] Ensure that the depth sensor and visible light camera are fixedly installed to ensure that the collected depth image and visible light image have a corresponding relationship.
[0068] Collect data:
[0069] Collect high-resolution depth images and visible light images of the sample objects at the same time. Make sure that each depth image has a corresponding visible light image to build a data pair set.
[0070] 2. Generate low-resolution depth images:
[0071] Reduce resolution: For each high-resolution depth image, use bilinear interpolation or Gaussian pyramid method to reduce its resolution. For example, reduce the high-resolution depth image from 640x480 to 320x240.
[0072] 3. Time and Space Alignment:
[0073] Align depth images with visible light images: Use camera calibration techniques to ensure that each low-resolution depth image is aligned with the corresponding visible light image in time and space. This includes handling possible differences in camera pose or viewpoint offset.
[0074] 4. Geometric transformation:
[0075] Rotation, scaling, and affine transformation: Rotation, scaling, and affine transformations are applied to each pair of aligned low-resolution depth and visible light images. This simulates changes in viewpoint or camera pose that may occur when photographing the same object under different conditions.
[0076] 5. Construct data pair set:
[0077] Form a data pair set: Combine each low-resolution depth image after rotation, scaling, and affine transformation with the corresponding visible light image into a data pair. Ensure that each data pair maintains a corresponding relationship for subsequent training of the deep image enhancement model.
[0078] Example Dataset:
[0079] Suppose we have a set of sample objects, each captured by a depth sensor and a visible light camera at different angles and distances. Here is an example data pair set:
[0080] High-resolution depth and visible light images:
[0081] Object A: high-resolution depth image (640x480), visible light image (1920x1080);
[0082] Object B: high-resolution depth image (640x480), visible light image (1920x1080);
[0083] The low-resolution depth image after downscaling:
[0084] Object A: low-resolution depth image (320x240);
[0085] Object B: low-resolution depth image (320x240);
[0086] The data pair set after alignment and geometric transformation:
[0087] Object A: Low-resolution depth image (320x240) after rotation, scaling, and affine transformation, and the corresponding visible light image (1920x1080);
[0088] Object B: Low-resolution depth image (320x240) after rotation, scaling, and affine transformation, and the corresponding visible light image (1920x1080);
[0089] The data pairs generated in this way can be used to train deep learning models, for example for deep image enhancement or joint learning tasks between depth and RGB images.
[0090] In the specific implementation, assume that we have a set of aligned low-resolution depth data and visible light data, each data pair includes a depth image and a visible light image. Our goal is to use contrastive learning methods to fine-tune the preset model so that the aligned samples in the feature space are as close as possible, while the unpaired samples are as far apart as possible. Select a preset model, such as using a pre-trained convolutional neural network (CNN) as a feature extractor. Input each data pair into the model to obtain the feature representation of the low-resolution depth data and the visible light data.
[0091] Use back-propagation and stochastic gradient descent or other optimization algorithms to minimize the loss function, thereby fine-tuning the weights of the preset model. Ensure that during the training process, the loss function can effectively drive the feature representations of paired samples to be similar and the feature representations of unpaired samples to be far apart.
[0092] After fine-tuning, the model's performance on new samples is evaluated by checking the distance of the aligned depth-visible samples in feature space to ensure their similarity in feature space. This approach can be used for a variety of cross-modal learning tasks such as depth estimation and image enhancement.
[0093] Through the above steps, you can implement the application of cross-modal contrastive loss function and use contrastive learning method to fine-tune the model so that the distance between aligned depth-visible samples in feature space is as small as possible and the distance between unpaired samples is as large as possible.
[0094] Optionally, when building a preset model, if you do not use the cross-modal contrast loss function, there are some alternatives that you can consider for fine-tuning and optimization:
[0095] The following are some alternatives:
[0096] CycleGAN or Pix2Pix framework:
[0097] Use CycleGAN or Pix2Pix models, which are specially designed to handle cross-modal image translation problems, such as deep image to visible light image translation. They maintain the consistency between input and output through adversarial training and cycle consistency loss without the need for additional contrastive loss functions.
[0098] Methods based on reconstruction loss:
[0099] The generator network can be trained using only the reconstruction loss. The reconstruction loss is usually used to measure the pixel-level difference between the generated image and the real image, such as the mean square error (MSE) or the structural similarity index (SSIM). By minimizing the reconstruction loss, the generator can gradually improve the quality of the generated images.
[0100] Content loss and style loss:
[0101] The quality of the generated image can be optimized by combining content loss (such as feature matching loss) and style loss. Content loss ensures that the generated image is similar to the real image in feature space, while style loss keeps the style and texture features of the generated image consistent with the real image.
[0102] Supervised learning methods:
[0103] A supervised learning approach can be used, where the generator network is trained in a supervised manner, such as using low-resolution depth images and high-quality visible light images with true labels. This approach directly optimizes the generator network so that it can accurately generate images of the target modality.
[0104] Comprehensive consideration:
[0105] When choosing an alternative, consider the following:
[0106] Task complexity and data availability: The cross-modal contrastive loss function is suitable for tasks that require accurate modeling of the relationship between modalities, especially when the training data is limited or there are significant differences between modalities.
[0107] Model implementation and computational efficiency: Some methods may be more feasible in implementation and computational efficiency, especially for large-scale data and complex models.
[0108] Impact of loss function on generation results: Different loss functions may affect the features learned by the generator and the quality of the generated images, so they need to be selected and optimized according to specific tasks and evaluation criteria.
[0109] In summary, according to the specific task requirements and data characteristics, suitable alternatives can be selected for model fine-tuning and optimization to improve the performance of the generator network and the quality of the generated results.
[0110] Step 2: Input the data set into the preset training to obtain the target model.
[0111] Based on the above embodiment, step 2 specifically includes:
[0112] Step 2.1, taking the depth-visible light samples in the data pair set as input, start training the generator network of the preset model. During the training process, the generator optimizes the cross-modal contrast loss function through back propagation;
[0113] Step 2.2, during the training process, the similarity between the predicted image generated by the generator network and the true label of the corresponding input data is evaluated according to a preset period to obtain an evaluation result;
[0114] Step 2.3: According to the evaluation results, the cross-modal contrast loss function is adjusted and optimized until it meets the preset requirements to obtain the target model.
[0115] In the specific implementation, suppose we have a task to generate high-quality visible light images from low-resolution depth images. We use a generative adversarial network (GAN) as the preset model, which includes a generator network and a discriminator network. We hope to optimize the generator network through a cross-modal contrast loss function so that it can effectively generate realistic visible light images.
[0116] Step 2: Train the generator network of the preset model
[0117] Step 2.1: Train the Generator Network
[0118] We prepare paired depth-visible image data from the dataset:
[0119] ((\mathbf{I}^d_i,\mathbf{I}^v_i)) represents the (i)th pair of low-resolution depth image (\mathbf{I}^d_i) and the corresponding visible light image (\mathbf{I}^v_i).
[0120] We use a generative adversarial network (GAN), where the goal of the generator (G) is to convert the input low-resolution depth image (\mathbf{I}^d_i) into a realistic high-resolution visible light image (\hat{\mathbf{I}}^v_i). During the training of the generator, we simultaneously optimize the cross-modal contrastive loss function.
[0121] Step 2.2: Evaluate the similarity between the generated predicted images and the true labels
[0122] During training, we periodically evaluate the similarity of images generated by the generator network to their corresponding real-light images.
[0123] The specific evaluation method can be to calculate the difference at the pixel level or use indicators such as the structural similarity index (SSIM) to measure the similarity between the generated image and the real image. We use these evaluation results to adjust and optimize the cross-modal contrast loss function.
[0124] Step 2.3: Adjust and optimize the cross-modal contrastive loss function
[0125] Based on the evaluation results, we can adjust the temperature hyperparameter τ or margin (m) of the cross-modal contrastive loss function to optimize the training process of the generator. For example, if the generated image has a low similarity to the real label, we can increase (m) to encourage the generator to generate results closer to the real image.
[0126] Example summary:
[0127] Suppose at some point in the training, we evaluate the structural similarity index (SSIM) between the images generated by the generator and the real images to be 0.75. Based on this evaluation result, we find that the quality of the generated images can be further improved. We can adjust the parameters of the cross-modal contrast loss function, such as increasing (m), or redesign the similarity metric function to improve the performance of the generator network.
[0128] Through these steps, we can gradually optimize the generator network so that it can effectively generate high-quality visible light images from low-resolution depth images, while using the cross-modal contrastive loss function to help the network learn the alignment relationship between corresponding modalities.
[0129] Optionally, in step 2, for the cross-modal contrastive loss function to be adjusted and optimized, some alternatives can be considered, especially for the training and evaluation process of the generator network. These alternatives can be selected based on the task requirements and the characteristics of the available data:
[0130] Alternatives may include the following:
[0131] Methods based on reconstruction loss:
[0132] Use reconstruction loss (such as mean squared error or structural similarity index) as the main optimization objective. Train the generator network by minimizing the pixel-level difference between the generated image and the target image without relying on an additional contrastive loss function.
[0133] Cycle consistency loss:
[0134] Introducing a cycle consistency loss, especially when the task involves cyclic or recursive generation. This loss ensures that the generator maintains consistency between generation and reconstruction, helping to reduce information loss between modalities.
[0135] Content loss and style loss:
[0136] Combining content loss (such as feature matching loss) and style loss to optimize the quality of generated images. Content loss ensures that the generated images are similar to the real images in feature space, while style loss keeps the style and texture features of the generated images consistent with the real images.
[0137] Supervised learning methods:
[0138] If labeled data is available, supervised learning methods can be used to directly train the generator network. This approach is suitable for directly optimizing the generator so that it can accurately generate images of the target modality without relying on complex contrastive loss functions.
[0139] Comprehensive consideration:
[0140] When choosing an alternative, consider the following:
[0141] Task complexity and data availability: Alternatives should take into account the availability of training data and the degree of differences between modalities.
[0142] Model implementation and computational efficiency: Different loss functions and training methods may have different effects on model implementation and computational efficiency.
[0143] Impact of loss function on generation results: Different loss functions will affect the features learned by the generator and the quality of the generated images, so they need to be selected and optimized according to specific tasks and evaluation criteria.
[0144] In summary, according to the specific task requirements and data characteristics, suitable alternatives can be selected to replace or supplement the cross-modal contrast loss function adjustment and optimization process in step 2 to achieve the goal of optimizing the generator network and improving the quality of generated images.
[0145] The depth image quality enhancement model training method provided in this embodiment can help the model better learn the relationship between depth information and other related modalities by introducing a cross-modal contrast loss function on the basis of the traditional loss function, so as to improve the representation ability and accuracy of the depth information; by fine-tuning the preset model through contrast learning, it can make it more specialized and adaptable in specific tasks, thereby accelerating the training convergence speed and improving the model accuracy; the data pair set consists of low-quality depth images and corresponding high-quality depth images. Such a data set helps the model learn how to recover high-quality depth information from low-quality depth images, thereby improving the adaptability and generalization ability of the model in practical applications.
[0146] Corresponding to the above method embodiment, see Figure 2 The present disclosure also provides a deep image quality enhancement model training system 20, including:
[0147] A construction module 201 is used to construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrast learning;
[0148] The training module 202 is used to input the data set into a preset training mode to obtain a target model.
[0149] Figure 2 The system shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.
[0150] See also Figure 3 The embodiment of the present disclosure further provides an electronic device 30, which includes: at least one processor and a memory connected to the at least one processor. The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the deep image quality enhancement model training method in the aforementioned method embodiment.
[0151] The embodiments of the present disclosure also provide a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the deep image quality enhancement model training method in the aforementioned method embodiment.
[0152] The embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the deep image quality enhancement model training method in the aforementioned method embodiment.
[0153] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 30 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0154] like Figure 3 As shown, the electronic device 30 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 30 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0155] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication device 309 may allow the electronic device 30 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 30 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.
[0156] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0157] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0158] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0159] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can execute the relevant steps of the method embodiment.
[0160] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the method embodiment.
[0161] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0163] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware.
[0164] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
[0165] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A deep image quality enhancement model training method, characterized in that: include: Step 1: construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning; Step 2: Input the data set into the preset training to obtain the target model.
2. The method according to claim 1, characterized in that , the step 1 specifically includes: A cross-modal contrastive loss function corresponding to the preset model is constructed, and the preset model is fine-tuned through contrastive learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, while the distance between the unpaired samples is as large as possible.
3. The method according to claim 2, characterized in that ,The expression of the cross-modal contrast loss function is contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)where, sim is the similarity metric, τ is the temperature hyperparameter for contrastive learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.
4. The method according to claim 3, characterized in that , the step 2 specifically includes: Step 2.1, taking the depth-visible light samples in the data pair set as input, start training the generator network of the preset model. During the training process, the generator optimizes the cross-modal contrast loss function through back propagation; Step 2.2, during the training process, the similarity between the predicted image generated by the generator network and the true label of the corresponding input data is evaluated according to a preset period to obtain an evaluation result; Step 2.3: According to the evaluation results, the cross-modal contrast loss function is adjusted and optimized until it meets the preset requirements to obtain the target model.
5. The method according to claim 1, characterized in that , before step 1, the method further includes: Use depth sensing to obtain a high-resolution depth image corresponding to the sample object, and use a visible light camera to obtain a visible light image corresponding to the sample object; Reducing the resolution of the high-resolution depth image through a preset operation to obtain a low-resolution depth image; The low-resolution depth image is temporally and spatially aligned with the corresponding visible light image and preprocessed to obtain a data pair set.
6. The method according to claim 5, characterized in that The step of preprocessing it to obtain a data pair set includes: The aligned low-resolution depth image and visible light image are rotated, scaled and affine transformed to obtain new low-resolution depth image and visible light image, and the aligned low-resolution depth image and visible light image are combined to form a data pair set.
7. The method according to claim 5, characterized in that ,The preset operation is either bilinear interpolation or Gaussian pyramid.
8. A deep image quality enhancement model training system, characterized in that: include: A construction module is used to construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning; The training module is used to input the data set into the preset preset for training to obtain the target model.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the deep image quality enhancement model training method described in any one of the preceding claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the deep image quality enhancement model training method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Cross-modal super-resolution reconstruction method, terminal equipment and readable storage medium
CN117611445A
Method and system for generating ultraviolet light image based on visible light image and application
CN117726701A
Image text generation method and device based on deep learning
CN118397147A
Cited By
Multi-source and multi-physics field gravity data super-resolution reconstruction method
CN120634864A
A multi-source, multi-physical-field gravity data super-resolution reconstruction method
CN120634864B
