Depth image quality enhancement method, system, equipment and medium

Abstract

The embodiment of the invention provides a depth image quality enhancement method, system and device and a medium, and belongs to the technical field of image processing, and the method specifically comprises the steps: 1, aligning an initial depth data set with a corresponding visible light data set, and forming a data pair set of depth data and visible light data; and step 2, obtaining a target depth image after quality enhancement according to the data pair set and a preset model. Through the scheme disclosed by the invention, the hardware cost of system deployment is greatly reduced, the dependence on large-scale high-quality annotation data is avoided, the workload of data acquisition and annotation is greatly reduced, and the reconstruction efficiency, accuracy and adaptability of the low-resolution depth image are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a method, system, device and medium for enhancing the quality of deep images. Background Art

[0002] At present, the technology of human behavior understanding mainly focuses on identifying human activities through videos captured by traditional visible light cameras. With the advancement of imaging technology, the recent emergence of depth sensors has brought great advantages to the task of human behavior recognition. Compared with conventional color frames in human activity recognition, depth maps are more robust to changing lighting conditions, which is of great help for systems working in dark environments. More importantly, depth sensors protect human privacy compared to visible light cameras, allowing the system to be deployed in some sensitive scenarios such as toilets, national institutions, etc. There are many forms of representation of human behavior understanding in depth sensor data, including: the most important methods are skeleton joints and point clouds. The mainstream skeleton-based method uses joint differences to capture the activity clues of static posture, continuous motion and overall dynamics of people. There are also methods that use the "moving posture" of joint velocity, configuration and acceleration. Compared with skeleton joints, point clouds are more robust to noise and occlusion. The mainstream method uses spatiotemporal occupancy patterns to divide into spatiotemporal units and calculate the corresponding occupancy values ​​to describe the depth appearance.

[0003] Existing methods that directly use methods based on 3D skeleton modeling have poor effects in multi-person scenarios, because low-resolution depth data is prone to occlusion and precision problems, and cannot accurately restore the skeleton information of each person. Although methods based on depth image features do not rely on accurate 3D skeleton data, the features in low-resolution data are still relatively rough, the recognition accuracy is low, and it is also easily affected by environmental factors. In addition, low-cost depth sensors themselves have low resolution, limited measurement accuracy, and large data noise, which further aggravates the bottleneck of algorithm performance.

[0004] It can be seen that there is an urgent need for a deep image quality enhancement method with high reconstruction efficiency, accuracy and adaptability. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a method, system, device and medium for enhancing the quality of deep images, which at least partially solve the problems of poor reconstruction efficiency, accuracy and adaptability in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for enhancing deep image quality, including:

[0007] Step 1: aligning the initial depth data set with the corresponding visible light data set to form a data pair set of depth data and visible light data;

[0008] Step 2: Obtain a quality-enhanced target depth image based on the data pair set and the preset model.

[0009] According to a specific implementation of the embodiment of the present disclosure, before step 1, the method further includes:

[0010] A low-resolution depth dataset containing various human behaviors is collected as the initial depth dataset.

[0011] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:

[0012] Step 1.1, normalize the low-resolution depth dataset and the visible light dataset:

[0013] Step 1.2, using feature point matching method to align the normalized low-resolution depth dataset and visible light dataset in time and space;

[0014] Step 1.3, using a pre-trained multimodal encoder to map the data in the low-resolution depth dataset and the visible light dataset into a shared feature embedding space, and establish a connection between the low-resolution depth dataset and the visible light dataset;

[0015] Step 1.4, forming a data pair set of depth-visible light samples.

[0016] According to a specific implementation of the embodiment of the present disclosure, step 1.2 specifically includes:

[0017] Step 1.2.1, extract feature points from the low-resolution depth dataset and the visible light dataset respectively;

[0018] Step 1.2.2, calculate the feature vector for each feature point;

[0019] Step 1.2.3, use a feature matching algorithm to match the feature points in the low-resolution depth dataset and the visible light dataset;

[0020] In step 1.2.4, the matched feature points are used to perform geometric correction on the registered data in the low-resolution depth dataset and the visible light dataset.

[0021] According to a specific implementation of the embodiment of the present disclosure, step 1.3 specifically includes:

[0022] Step 1.3.1, using the Transformer model as the base, through end-to-end training, learn the mapping function from visible light features and deep features to the shared feature embedding space, where the mapping function is

[0023] zdepth=f(xdepth,xrgb)

[0024] Among them, xdepth is a low-resolution depth dataset, xrgb is a visible light dataset, and f(·) is a cross-modal mapping function;

[0025] Step 1.3.2, use the mapping function to establish the connection between the low-resolution depth dataset and the visible light dataset.

[0026] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0027] Step 2.1, construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning;

[0028] Step 2.2, input the data pair set into the cross-modal generator for training;

[0029] Step 2.3, collect the low-resolution depth image corresponding to the target human body and input it into the trained cross-modal generator to generate a high-resolution target depth image, where the expression of the target depth image is bf{z}_{depth}=G(xdepth|xrgb)

[0030] Among them, G represents the cross-modal generator.

[0031] According to a specific implementation of the embodiment of the present disclosure, step 2.1 specifically includes:

[0032] Construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrast learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, and the distance between the unpaired samples is as large as possible, where the expression of the cross-modal contrast loss function is:

[0033] contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)

[0034] Among them, sim is the similarity metric, τ is the temperature hyperparameter of contrastive learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.

[0035] In a second aspect, an embodiment of the present disclosure provides a deep image quality enhancement system, including:

[0036] An alignment module, used to align an initial depth data set with a corresponding visible light data set to form a data pair set of depth data and visible light data;

[0037] The enhancement module is used to obtain a target depth image after quality enhancement according to a data pair set and a preset model.

[0038] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0039] at least one processor; and,

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the depth image quality enhancement method in the aforementioned first aspect or any implementation of the first aspect.

[0042] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the depth image quality enhancement method in the aforementioned first aspect or any implementation of the first aspect.

[0043] In the fifth aspect, the embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the depth image quality enhancement method in the aforementioned first aspect or any implementation of the first aspect.

[0044] The depth image quality enhancement scheme in the embodiment of the present disclosure includes: step 1, aligning the initial depth data set with the corresponding visible light data set to form a data pair set of depth data and visible light data; step 2, obtaining the target depth image after quality enhancement based on the data pair set and the preset model.

[0045] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the capabilities of the large basic model are utilized to achieve cross-modal integration of two heterogeneous data modalities, low-resolution depth sensor and visible light data, into a unified embedding space, so as to solve the problems of scarcity of low-resolution depth sensor data and high difficulty in understanding. The inherent correlation properties of the image are utilized to align the embeddings of the two modalities with the image embedding, and then the aligned depth-visible light samples are fine-tuned through contrast learning. This cross-modal alignment method can produce powerful zero-sample behavior, automatically associate modal combinations that have never appeared in the training data, and improve the reconstruction efficiency, accuracy and adaptability of low-resolution depth images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of a flow chart of a method for enhancing the quality of a deep image provided by an embodiment of the present disclosure;

[0048] Figure 2 A low-resolution depth image quality enhancement graph of a depth image quality enhancement method provided by an embodiment of the present disclosure;

[0049] Figure 3 A schematic diagram of the structure of a deep image quality enhancement system provided by an embodiment of the present disclosure;

[0050] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0052] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0053] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0054] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0055] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0056] There are some key challenges and difficulties when using low-resolution depth sensors for human activity recognition. The most important of these is that in multi-person scenarios, the direct use of 3D skeleton modeling methods is less effective. This is because low-resolution depth data is prone to occlusion and accuracy issues, and cannot accurately restore each person's skeleton information. Another difficulty is that the performance of human activity recognition algorithms is easily affected by factors such as installation angle and ambient lighting, and a large amount of data from different scenes needs to be collected to train a more robust model. This not only requires a lot of data collection work, but also increases the difficulty of modeling. How to use low-cost depth sensors in complex practical environments to achieve accurate and effective human activity recognition is an important issue worthy of further exploration.

[0057] The disclosed embodiments provide a method for enhancing the quality of a deep image, which can be applied to human behavior recognition in scenarios such as smart homes, medical health, and industrial monitoring.

[0058] See also Figure 1 , is a flow chart of a method for enhancing the quality of a deep image provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:

[0059] Step 1: aligning the initial depth data set with the corresponding visible light data set to form a data pair set of depth data and visible light data;

[0060] Optionally, before step 1, the method further includes:

[0061] A low-resolution depth dataset containing various human behaviors is collected as the initial depth dataset.

[0062] Optionally, the low-resolution depth dataset of multiple human behaviors includes low-resolution depth images corresponding to sample person actions of checking a watch, crossing arms, sitting, standing up, turning around, falling, lying down, walking, waving, punching, kicking, throwing and picking up.

[0063] Furthermore, the step 1 specifically includes:

[0064] Step 1.1, normalize the low-resolution depth dataset and the visible light dataset:

[0065] Step 1.2, using feature point matching method to align the normalized low-resolution depth dataset and visible light dataset in time and space;

[0066] Step 1.3, using a pre-trained multimodal encoder to map the data in the low-resolution depth dataset and the visible light dataset into a shared feature embedding space, and establish a connection between the low-resolution depth dataset and the visible light dataset;

[0067] Step 1.4, forming a data pair set of depth-visible light samples.

[0068] Furthermore, the step 1.2 specifically includes:

[0069] Step 1.2.1, extract feature points from the low-resolution depth dataset and the visible light dataset respectively;

[0070] Step 1.2.2, calculate the feature vector for each feature point;

[0071] Step 1.2.3, use a feature matching algorithm to match the feature points in the low-resolution depth dataset and the visible light dataset;

[0072] In step 1.2.4, the matched feature points are used to perform geometric correction on the registered data in the low-resolution depth dataset and the visible light dataset.

[0073] Furthermore, the step 1.3 specifically includes:

[0074] Step 1.3.1, using the Transformer model as the base, through end-to-end training, learn the mapping function from visible light features and deep features to the shared feature embedding space, where the mapping function is

[0075] zdepth=f(xdepth,xrgb)

[0076] Among them, xdepth is a low-resolution depth dataset, xrgb is a visible light dataset, and f(·) is a cross-modal mapping function;

[0077] Step 1.3.2, use the mapping function to establish the connection between the low-resolution depth dataset and the visible light dataset.

[0078] In specific implementation, a low-resolution depth dataset containing a variety of human behaviors is collected, including sample personnel checking watches, crossing arms, sitting, standing up, turning around, falling, lying, walking, waving, punching, kicking, throwing, and picking up actions as follows:

[0079] 1) Determine the source and acquisition method of the dataset:

[0080] First, we need to identify suitable sources that can provide these motion data, which may be through existing public datasets, datasets from scientific research institutions, or through self-collection.

[0081] 2) Data acquisition equipment preparation:

[0082] Use a suitable depth sensor for data collection. Common choices include Microsoft Kinect, Intel Real Sense, or other depth cameras. These devices can obtain human posture and motion information at low cost and high efficiency.

[0083] 3) Scene setting:

[0084] During data collection, ensure that the scene can support the required variety of actions. For example, a spacious space is required to accommodate the execution of various behavioral actions and ensure that the depth sensor can capture the action details well.

[0085] 4) Data collection process:

[0086] Each action should be recorded in a predetermined order. For example, the sample person checking their watch might be recorded first, followed by crossing their arms, and so on. Multiple instances of each action should be recorded to cover different execution styles and variations.

[0087] 5) Data recording and processing:

[0088] A depth camera is used to record a sequence of low-resolution depth images for each action. The depth image captures the three-dimensional information of the human body in the scene, which is very useful for subsequent action analysis.

[0089] 6) Labeling and data organization:

[0090] The recorded depth image data is annotated to mark the start and end frames of each action, as well as the label of the action type. These annotations will help train machine learning models to recognize and classify different actions.

[0091] 7) Dataset validation and preparation for release:

[0092] After the data is recorded and annotated, the dataset is verified and organized to ensure the integrity and quality of the data for each action. This step also includes the documentation of the dataset and preparing it for publication or use in research.

[0093] Through the above steps, low-resolution depth datasets containing a variety of human behaviors can be collected and organized. These datasets are of great practical significance for developing and evaluating human action recognition algorithms, posture estimation systems, and augmented reality / virtual reality applications.

[0094] Optionally, for some actions, image processing and data augmentation techniques can be used to generate simulated low-resolution depth images. For example, a high-resolution depth image can be blurred, noise can be added, or pixel resolution can be reduced to simulate the capture effect of a low-quality depth sensor.

[0095] Alternatively, low-resolution depth images can be generated using generative adversarial networks (GANs) or other deep learning generative models. This approach can overcome the limitations of sample data collection and processing by training the model to learn how to generate low-resolution versions from high-resolution depth images.

[0096] In the specific implementation, for the low-resolution depth dataset and the visible light dataset, we first perform spatial and temporal registration to ensure that the data of the two modalities can correspond. The technical means of feature point matching is adopted. In addition, the two data need to be normalized to make their feature distribution closer, which is convenient for subsequent cross-modal learning. We use a pre-trained multimodal encoder to map the low-resolution depth data and visible light data to a shared feature embedding space. Specifically, using the Transformer model as the base, through end-to-end training, we learn the mapping function from visual and depth features to a unified embedding representation. We obtain a unified embedding representation space, in which the low-resolution depth data and visible light data present highly correlated feature distributions. Specifically, it can be regarded as a nonlinear feature mapping or correction relationship, which can compensate and correct the details that are difficult to capture in the low-resolution depth data by associating it with the visible light data. The low-resolution depth dataset is aligned with the preset visible light dataset, and the connection between the low-resolution depth dataset and the visible light dataset is established using the large basic model to form the specific process of depth-visible light samples. The following can be shown:

[0097] Step 1.1: Normalize the depth dataset and visible light dataset;

[0098] Suppose we have a low-resolution depth dataset and a corresponding high-resolution visible light dataset. First, we need to normalize them to ensure that they are aligned in time and space.

[0099] Low-resolution depth datasets: These data usually exist in the form of grayscale images, and the pixel values ​​represent the depth information.

[0100] Visible light dataset: These data are RGB images with color information for each pixel.

[0101] Normalization can include the following steps:

[0102] For depth datasets, calibration and scaling may be required based on the characteristics of the acquisition device to ensure consistency and comparability of depth values.

[0103] For visible light datasets, color space standardization is usually required, such as conversion to Lab color space or HSV color space, to make the color information more consistent and comparable.

[0104] Step 1.2: feature point matching method for time and space alignment;

[0105] After normalization, we use a feature point matching method to align the low-resolution depth dataset with the visible light dataset in time and space. This can be achieved as follows:

[0106] Feature point extraction: Extract feature points such as corners, edges, or SIFT feature points from each dataset.

[0107] Feature point description: Generate a descriptor for each feature point for subsequent matching.

[0108] Matching: Use a matching algorithm (such as nearest neighbor matching) to match the feature points in the depth dataset with the feature points in the visible light dataset.

[0109] This alignment ensures that each depth data sample has a corresponding visible light data sample so that the connection between them can be established in subsequent steps.

[0110] Step 1.3: Feature mapping based on pre-trained multimodal encoder;

[0111] Next, a pre-trained multimodal encoder can be used, such as a jointly trained deep learning model, to map the data from the low-resolution depth dataset and the visible light dataset into a shared feature embedding space.

[0112] Model selection: Choose a pre-trained model suitable for processing multimodal data, such as the multimodal extension of BERT (Bidirectional Encoder Representations from Transformers), or a custom multimodal encoder.

[0113] For example, in step 2.3.1, we will use the Transformer model as the base to learn the mapping function from visible light features and low-resolution deep features to the shared feature embedding space through end-to-end training. This mapping function can be expressed as:

[0114] zdepth=f(xdepth,xrgb)

[0115] Among them, xdepth is a low-resolution depth dataset, xrgb is a visible light dataset, and f(·) is a cross-modal mapping function.

[0116] Next, we use this mapping function to establish a connection between the low-resolution depth dataset and the visible light dataset.

[0117] Data preparation and preprocessing:

[0118] Assume that we have a specific example dataset consisting of the following:

[0119] A low-resolution depth dataset (D = {d_i}), where (d_i) is a low-resolution depth image.

[0120] Visible light dataset (V = {v_j}), where (v_j) is the corresponding RGB image.

[0121] Model selection and training

[0122] Selecting a Transformer model: We choose a pre-trained Transformer model as a base, such as BERT (Birect iona l Encoder Representations from Transformers), and adjust it according to our needs.

[0123] Model construction: Build a multimodal Transformer encoder:

[0124] Depth Encoder (E_d): Takes in a low-resolution depth image (d_i) and encodes it into a fixed-length feature vector (E_d(d_i)).

[0125] Visible Encoder (E_v): Takes an RGB image (v_j) and encodes it into a fixed-length feature vector (E_v(v_j)).

[0126] Cross-modal mapping function (M): Design a mapping function to map (E_d(d_i)) and (E_v(v_j)) to a shared feature embedding space.

[0127] Training process

[0128] End-to-end training: The entire model is trained in an end-to-end manner to optimize the parameters of the cross-modal mapping function (M) so that deep features and visible light features can be effectively aligned in the shared feature space.

[0129] Establishing the connection between depth and visible light samples

[0130] Inference using mapping function: After training, the trained model and mapping function can be used to establish the connection between depth and visible light samples. For a given low-resolution depth image (d_i), the feature vector (E_d(d_i)) is obtained by encoding (E_d).

[0131] For the corresponding visible light image (v_j), the feature vector (E_v(v_j)) is obtained by (E_v) encoding.

[0132] Input (E_d(d_i)) and (E_v(v_j)) into the cross-modal mapping function to obtain their mapping (f(d_i,v_j)) in the shared feature embedding space.

[0133] Through such an embodiment, we can effectively establish a connection between the low-resolution depth dataset and the visible light dataset, so that they can perform meaningful distance and relationship calculations in a shared feature space.

[0134] Step 1.4: Forming a depth-visible light sample;

[0135] Finally, through the mapping in step 2.3, we can form depth-visible samples that have related representations in the shared feature space.

[0136] These depth-visible light samples can be used for various tasks such as visual SLAM (Simultaneous Localization and Mapping), 3D object recognition, action recognition, etc., thereby utilizing the complementary information of the two modal data.

[0137] Through the above steps, we successfully integrated the low-resolution depth dataset and the visible light dataset, established the connection between them, and made them comparable and applicable in a shared feature space.

[0138] Optionally, in addition to using the Transformer model to establish the connection between depth and visible light samples, there are some alternative or extended solutions that can be considered, depending on the characteristics of the dataset, the available computing resources, and the requirements of the application scenario:

[0139] Convolutional Neural Network (CNN):

[0140] Feature extraction and alignment: Use pre-trained convolutional neural networks, such as ResNet and VGG, to extract features from low-resolution depth images and visible light images respectively.

[0141] Feature fusion: The features of the depth image and the visible light image are fused. Some fusion strategies can be used, such as simple cascading or attention mechanisms.

[0142] Generative Adversarial Networks (GANs):

[0143] Generate High-Resolution Depth Images: Generate high-resolution depth images using generative adversarial networks to enhance the quality of depth information.

[0144] Depth-visible light alignment: Through the generation process of GAN, the complex mapping relationship between depth and visible light data can be learned.

[0145] Autoencoder:

[0146] Feature Learning and Reconstruction: Use autoencoders to learn feature representations for depth and visible light data, and then align them in the encoding space.

[0147] Reconstruction and mapping: Through the reconstruction process of the autoencoder, the mapping between depth and visible light samples can be achieved.

[0148] Other models for cross-modal learning:

[0149] Multimodal learning models: such as the Multimodal Neural Image Processing model (MM-D IP) or the Multimodal Deep Learning model (MDL), are specifically designed to process the relationship between depth and visible light images.

[0150] Task-specific model adaptation: According to specific task requirements, adjust the existing cross-modal learning model to better adapt to the characteristics of depth and visible light data.

[0151] Fine-tuning using a pre-trained model:

[0152] Transfer learning: Use the feature representation of a pre-trained model (such as BERT or GPT) to extract features from depth and visible light data, and build a connection between depth-visible light samples based on this.

[0153] Graph Neural Networks (GNN):

[0154] Relationship modeling of spatial data: If the data contains topological or structural information (such as geographic data or 3D models), graph neural networks can be used to model the spatial relationships of depth and visible light data.

[0155] It can be selected and adjusted according to specific circumstances to better meet task requirements and data characteristics.

[0156] Step 2: Obtain a quality-enhanced target depth image based on the data pair set and the preset model.

[0157] Based on the above embodiment, step 2 specifically includes:

[0158] Step 2.1, construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning;

[0159] Step 2.2, input the data pair set into the cross-modal generator for training;

[0160] Step 2.3, collect the low-resolution depth image corresponding to the target human body and input it into the trained cross-modal generator to generate a high-resolution target depth image, where the expression of the target depth image is bf{z}_{depth}=G(xdepth|xrgb)

[0161] Among them, G represents the cross-modal generator.

[0162] According to a specific implementation of the embodiment of the present disclosure, step 2.1 specifically includes:

[0163] Construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrast learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, and the distance between the unpaired samples is as large as possible, where the expression of the cross-modal contrast loss function is:

[0164] contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)

[0165] Among them, sim is the similarity metric, τ is the temperature hyperparameter of contrastive learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.

[0166] In the specific implementation, a cross-modal contrast loss function is constructed, the goal is to make the distance between paired depth-RGB samples in the latent feature space as small as possible, and the distance between unpaired samples as large as possible. When τ takes a small value (close to 0), the similarity score becomes more "concentrated". That is, the similarity score of positive samples (paired samples) will be amplified, and the similarity score of negative samples (unpaired samples) will be suppressed. This is conducive to the model learning more differentiated feature representations. When $τ$ takes a larger value, the distribution of similarity scores becomes more "flat". That is, the difference in similarity scores between positive and negative samples will become smaller, and the feature representation learned by the model will be more "smooth". The key advantage of this method is that even if a specific modal combination has not been seen in the training data, the model can spontaneously establish a connection between these modalities, resulting in a strong "zero-sample" behavior, which not only solves the problem of low-resolution depth sensors in human activity recognition.

[0167] For example, suppose we have some paired depth images and visible light images, and we want to learn to align them in the latent feature space. Suppose we have the following paired dataset:

[0168] Depth image: [D_1,D_2,D_3,...]

[0169] Visible light image: [V_1,V_2,V_3,...]

[0170] We hope that during the learning process, for the matching (positive) sample pairs ((D_i, Vi_i)), their distance in the latent feature space is as small as possible; while for the mismatching (negative) sample pairs ((D_i, V_j)), their distance is as large as possible. First, a pre-trained convolutional neural network (such as ResNet) is used to extract the feature representations (f(D)) and (f(V)) of each image. Then, we adjust the feature space by optimizing the loss function defined above, so that the matching sample pairs (f(D_i)) and (f(V_i)) are as close as possible, while the mismatching sample pairs (f(D_i)) and (f(V_j)) are as far away as possible.

[0171] This approach is able to effectively optimize the cross-modal alignment between depth and visible light images during the learning process, thereby better representing their relationship in the latent feature space.

[0172] Alternatively, in addition to contrastive learning, there are some alternative or scalable methods that can be considered. These methods may be selected according to the specific application scenario and data set characteristics:

[0173] Generative Adversarial Networks (GANs):

[0174] Using a generative adversarial network, you can directly generate depth images or visible light images, thereby achieving the purpose of cross-modal alignment. The generator and discriminator networks of GANs can be designed to generate images of corresponding modalities so that the generated images are aligned in the latent space. This method is particularly suitable for situations where a large amount of data is required for alignment.

[0175] Self-supervised learning methods:

[0176] Using self-supervised learning methods, such as using image self-reconstruction or self-supervised tasks in the image domain, feature representations can be learned without explicit pairing. These methods usually learn the correspondence between modalities while maintaining the image content.

[0177] Multi-task learning:

[0178] The depth-visible light alignment task is considered as a multi-task learning problem, which optimizes multiple objectives in the feature space at the same time. For example, it can be combined with tasks such as semantic segmentation or depth estimation to improve the consistency and usability of feature representation.

[0179] Metric Learning:

[0180] Consider using metric learning methods to directly optimize the distance metric of images in the feature space. Unlike contrastive learning, metric learning can more flexibly define the similarity metric between images to better capture the relationship between cross-modal image pairs.

[0181] Methods based on generative models:

[0182] Similar to GANs, other approaches based on generative models, such as variational autoencoders (VAEs), can also be considered to learn latent variables between depth and visible light images for alignment.

[0183] These alternative methods or extensions can be selected and adjusted according to the specific task requirements and available data resources. In practical applications, the decision on which method or combination of methods to use is usually based on the quality of the data, the size of the dataset, the computing resources, and the complexity of the specific task.

[0184] Step 4: Input the fine-tuned depth-visible light samples into the cross-modal generator for training;

[0185] In specific implementation, when the fine-tuned depth-visible light samples are input into the cross-modal generator for training, the following implementation steps can be used as an example:

[0186] Data preparation:

[0187] Make sure that the depth image and visible light image have been aligned and fine-tuned so that they have a certain degree of correspondence and matching in the feature space.

[0188] Select the cross-modal generator:

[0189] Choose a cross-modal generator that is appropriate for your task, such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). These generators can take a depth image as input and try to generate the corresponding visible light image.

[0190] Network architecture design:

[0191] Design the network architecture of the generator. For example, for GANs, the design of the network structure including the generator and the discriminator is crucial. The generator is responsible for generating visible light images from the input depth images, while the discriminator evaluates the difference between the generated images and the real visible light images.

[0192] Loss function definition:

[0193] Define an appropriate loss function to guide the training process of the generator. Usually includes: Adversarial Loss: used to encourage the generated visible light images to be visually similar to the real visible light images.

[0194] Content Loss: Ensures that the generated image corresponds to the input depth image in feature space.

[0195] Other auxiliary losses: such as cycle consistency loss or perceptual loss, help improve the quality and realism of generated images.

[0196] Training process:

[0197] The fine-tuned depth-visible light samples are used as input to start training the generator network. During the training process, the generator optimizes the loss function through back-propagation to generate results as close to the real visible light images as possible.

[0198] Monitoring and Evaluation:

[0199] Monitor the quality of generated images and the convergence of the generator network during training. You can use a validation set or periodically evaluate the images generated by the generator to ensure that it can generate accurate and realistic visible light images under different conditions.

[0200] Adjustments and Optimizations:

[0201] Based on the evaluation results and performance during training, the network architecture, hyperparameters, or loss function are adjusted and optimized to further improve the performance of the generator and the quality of generated images.

[0202] Alternatively, one can explore using end-to-end deep learning models, such as neural network-based direct converters. These models generate visible light images directly from depth images without explicit fine-tuning. This approach may require larger datasets and more complex network architectures, but can simplify the process to some extent.

[0203] Optionally, data augmentation techniques can be used to enhance the fine-tuned samples to improve the generalization ability of the cross-modal generator. This includes introducing changes such as rotation, scaling, and affine transformations during training to simulate changes between depth image and visible light image pairs in different environments.

[0204] Step 5: Collect the low-resolution depth image corresponding to the target human body and input it into the trained cross-modal generator to generate a high-resolution depth image.

[0205] Furthermore, the expression of the high-resolution depth image is:

[0206] bf{z}_{depth}=G(xdepth|xrgb)

[0207] Among them, G represents the cross-modal generator.

[0208] Through cross-modal learning, we can establish a correspondence between the two types of data. Specifically, the model will learn how to map the visual features contained in the visible light data to the spatial structure of the depth data. This process can be seen as an implicit feature completion or super-resolution process. Although low-resolution depth data cannot directly capture high-frequency details, through associative learning with visible light data, the model can infer these missing high-frequency information, thereby generating more realistic high-quality depth images.

[0209] When implementing, input data:

[0210] We have a low-resolution depth image of the target person as input. This depth image may be generated by a sensor such as Kinect or ToF camera, so the resolution may be low and the depth information may not be detailed enough.

[0211] Associative Learning:

[0212] The cross-modal generator model has learned the association between low-resolution depth images and visible light images through a large amount of training data. This association learning is usually done by jointly training the model, with inputs including low-resolution depth images and corresponding high-resolution RGB images (visible light data).

[0213] Feature extraction and mapping:

[0214] The model uses the input low-resolution depth image and the corresponding RGB image for feature extraction. These features may include features related to depth information such as contours and textures, while also considering the color information and structural information in the RGB image.

[0215] High-frequency information inference:

[0216] Through training, the cross-modal generator learns how to infer the missing high-frequency information from low-resolution depth images. This high-frequency information usually includes subtle depth changes and details such as edges and slight undulations on the surface.

[0217] Generate high-quality depth images:

[0218] After inferring the missing high-frequency information, the model incorporates this information into the generation process of low-resolution depth images. This process involves using a deep generative model (which may be based on GAN or other generative models) to generate high-quality, realistic high-resolution depth images by gradually optimizing and enhancing low-resolution depth images. Figure 2 shown.

[0219] Output and Evaluation:

[0220] The final generated high-resolution depth image can be evaluated by standard evaluation metrics (e.g., PSNR, SSIM) to ensure the similarity and quality between the generated image and the real high-resolution depth image.

[0221] The cross-modal generator learns the association between low-resolution depth images and RGB images through joint training, so that it can infer the missing high-frequency information in the low-resolution depth images, and then generate more realistic and high-quality high-resolution depth images. This method has a wide range of application potentials in depth sensors and visual applications, and can significantly improve the quality and accuracy of depth images.

[0222] Optionally, use image enhancement techniques, such as super-resolution algorithms, noise removal algorithms, etc., to pre-process low-resolution depth images to provide clearer and more accurate input data to the cross-modal generator. These techniques can enhance the data quality before the generator, thereby improving the effect of generating depth images.

[0223] The depth image quality enhancement method provided in this embodiment realizes the cross-modal integration of two heterogeneous data modes, low-resolution depth sensor and visible light data, into a unified embedding space by utilizing the capabilities of a large basic model, so as to solve the problems of scarcity and difficulty in understanding low-resolution depth sensor data. The inherent correlation properties of the image are utilized to align the embeddings of the two modalities with the image embedding, and then the aligned depth-visible light samples are fine-tuned through comparative learning. This cross-modal alignment method can produce powerful zero-sample behavior and automatically associate modal combinations that have never appeared in the training data. Even when using low-cost sensors such as low-resolution depth cameras, it can achieve an accuracy comparable to that of high-end sensors in human activity recognition tasks, greatly reducing the hardware cost of system deployment. In addition, it avoids dependence on large-scale high-quality annotated data, greatly reduces the workload of data collection and annotation, provides a replicable solution for similar applications, and improves the reconstruction efficiency, accuracy and adaptability of low-resolution depth images.

[0224] Corresponding to the above method embodiment, see Figure 3 The present disclosure also provides a deep image quality enhancement system 30, including:

[0225] An alignment module 301 is used to align an initial depth data set with a corresponding visible light data set to form a data pair set of depth data and visible light data;

[0226] The enhancement module 302 is used to obtain a target depth image after quality enhancement according to the data pair set and the preset model.

[0227] Figure 3 The system shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.

[0228] See also Figure 4 The embodiment of the present disclosure further provides an electronic device 40, which includes: at least one processor and a memory connected to the at least one processor. The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the depth image quality enhancement method in the aforementioned method embodiment.

[0229] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the depth image quality enhancement method in the aforementioned method embodiment.

[0230] The embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, enable the computer to execute the depth image quality enhancement method in the aforementioned method embodiment.

[0231] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 40 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0232] like Figure 4 As shown, the electronic device 40 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 40 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0233] Typically, the following devices may be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication device 409 may allow the electronic device 40 to communicate wirelessly or wired with other devices to exchange data. Although the electronic device 40 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0234] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0235] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0236] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0237] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can execute the relevant steps of the method embodiment.

[0238] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the method embodiment.

[0239] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0240] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0241] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware.

[0242] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware or a combination thereof.

[0243] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for enhancing deep image quality, characterized in that: include: Step 1: aligning the initial depth data set with the corresponding visible light data set to form a data pair set of depth data and visible light data; Step 2: Obtain a quality-enhanced target depth image based on the data pair set and the preset model.

2. The method according to claim 1, characterized in that , before step 1, the method further includes: A low-resolution depth dataset containing various human behaviors is collected as the initial depth dataset.

3. The method according to claim 1, characterized in that , the step 1 specifically includes: Step 1.1, normalize the low-resolution depth dataset and the visible light dataset: Step 1.2, using feature point matching method to align the normalized low-resolution depth dataset and visible light dataset in time and space; Step 1.3, using a pre-trained multimodal encoder to map the data in the low-resolution depth dataset and the visible light dataset into a shared feature embedding space, and establish a connection between the low-resolution depth dataset and the visible light dataset; Step 1.4, forming a data pair set of depth-visible light samples.

4. The method according to claim 3, characterized in that , the step 1.2 specifically includes: Step 1.2.1, extract feature points from the low-resolution depth dataset and the visible light dataset respectively; Step 1.2.2, calculate the feature vector for each feature point; Step 1.2.3, use a feature matching algorithm to match the feature points in the low-resolution depth dataset and the visible light dataset; In step 1.2.4, the matched feature points are used to perform geometric correction on the registered data in the low-resolution depth dataset and the visible light dataset.

5. The method according to claim 3, characterized in that , the step 1.3 specifically includes: Step 1.3.1, using the Transformer model as the base, through end-to-end training, learn the mapping function from visible light features and deep features to the shared feature embedding space, where the mapping function is zdepth=f(xdepth,xrgb) Among them, xdepth is a low-resolution depth dataset, xrgb is a visible light dataset, and f(·) is a cross-modal mapping function; Step 1.3.2, use the mapping function to establish the connection between the low-resolution depth dataset and the visible light dataset.

6. The method according to claim 5, characterized in that , the step 2 specifically includes: Step 2.1, construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrastive learning; Step 2.2, input the data pair set into the cross-modal generator for training; Step 2.3, collect the low-resolution depth image corresponding to the target human body and input it into the trained cross-modal generator to generate a high-resolution target depth image, where the expression of the target depth image is: bf{z}_{depth}=G(xdepth|xrgb) Among them, G represents the cross-modal generator.

7. The method according to claim 5, characterized in that , the step 2.1 specifically includes: Construct a cross-modal contrast loss function corresponding to the preset model, and fine-tune the preset model through contrast learning, so that the distance between the aligned depth-visible light samples in the latent feature space is as small as possible, and the distance between the unpaired samples is as large as possible, where the expression of the cross-modal contrast loss function is: contrast=-logexp(sim(zdepth,zrgb) / τ)∑iexp(sim(zdepth,zrgbi) / τ)where sim is the similarity measure, τ is the temperature hyperparameter for contrast learning, and zdepth and zrgb are the aligned low-resolution depth data and visible light data, respectively.

8. A deep image quality enhancement system, characterized in that: include: An alignment module, used to align an initial depth data set with a corresponding visible light data set to form a data pair set of depth data and visible light data; The enhancement module is used to obtain a target depth image after quality enhancement according to a data pair set and a preset model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the depth image quality enhancement method described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the deep image quality enhancement method according to any one of claims 1 to 7.

Images