Image detail enhancement method, device and equipment of target region and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for enhancing image details in a target region. The method includes: acquiring a video to be processed; determining whether the video meets the conditions for image detail enhancement; if it meets the conditions, performing a template transformation on the frame image of the video to be processed to obtain an input matrix of the target region; inputting the input matrix into an enhancement network model to generate an enhancement result of the target region based on extracted multi-layer features, latent vectors, and random signals; performing a restoration process corresponding to the template transformation on the enhancement result of the target region, and fusing the result of the inverse transformation with the original frame image to obtain the image detail enhancement result. This solution can improve the quality and accuracy of image processing, thereby enhancing the user's viewing experience. Simultaneously, while enhancing image details, it also precisely controls the computational load to ensure processing efficiency.

Description

Technical Field

[0001] This application relates to the field of video data processing technology, and in particular to a method, apparatus, device, and storage medium for enhancing image details of a target area. Background Technology

[0002] With the increasing maturity of live streaming video technology and the widespread adoption of related applications, the quality of video content presentation has become a core competitive advantage that various live streaming applications need to focus on. When watching video programs, viewers tend to have a stronger preference for and linger on specific areas of interest, such as faces, objects, and landscapes. Therefore, enhancing the detail in specific areas is also an important part of improving the subjective quality of the video.

[0003] Nowadays, by using convolutional deep neural networks to downsample and then upsample an image multiple times, and by making full use of the image features learned by different neural network layers, high-resolution images that still retain all details can be obtained.

[0004] However, convolutional neural networks do not specifically process specific regions of an image. While they can achieve decent super-resolution results for specific regions, they still cannot accurately reproduce the detailed features of those regions. Therefore, how to enhance the features of specific regions in an image to meet users' demands for high-quality content in various video application scenarios, thereby improving the user experience and increasing user retention time, is a pressing issue in this field. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for enhancing image details in a target area, solving the problem in existing technologies where the inability to accurately reproduce detailed features in specific areas of a video leads to reduced video quality. By employing this solution, image processing quality and accuracy can be improved, thereby enhancing the user's viewing experience. Simultaneously, while enhancing image details, the computational load is precisely controlled to ensure processing efficiency.

[0006] In a first aspect, embodiments of this application provide an image detail enhancement method for a target region, the method comprising:

[0007] Acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0008] If the image detail enhancement conditions are met, then the frame image of the video to be processed is subjected to template transformation processing of the target region to obtain the input matrix of the target region;

[0009] The input matrix is ​​fed into the enhancement network model, and the enhanced result of the target region is generated based on the extracted multi-layer features, latent vectors and random signals.

[0010] The enhancement result of the target region is subjected to the inverse transformation of the template transformation process, and the inverse transformation result is fused with the original frame image to obtain the image detail enhancement result.

[0011] Secondly, embodiments of this application also provide an image detail enhancement device for a target region, comprising:

[0012] The acquisition module is used to acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0013] The frame image processing module is used to perform template transformation processing on the frame image of the video to be processed to obtain the input matrix of the target region if the image detail enhancement conditions are met.

[0014] The target region enhancement result generation module is used to input the input matrix into the enhancement network model and generate the enhancement result of the target region based on the extracted multi-layer features, latent vectors and random signals.

[0015] The image detail enhancement result generation module is used to perform inverse transformation processing on the enhancement result of the target region using the template transformation processing, and to fuse the inverse transformation processing result with the original frame image to obtain the image detail enhancement result.

[0016] Thirdly, embodiments of this application also provide an image detail enhancement device for a target region, the device comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs.

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image detail enhancement method for the target region described in the embodiments of this application.

[0020] Fourthly, embodiments of this application also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image detail enhancement method for the target region described in embodiments of this application.

[0021] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the image detail enhancement method for the target region described in embodiments of this application.

[0022] In this embodiment, a video to be processed is acquired, and it is determined whether the video meets the conditions for image detail enhancement. If it does, a template transformation of the target region is performed on the frame image of the video to be processed to obtain an input matrix of the target region. The input matrix is ​​input into an enhancement network model, and an enhancement result of the target region is generated based on the extracted multi-layer features, latent vectors, and random signals. The enhancement result of the target region is subjected to inverse transformation processing of the template transformation, and the inverse transformation result is fused with the original frame image to obtain the image detail enhancement result. This method for enhancing image details in the target region can improve image processing quality and accuracy, thereby enhancing the user's viewing experience. Simultaneously, while enhancing image details, the computational load is precisely controlled to ensure processing efficiency. Attached Figure Description

[0023] Figure 1 A schematic flowchart illustrating the image detail enhancement method for the target region provided in Embodiment 1 of this application;

[0024] Figure 2 A schematic diagram of the framework of the network structure for enhancing image details of the target region provided in Embodiment 1 of this application;

[0025] Figure 3 A schematic diagram illustrating the steps of the image detail enhancement and restoration process for the target area provided in Embodiment 1 of this application;

[0026] Figure 4 This is a flowchart illustrating the image detail enhancement method for the target region provided in Embodiment 2 of this application;

[0027] Figure 5 This is a schematic diagram of the structure of the image detail enhancement device for the target area provided in Embodiment 3 of this application;

[0028] Figure 6 This is a schematic diagram of the structure of the image detail enhancement device for the target area provided in Embodiment 4 of this application. Detailed Implementation

[0029] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] Example 1

[0032] Figure 1 This is a schematic flowchart of the image detail enhancement method for the target region provided in Embodiment 1 of this application. Figure 1 As shown, the specific steps include the following:

[0033] S101, Obtain the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0034] First, the application scenario of this solution can be in a smart terminal to process video, enhance the details of the target area of ​​the frame image of the video, and output the enhanced image.

[0035] Based on the above usage scenarios, it is understandable that the executing entity of this application can be the smart terminal, and no further restrictions are imposed here.

[0036] In this solution, the video to be processed can refer to video files that require image detail enhancement processing. The specific video files included depend on the application scenario and requirements. For example, in the field of video editing, it is necessary to enhance the image details of the recorded raw video to improve picture quality and visual effects.

[0037] The video to be processed can be obtained in the following ways:

[0038] 1. If the video to be processed is already saved on the local computer, it can be obtained directly by reading the file.

[0039] 2. If you need to process the video captured by the camera in real time, you can use the camera on your computer or mobile device to capture the video.

[0040] 3. If you need to process online video streams, you can obtain the video stream through network requests and convert it into a format that you can process.

[0041] 4. Some video processing APIs (Application Programming Interfaces) provide methods for obtaining videos, which can be used to obtain the video to be processed.

[0042] If the video to be processed contains parts that require detail enhancement, it can be considered to meet the image detail enhancement conditions. In this solution, when the video to be processed is mainly about people, the facial details can be enhanced. Accordingly, the conditions for image detail enhancement can be set as follows: if the video frame contains a face and the area occupied by the face is less than 90% of the total image area, it can be considered to meet the image detail enhancement conditions.

[0043] Taking a video with a human figure as the main subject, requiring enhancement of facial details, as an example, to determine whether the video meets the image detail enhancement conditions, the algorithm module first reads a single-frame YUV / RGB image of size H*W*3 with pixel values ​​ranging from [0, 255] and feeds it into an existing face detector to determine if the video contains a face. Videos without faces are skipped. Frames containing faces can obtain detection bounding boxes containing faces through the face detector. The length and width of the face contained within the detected bounding box can be further calculated based on its coordinates, denoted as face_W and face_H respectively. The standard face size is 512*512. When the ratio of the face region to the standard face size is less than 90%, the image detail enhancement conditions are met. The ratio calculation formula is as follows:

[0044] ratio=(face_W×face_H)÷(512*512);

[0045] S102, if the image detail enhancement conditions are met, then the frame image of the video to be processed is subjected to template transformation processing of the target region to obtain the input matrix of the target region.

[0046] The video to be processed consists of a series of consecutive frame images, each of which is a static image. In computers, video is usually composed of a series of image frames arranged in a certain time sequence; the faster the playback speed, the shorter the time interval between adjacent frames.

[0047] The target region can be a specific area within the video frame image to be processed, and its selection depends on the specific application scenario and requirements. In this solution, the target region can be a rectangular area containing a face.

[0048] In the template transformation process of the target region, the input matrix of the target region can be the matrix obtained after converting the target region into matrix form, which can include pixel values, pixel coordinates and color information in the target region.

[0049] The input matrix of the target region can be obtained through the following steps:

[0050] 1. Based on specific application requirements, determine the target area in the video frame image to be processed.

[0051] 2. Extract features from the target region. For example, you can use color histograms, texture features, and shape descriptions to represent the features of the target region as vectors or matrices.

[0052] 3. Match the extracted target region features with the entire frame image to find the image region most similar to the target region. Deep learning methods, such as convolutional neural networks, can be used for this matching.

[0053] 4. Based on the matching results, calculate the transformation matrix to map the features of the target region onto the entire image. The transformation matrix can be calculated using affine transformation or projection transformation, etc.

[0054] 5. Apply the transformation matrix: Apply the calculated transformation matrix to perform transformation operations on the entire frame image, thereby achieving image enhancement, deformation, or other processing.

[0055] 6. Extract the input matrix of the target region: Based on the transformed target region, extract the pixel values, pixel coordinates, and color information of the target region, and convert them into matrix form to obtain the input matrix of the target region.

[0056] S103, the input matrix is ​​input into the enhancement network model, and the enhancement result of the target region is generated based on the extracted multi-layer features, latent vectors and random signals.

[0057] Image enhancement network models refer to deep learning-based image enhancement methods that can enhance each frame of an input video, thereby improving video quality and visual effects. They typically use convolutional neural networks or similar neural network architectures to learn the mapping from input to output images. During training, the enhancement network model learns by using existing correspondences between high-quality and low-quality images, thus learning how to extract more information from low-quality images to generate higher-quality ones. During testing, the model takes a low-quality input image as input and, using the learned mapping, generates a higher-quality output image.

[0058] Multilayer features can be features extracted from multiple convolutional layers or other layers in a neural network. These features can represent different levels of information in the input data, from low-level image texture to high-level semantic information.

[0059] Latent vectors can refer to latent space vectors generated in certain layers of a neural network. They typically have low dimensionality and can represent abstract features in the input data.

[0060] Random signals can be some noise or random variables added to augment network models, and are often used to increase the model's generalization ability and robustness.

[0061] The enhancement result of the target region refers to the result generated by the enhancement network model after performing image enhancement operations on a specified target region in the input matrix. Specifically, in the enhancement network model, the input matrix is ​​first divided into several different regions, and then enhancement operations are performed on each region separately. During the enhancement operation, the model generates the corresponding target region enhancement result based on the input multi-layer features, latent vectors, and random signals. This result can be an enhanced image, feature map, or other forms of output.

[0062] In this scheme, taking face recognition as an example, if the target region is a face region, after performing face region template transformation, the original face is obtained by scaling the size to 512*512 and normalizing the pixel value range to the [-1, 1] interval. The input network first performs feature extraction and downsampling through multiple convolutional layers, extracting the core features of the original face with low quality in a hierarchical manner. The downsampled features are stretched and input into a small fully connected layer network to obtain a latent vector representing the input original face. The latent vector, combined with the downsampled feature map obtained layer by layer during the downsampling process, and some random signals as input, is input to the generator of the enhancement network model. It is gradually restored layer by layer through multiple Generator Blocks (GAN Blocks) with upsampling function to restore the enhanced face feature map of different resolutions. This process is iterated until the enhanced result of the face region with the same size as the input face of 512*512 is obtained.

[0063] Based on the above technical solutions, optionally, the input matrix is ​​input to an enhancement network model to generate an enhancement result for the target region based on the extracted multi-layer features, latent vectors, and random signals, including:

[0064] The input matrix is ​​fed into the convolutional layer of the augmented network model to obtain multi-layer features;

[0065] The features output from the last convolutional layer are input into the linear connection layer to obtain the latent vector of the frame image;

[0066] The multi-layer features, the latent vectors, and the pre-obtained random signal are input into the generator of the enhancement network model to obtain the enhancement result of the target region; wherein, the random signal is random noise that conforms to a Gaussian distribution obtained based on the signal generation network.

[0067] In this approach, convolutional layers are used to extract multi-layered features from the input image for subsequent tasks such as classification, detection, and segmentation. The convolutional layers perform convolution operations on the input image using sliding kernels, generating a series of convolutional feature maps, each corresponding to a kernel. As the depth increases, the convolutional neural network can extract increasingly abstract features, thereby improving model performance.

[0068] The input matrix of the target region is fed into the convolutional layer of the augmentation network model, and features are extracted from the input matrix using convolution operations. The role of the convolutional layer is to extract features from the input matrix by performing convolution operations on the input matrix using a series of learnable convolutional kernels. The convolution operation performs convolution operations on each pixel of the input matrix with its surrounding pixels, resulting in a series of convolutional feature maps. By using multiple convolutional layers, features at different levels can be extracted, forming multi-layered feature representations for subsequent latent vector generation and augmentation processing.

[0069] Linearly connected layers are a common type of layer in deep learning. Their function is to map the input feature matrix to a new feature space through matrix multiplication and the addition of bias terms. In convolutional neural networks, the feature matrix output from the last convolutional layer is typically used as input, flattened, and then connected to a fully connected layer.

[0070] In deep learning, linearly connected layers, also known as fully connected layers, are used to reduce the dimensionality and reconstruct the feature maps extracted by convolutional layers, resulting in more abstract and higher-dimensional feature representations. For the features output by the last convolutional layer, global pooling is typically used to reduce the dimensionality of the feature map, and then the reduced features are reconstructed through a fully connected layer to obtain the final latent vector representation.

[0071] The generator of an augmented network model can refer to a neural network model capable of generating augmented results from latent vectors, random signals, and multi-layer features. Its main function is to synthesize the input latent vectors and random signals with multi-layer features to generate augmented results.

[0072] A signal generation network can be a deep learning model whose purpose is to generate random signals that conform to a specific distribution. In the generator model, random noise can be obtained by sampling from a known distribution. The Gaussian distribution, also known as the normal distribution, is a continuous probability distribution. Random noise conforming to a Gaussian distribution can be obtained by sampling from a Gaussian distribution with a mean of 0 and a variance of 1.

[0073] Multi-layer features, latent vectors, and random signals are concatenated and fed as input into the generator network. The generator network maps these inputs to an enhanced target region, which can be achieved by progressively upsampling using deconvolutional or transposed convolutional layers, while simultaneously performing convolution operations and activation function processing to gradually restore the resolution and detail information of the original image, ultimately yielding an enhanced target region image.

[0074] In this approach, convolutional layers capture information from different levels of the input matrix. These features reflect the spatial structure and texture information of the input matrix, which can be used for subsequent processing. The generator of the augmentation network model utilizes multi-layer features, latent vectors, and pre-obtained random signals to generate augmented results for the target region, making the augmented results more realistic and diverse, and improving the model's generalization ability.

[0075] Based on the above technical solutions, optionally, the enhanced network model further includes:

[0076] A discriminator is used to form an adversarial neural network with the generator during training to distinguish between genuine and fake augmented results of the target region generated by the generator.

[0077] In this solution, facial recognition is used as an example. Figure 2 This is a schematic diagram of the framework of the network structure for enhancing image details in the target region provided in Embodiment 1 of this application, as shown below. Figure 2 As shown, the overall network structure consists of three main parts: feature extraction (composed of convolutional layers), generator, and discriminator. The feature extraction part receives low-quality faces as input, generates image-related features layer by layer, and then obtains a latent vector through a linear connection layer. The latent vector is combined with intermediate layer features and some random const signals and input to the generator to restore high-definition enhanced faces layer by layer. Because the generator may diverge and not guarantee the authenticity of the face, a discriminator is needed as a constraint. During the process of the generator continuously generating faces, the discriminator is used to judge whether the authenticity and approximation of the generated face match the actual face.

[0078] A discriminator, in a generative adversarial network (GAN), is a neural network model used to distinguish between real and fake images generated by the generator. The discriminator typically employs a convolutional neural network (CNN) structure. Its input is an image (which can be real or generated by the generator), and its output is a probability value indicating whether the input image is real or fake. The training goal of the discriminator is to enable it to accurately distinguish between real and generated images. In an adversarial network, the generator and discriminator are optimized through adversarial training. The generator aims to produce images that can deceive the discriminator, making it unable to distinguish between real and generated images, while the discriminator aims to accurately determine whether the input image is real or generated. Through adversarial training, the generator and discriminator can gradually improve their abilities, ultimately generating more realistic images.

[0079] An adversarial neural network (ANN) can be a deep learning model consisting of two neural networks: a generator and a discriminator. The generator is responsible for generating samples similar to the training data, while the discriminator is responsible for distinguishing the samples generated by the generator from the training data. During training, the generator and discriminator compete against each other, continuously updating their parameters so that the samples generated by the generator become increasingly closer to the real data, and the probability of the discriminator misclassifying the samples generated by the generator decreases.

[0080] During training, both real samples and samples generated by the generator are first input into the discriminator for authentication, and the discriminator's loss value for these samples is calculated. Then, based on the samples generated by the generator and the discriminator's judgment of these samples, the generator's loss value is calculated. The smaller the generator's loss value, the more likely the generated samples are to fool the discriminator, meaning they are closer to real samples. Therefore, the generator will continuously adjust its generation strategy to improve the quality of the generated samples.

[0081] In this scheme, by introducing a training method that combines a discriminator and a generator to form an adversarial neural network, the enhancement effect of the target region generated by the generator can be effectively improved.

[0082] S104, the enhancement result of the target region is subjected to the inverse transformation processing of the template transformation processing, and the inverse transformation processing result is fused with the original frame image to obtain the image detail enhancement result.

[0083] Figure 3 This is a schematic diagram illustrating the steps of the image detail enhancement and restoration process for the target area provided in Embodiment 1 of this application, as shown below. Figure 3As shown, the restoration process of template transformation can be the inverse template transformation process. Based on a given template transformation matrix, the inverse matrix is ​​solved to restore the original image from the template-transformed image. When performing template transformation, geometric transformations are usually required on the original image, such as translation, rotation, scaling, and distortion. These operations can all be represented in the form of matrix transformations.

[0084] Image detail enhancement results can refer to a processed image that emphasizes details in the image, making it clearer and easier to identify. This includes enhancement processing of target areas and the final result obtained by fusing the enhancement results with the original frame image.

[0085] The following steps can be used to obtain image detail enhancement results:

[0086] 1. Convert the original image and the enhanced result of the target region to the same color space and pixel depth to ensure that they have the same image properties.

[0087] 2. Use appropriate fusion methods to fuse the original image and the enhancement results of the target region. Fusion methods may include weighted average, maximum value, minimum value and pixel-level fusion.

[0088] 3. Perform some post-processing on the merged image, such as noise reduction and color correction, to make the merged image clearer and more natural.

[0089] 4. The enhancement results are fused with the original image to obtain image detail enhancement results for multiple target areas.

[0090] Based on the above technical solutions, optionally, the enhancement result based on the target region is fused with the original frame image to obtain the image detail enhancement result, including:

[0091] The enhancement result of the target region is weighted and compared with the target region in the original frame image to obtain the weighted processing result.

[0092] The segmentation objects in the weighted processing result are determined by using a segmentation algorithm; or, the segmentation objects in the enhanced result are determined by using a segmentation algorithm.

[0093] The segmented objects in the weighted processing result or the segmented objects in the enhancement result are stitched together with the non-segmented object regions in the original frame image to obtain the image detail enhancement result.

[0094] In this scheme, the weighted processing result can be obtained by weighting the target region in the original frame image with the target region after enhancement. Specifically, the weighted processing result refers to averaging the two regions to produce a new image that combines information from both regions. The weights of the average can be adjusted according to the application requirements to achieve a reasonable balance between the enhancement effect and the weight of the original information, thereby achieving better visual effects or other application needs.

[0095] The weighted processing result can be obtained through the following steps:

[0096] 1. Align the original frame image and the enhanced target area to ensure they have the same size and resolution.

[0097] 2. Determine the weights for the weighted average based on the application's needs and objectives. For example, the weights for the original image and the enhanced image can be set to 0.5, representing the average value. If more emphasis is placed on the enhanced image information, the weight can be set to 0.7; if more emphasis is placed on the original image information, the weight can be set to 0.3. The original image and the enhanced target region are then weighted and averaged to obtain the weighted processing result.

[0098] 3. For other unprocessed areas, they can be retained or processed as needed to obtain the final processing result.

[0099] A segmentation algorithm is a computer vision algorithm that divides a digital image into multiple sub-regions. In this scheme, a computer algorithm is used to segment the weighted processing result into different objects or regions.

[0100] The segmented objects in the weighted processing result can refer to the different objects or regions identified by the segmentation algorithm in the weighted processing result. These objects or regions can be objects, backgrounds, and boundaries in the image. In this scheme, the segmented object can be a face region.

[0101] The following steps can be used to determine the segmentation target:

[0102] 1. Preprocess the weighted results, such as denoising and smoothing, to improve the accuracy and stability of the segmentation algorithm.

[0103] 2. Select a suitable segmentation algorithm to segment the weighted processing result. For example, deep learning algorithms can be used for segmentation, including semantic segmentation and instance segmentation.

[0104] 3. Depending on the algorithm, adjust some parameters or set some thresholds to optimize the segmentation results. For example, threshold segmentation requires selecting an appropriate threshold, while region growing requires setting growth conditions, etc.

[0105] 4. The results obtained by the segmentation algorithm may include multiple segmented objects, which need to be further processed, such as merging and filtering.

[0106] The segmented objects in the enhanced results can be objects, backgrounds, edges, etc. in the image, depending on the design of the segmentation algorithm and the application requirements.

[0107] The segmentation objects in the augmentation results can be determined by following these steps:

[0108] 1. Preprocess the enhancement results to enable the segmentation algorithm to process them better. For example, the image can be filtered, binarized, and noise removed.

[0109] 2. Based on the image type and target features, select a suitable segmentation algorithm. Commonly used segmentation algorithms include threshold-based segmentation algorithms, region growing algorithms, edge detection algorithms, and graph theory algorithms.

[0110] 3. Apply the selected segmentation algorithm to the enhanced image to segment the target object. After segmentation, a binary segmented image will be obtained, where the segmented object is represented by a white area and the background by a black area.

[0111] 4. Post-process the segmentation results to obtain more accurate segmentation results. For example, morphological processing can be performed on the segmented image to remove small disconnected regions and fill holes. Extract the objects to be segmented as needed.

[0112] 5. Extract the desired segmented objects from the segmentation result image according to specific needs. For example, the outline of the segmented object can be extracted, or the segmented object can be composited with the original image for display.

[0113] In this scheme, taking face recognition as an example, the enhancement result of the face region is fed into a simple face segmentation operator to obtain a binary segmentation image of the same size as the enhancement result of the face region. The face region related to the enhancement is segmented by the model and assigned a value of 1, while the rest of the irrelevant regions are assigned a value of 0. At this time, the binary segmentation image outlines the contour of the face to be enhanced. However, the face position is still in the standard position after alignment and rotation. In the preprocessing, the rotation transformation of the face restores the enhanced face image and the face segmentation contour to their corresponding positions in the original image frame H*W before alignment. This yields an image frame of the same size as the original image frame containing the enhanced face and a mask of the face segmentation part. In order to easily control the degree of face enhancement, so as not to make it too strong and cause local distortion, or too weak and have no intuitive effect, a hyperparameter alpha is first set, with a value range of [0, 1], so that the final enhanced face part is a weighted sum of the enhanced face and the original face. The code for determining the segmented face region can be:

[0114] final_enhance_face=enhance_face*alpha+origin_face*(1-alpha);

[0115] tmp_img = warpAffine(final_enhance_face, 5points_inv) / / Where tmp_img is of size H*W;

[0116] tmp_mask = warpAffine(face_mask, 5points_inv) / / where tmp_mask has a size of H*W;

[0117] The following steps can be followed to obtain image detail enhancement results:

[0118] 1. Preprocess the original frame image to facilitate subsequent processing. For example, operations such as scaling, cropping, and noise removal can be performed on the image.

[0119] 2. Extract the segmented objects from the enhanced results and perform further processing on them as needed. For example, morphological operations and contour extraction can be performed on the segmented objects to obtain more accurate segmentation results.

[0120] 3. Using the same segmentation algorithm as the enhancement result, segment the non-segmented object regions in the original frame image. After segmentation, a binary segmented image will be obtained, where the segmented objects are white regions and the background is a black region.

[0121] 4. Merge the segmented objects in the enhancement result with the non-segmented object regions in the original frame image to obtain the image detail enhancement result. Image masking techniques can be used to fuse the segmented and non-segmented object regions.

[0122] 5. Perform post-processing on the merged results, such as denoising, sharpening, and color adjustment, to achieve better visual results.

[0123] In this scheme, taking face recognition as an example, the coordinates of the enhanced face are restored to their state before rotation and alignment. A segmentation mask of the enhanced face region is also obtained through the segmenter. Therefore, it is necessary to further fuse this enhanced portion of the face into the corresponding face position in the original video frame. This requires using the binary mask obtained during segmentation. The face regions in the mask with values ​​of 1 are filled with the values ​​of the enhanced face, and the remaining regions are filled with the pixel values ​​corresponding to the original video frame. This yields the image detail enhancement result. The following is a partial code snippet:

[0124] Final_img=origin_img*(1-tmp_mask)+tmp_img*tmp_mask;

[0125] In this scheme, the image processing method described above can enhance the target region and improve the clarity of image details. Using a segmentation algorithm to determine the segmentation objects in the enhancement result or the weighted processing result allows for more accurate extraction of the target region, improving the accuracy and effectiveness of segmentation.

[0126] Based on the above technical solutions, optionally, the weighted processing result or the enhancement result includes the values ​​of the three channels of the color space;

[0127] Accordingly, a segmentation algorithm is used to determine the segmentation objects in the weighted processing result; or, a segmentation algorithm is used to determine the segmentation objects in the enhanced result, including:

[0128] The segmentation object in the weighted processing result is determined by using a segmentation algorithm based on at least one channel value in the weighted processing result; or, the segmentation object in the enhanced result is determined by using a segmentation algorithm based on at least one channel value in the enhanced result.

[0129] In this scheme, color space refers to a way of representing colors as spatial coordinates of different dimensions for color processing and representation. RGB (Red, Green, Blue) is one of the most commonly used color spaces. The RGB color space consists of three color channels: Red (R), Green (G), and Blue (B). It represents colors as three numerical values, each representing the brightness intensity of the corresponding channel. The values ​​of these three channels typically range from 0 to 255, where 0 represents the darkest color in the channel, and 255 represents the brightest color. The three channel values ​​in the weighted processing or enhancement results refer to the three channel values ​​of each pixel in the target area within the RGB color space.

[0130] The following steps can be used to determine the segmentation target:

[0131] 1. First, an appropriate threshold needs to be determined to separate the pixels in the weighted processing or enhancement result into two parts: the target object and the background object. The threshold can be determined based on application requirements and image features.

[0132] 2. Use a defined threshold to segment the pixels in the weighted processing or enhancement result, obtaining two parts: the target object and the background object. Segmentation algorithms such as binarization and the watershed algorithm can be used for this process.

[0133] 3. Post-processing is usually required after segmentation to remove small noise points or connect adjacent regions. Morphological operations and connectivity analysis can be used for post-processing to obtain more accurate segmentation results.

[0134] In this scheme, a segmentation algorithm is used to effectively divide the image into different parts for subsequent processing and analysis. Using the three channels of the color space for segmentation allows for more accurate identification of the edge and color features of the target region or segmented object, enabling more refined enhancement and stitching processes and resulting in higher-quality image detail enhancement.

[0135] Based on the above technical solutions, optionally, before stitching the segmented object in the weighted processing result or the segmented object in the enhancement result with the non-segmented object region in the original frame image, the method further includes:

[0136] Obtain the transformation rules for template transformation processing;

[0137] Based on the inverse operation of the transformation rule, and the segmented objects in the weighted processing result or the segmented objects in the enhancement result, the non-segmented object regions in the original frame image are determined.

[0138] In this scheme, the transformation rule for template transformation processing can be an affine transformation rule. An affine transformation is a two-dimensional graphic transformation method that can be described by a set of basic transformations. It maps the original image coordinates to the target image coordinates through linear and translational transformations, so that the transformed image maintains the relative positional relationships in the original image.

[0139] If the transformation rule for template transformation is an affine transformation rule, the transformation rule can be obtained through the following steps:

[0140] 1. For each corresponding matching point in the source and target images, at least three pairs of matching points need to be selected to determine the six unknown parameters required for the affine transformation.

[0141] 2. Based on the selected matching point pairs, calculate the affine transformation matrix, which maps points in the source image to points in the target image. The affine transformation matrix can be a 2×3 matrix containing six unknown parameters, which can be solved using the known matching point pairs.

[0142] 3. Apply the calculated affine transformation matrix to all points in the source image to obtain the transformed image.

[0143] If the template transformation process follows an affine transformation rule, the inverse operation based on this rule first requires transforming the segmented objects in the weighted or enhanced results back to the coordinate system of the original image. Then, based on the coordinates of these inversely transformed segmented objects, the positions of the regions covered by these objects in the original image can be calculated. By inverting these covered regions in the original frame image, the non-segmented object regions can be determined.

[0144] In this scheme, the above method allows for local enhancement of the target region while preserving information about non-segmented objects in the original image, resulting in more accurate enhancement results. Furthermore, by employing a segmentation algorithm to determine the segmentation objects, enhancement of the entire image can be avoided, reducing processing load and improving efficiency.

[0145] In this embodiment, a video to be processed is acquired, and it is determined whether the video meets the conditions for image detail enhancement. If it does, a template transformation of the target region is performed on the frame images of the video to be processed to obtain an input matrix for the target region. The input matrix is ​​then input into an enhancement network model, which generates an enhancement result for the target region based on extracted multi-layer features, latent vectors, and random signals. The enhancement result for the target region is then subjected to an inverse transformation of the template transformation, and the result of the inverse transformation is fused with the original frame image to obtain the image detail enhancement result. This method for enhancing image details in the target region improves image processing quality and accuracy, thereby enhancing the user's viewing experience. Simultaneously, while enhancing image details, the computational load is precisely controlled to ensure processing efficiency.

[0146] Example 2

[0147] Figure 4 This is a flowchart illustrating the image detail enhancement method for the target region provided in Embodiment 2 of this application. Figure 4 As shown, the specific steps include the following:

[0148] S201, Obtain the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0149] S202, Obtain the target area of ​​the preset size.

[0150] A target region of a pre-defined size in a frame image refers to a region with a fixed size and shape that is pre-defined within the image, typically a rectangle or square. These target regions can be important targets, faces, or objects in the image. The size and shape of the target region can be set and adjusted according to specific application requirements. For example, in face recognition, the target region can include the position and size of the face.

[0151] The target region of a preset size in a frame image can be obtained through the following steps:

[0152] 1. First, you need to prepare a dataset containing target region annotations. For each image, you need to annotate the target region with a rectangle or other geometric shape.

[0153] 2. Use labeled datasets to train the object detection model.

[0154] 3. After the model training is complete, it can be used to detect targets in images.

[0155] 4. For each image, the model returns bounding boxes for one or more target regions and their corresponding scores.

[0156] 5. Since the model may detect multiple target regions, it is necessary to select the most suitable target region based on the score and other conditions. This can be done by comparing the size, position, and score of the bounding box.

[0157] 6. Obtain the target region by cropping the original image.

[0158] S203, Obtain key point information in the target area.

[0159] Key information can include the object's edge information, texture information, lighting information, and color information. In this solution, taking facial recognition as an example, eyebrows, eyes, and mouth can be used as key points.

[0160] Key point information in the target area can be obtained through the following steps:

[0161] 1. First, you need to prepare a dataset containing the target region and preprocess the data, such as adjusting the image size and enhancing the contrast.

[0162] 2. Use the dataset to train the keypoint detection model.

[0163] 3. After the model is trained, it can be used to detect key points in the target region. For each target region, the model will return a set of key point coordinates.

[0164] 4. Since the model may detect multiple sets of key points, key points can be selected by comparing their positions and scores.

[0165] 5. Visualize key information for better understanding and analysis.

[0166] Based on the above technical solutions, optionally, the key point information includes key point coordinates;

[0167] Based on the key point information and pre-set template transformation rules, the target region is transformed to obtain the input matrix of the target region, including:

[0168] Based on the key point coordinates and the pre-set template transformation rules, the target area is subjected to at least one of translation, rotation and scaling, and the missing area is identified in the processing result.

[0169] The missing regions are filled according to preset rules to obtain the input matrix of the target region.

[0170] In this scheme, keypoint coordinates can be the position coordinates of points marked as keypoints in an image, typically two-dimensional coordinates in pixels, used to describe the location of feature points of objects in the image. Taking face recognition as an example, the keypoint coordinates can be the coordinates of five keypoints: eyebrows, eyes, and mouth.

[0171] Taking face recognition as an example, the pre-set template transformation rules can refer to and map the coordinates of the face contained in the current image to the coordinates of the standard coordinate face, and after appropriate rotation transformation and scaling, obtain the aligned standard size of 512*512 face origin_face and the affine coordinates 5points_inv.

[0172] Missing regions refer to areas in the target region where information about certain pixels is lost or covered, resulting in blank or discontinuous areas in the image, making it impossible to reconstruct the complete information of the target region. In this solution, taking face recognition as an example, missing regions can be blank areas resulting from translation, rotation, and scaling of the face.

[0173] Taking facial recognition as an example, the following steps can be used to identify whether there are missing areas:

[0174] 1. Align and map the key point coordinates of the target area with the pre-set standard face key point coordinates to obtain the affine transformation matrix of the target area.

[0175] 2. Translate, rotate, and scale the target region according to the affine transformation matrix to obtain the input matrix of the processed target region.

[0176] 3. In the processed target region input matrix, check for any missing regions.

[0177] The preset rules can be the rules followed when filling missing regions, including zero-value filling, mean filling, nearest neighbor interpolation, bilinear interpolation, and content-based filling. In this solution, taking face recognition as an example, the preset rule can be to assign 0 values ​​to all blank areas to be filled during rotation.

[0178] The input matrix of the target region can refer to the matrix corresponding to the preprocessed target region image. In image detail enhancement, the input matrix of the target region is usually a two-dimensional matrix, where each element represents the brightness or color value of the corresponding pixel. The size of this matrix is ​​usually the same as the preset target region size. Taking face recognition as an example, the input matrix of the target region can be a two-dimensional matrix, where each element represents the grayscale value or color value of the corresponding pixel. The size of this matrix is ​​usually fixed and the same as the preset face size.

[0179] For missing regions in the target area, interpolation methods can be used to fill them. Interpolation is a method of inferring unknown data points based on known data points. It can estimate the value at an unknown location by using the position and value of known data points.

[0180] In this scheme, the above methods can make the input matrix more standardized and normalized, making subsequent algorithms and models easier to process and analyze. When applied to face recognition, aligning and normalizing face images can improve the accuracy and robustness of face recognition.

[0181] S204. Based on the key point information and the pre-set template transformation rules, the target region is transformed to obtain the input matrix of the target region.

[0182] To perform transformations on a target region, it is necessary to define template transformation rules or transformation matrices to describe the transformation. These rules or transformation matrices can be defined according to specific application scenarios and may include translation transformations, scaling transformations, rotation transformations, flip transformations, affine transformations, and perspective transformations, etc.

[0183] In this scheme, taking face recognition as an example, after determining that the video frame contains a face, the coordinates of the corresponding facial key points are detected. At the same time, affine transformation is used to rotate and scale the face in the video frame to obtain a face with standard position and fixed size (512*512). After numerical normalization, the input matrix required for network inference can be obtained.

[0184] S205, the input matrix is ​​input into the enhancement network model, and the enhancement result of the target region is generated based on the extracted multi-layer features, latent vectors and random signals.

[0185] S206, the enhancement result of the target region is subjected to the inverse transformation processing of the template transformation processing, and the inverse transformation processing result is fused with the original frame image to obtain the image detail enhancement result.

[0186] In this embodiment, preprocessing the target region can improve the accuracy and robustness of image detail enhancement. It also allows for adjustments and modifications based on application scenarios and requirements, enabling personalized image enhancement. Furthermore, by extracting key point information from the target region and setting template transformation rules, the processed image can be guaranteed to have good visualization effects and practicality.

[0187] Example 3

[0188] Figure 5 This is a schematic diagram of the image detail enhancement device for the target area provided in Embodiment 3 of this application. Figure 5 As shown, it specifically includes the following:

[0189] The acquisition module 301 is used to acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0190] The frame image processing module 302 is used to perform template transformation processing on the frame image of the video to be processed to obtain the input matrix of the target region if the image detail enhancement conditions are met.

[0191] The target region enhancement result generation module 303 is used to input the input matrix into the enhancement network model and generate the enhancement result of the target region based on the extracted multi-layer features, latent vectors and random signals.

[0192] The image detail enhancement result generation module 304 is used to perform inverse transformation processing on the enhancement result of the target region using the template transformation processing, and to fuse the inverse transformation processing result with the original frame image to obtain the image detail enhancement result.

[0193] In this embodiment, the acquisition module is used to acquire the video to be processed and determine whether the video meets the image detail enhancement conditions; the frame image processing module is used to perform a target region template transformation on the frame image of the video to be processed if the image detail enhancement conditions are met, to obtain an input matrix of the target region; the target region enhancement result generation module is used to input the input matrix into the enhancement network model, and generate the enhancement result of the target region based on the extracted multi-layer features, latent vectors, and random signals; the image detail enhancement result generation module is used to perform an inverse transformation on the enhancement result of the target region using the template transformation, and fuse the inverse transformation result with the original frame image to obtain the image detail enhancement result. Through the above-mentioned target region image detail enhancement device, the image processing quality and accuracy can be improved, thereby enhancing the user's viewing experience. At the same time, while performing image detail enhancement, the computational load is precisely controlled to ensure processing efficiency.

[0194] The image detail enhancement device for the target area provided in this application embodiment can achieve… Figure 1 as well as Figure 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0195] Example 4

[0196] Figure 6 This is a schematic diagram of the structure of an image detail enhancement device for a target area provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes a processor 401, a memory 402, an input device 403, and an output device 404; the number of processors 401 in the device can be one or more. Figure 6Taking a processor 401 as an example; the processor 401, memory 402, input device 403, and output device 404 in the device can be connected via a bus or other means. Figure 6 Taking a bus connection as an example, memory 402, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the image detail enhancement method for the target area in this embodiment. Processor 401 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in memory 402, thereby implementing the aforementioned image detail enhancement method for the target area. Input device 403 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. Output device 404 may include display devices such as a display screen.

[0197] This application embodiment also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image detail enhancement method for a target region described in the above embodiments, wherein the method includes:

[0198] Acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement.

[0199] If the image detail enhancement conditions are met, then the frame image of the video to be processed is subjected to template transformation processing of the target region to obtain the input matrix of the target region;

[0200] The input matrix is ​​fed into the enhancement network model, and the enhanced result of the target region is generated based on the extracted multi-layer features, latent vectors and random signals.

[0201] The enhancement result of the target region is subjected to the inverse transformation of the template transformation process, and the inverse transformation result is fused with the original frame image to obtain the image detail enhancement result.

[0202] It is worth noting that in the embodiments of the image detail enhancement device for the target area described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0203] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can perform an image detail enhancement method for a target region as described in an embodiment of this application. The program product can be implemented using any combination of one or more readable media.

Claims

1. A method for enhancing image details in a target region, characterized in that, include: Acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement. If the image detail enhancement conditions are met, the frame image of the video to be processed is subjected to template transformation processing of the target region to obtain the input matrix of the target region. The process includes: obtaining a target region of a preset size, obtaining key point information in the target region, and transforming the target region based on the key point information and a preset template transformation rule to obtain the input matrix of the target region. The input matrix is ​​fed into the enhancement network model, and the enhanced result of the target region is generated based on the extracted multi-layer features, latent vectors and random signals. The enhancement result of the target region is subjected to the inverse transformation of the template transformation process, and the inverse transformation result is fused with the original frame image to obtain the image detail enhancement result.

2. The image detail enhancement method for the target region according to claim 1, characterized in that, The key point information includes the coordinates of the key points; Based on the key point information and pre-set template transformation rules, the target region is transformed to obtain the input matrix of the target region, including: Based on the key point coordinates and the pre-set template transformation rules, the target area is processed by at least one of translation, rotation and scaling, and the missing area is identified in the processing result. The missing regions are filled according to preset rules to obtain the input matrix of the target region.

3. The image detail enhancement method for the target region according to claim 1, characterized in that, The input matrix is ​​fed into the enhancement network model, and the enhanced result of the target region is generated based on the extracted multi-layer features, latent vectors, and random signals, including: The input matrix is ​​fed into the convolutional layer of the augmented network model to obtain multi-layer features; The features output from the last convolutional layer are input into the linear connection layer to obtain the latent vector of the frame image; The multi-layer features, the latent vectors, and the pre-obtained random signal are input into the generator of the enhancement network model to obtain the enhancement result of the target region; wherein, the random signal is random noise that conforms to a Gaussian distribution obtained based on the signal generation network.

4. The image detail enhancement method for the target region according to claim 3, characterized in that, The enhanced network model also includes: A discriminator is used to form an adversarial neural network with the generator during training to distinguish between genuine and fake augmented results of the target region generated by the generator.

5. The image detail enhancement method for the target region according to claim 1, characterized in that, The enhancement results based on the target region are fused with the original frame image to obtain image detail enhancement results, including: The enhancement result of the target region is weighted and compared with the target region in the original frame image to obtain the weighted processing result. The segmentation objects in the weighted processing result are determined by using a segmentation algorithm; or, the segmentation objects in the enhanced result are determined by using a segmentation algorithm. The segmented objects in the weighted processing result or the segmented objects in the enhancement result are stitched together with the non-segmented object regions in the original frame image to obtain the image detail enhancement result.

6. The image detail enhancement method for the target region according to claim 5, characterized in that, The weighted processing result or the enhancement result both include the values ​​of the three channels in the color space; Accordingly, a segmentation algorithm is used to determine the segmentation objects in the weighted processing result; Alternatively, a segmentation algorithm may be used to determine the segmentation objects in the enhanced result, including: A segmentation algorithm is used to determine the segmentation object in the weighted processing result based on at least one channel value in the weighted processing result; Alternatively, a segmentation algorithm can be used to determine the segmentation objects in the enhancement result based on the value of at least one channel in the enhancement result.

7. The image detail enhancement method for the target region according to claim 5, characterized in that, Before concatenating the segmented objects in the weighted processing result or the segmented objects in the enhancement result with the non-segmented object regions in the original frame image, the method further includes: Obtain the transformation rules for template transformation processing; Based on the inverse operation of the transformation rule, and the segmented objects in the weighted processing result or the segmented objects in the enhancement result, the non-segmented object regions in the original frame image are determined.

8. An image detail enhancement device for a target region, characterized in that, include: The acquisition module is used to acquire the video to be processed and determine whether the video to be processed meets the conditions for image detail enhancement. The frame image processing module is used to perform template transformation processing on the frame image of the video to be processed if the image detail enhancement conditions are met, so as to obtain the input matrix of the target region. The module includes: obtaining a target region of a preset size, obtaining key point information in the target region, and performing transformation processing on the target region based on the key point information and a preset template transformation rule to obtain the input matrix of the target region. The target region enhancement result generation module is used to input the input matrix into the enhancement network model and generate the enhancement result of the target region based on the extracted multi-layer features, latent vectors and random signals. The image detail enhancement result generation module is used to perform inverse transformation processing on the enhancement result of the target region using the template transformation processing, and to fuse the inverse transformation processing result with the original frame image to obtain the image detail enhancement result.

9. An image detail enhancement device for a target region, the device comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image detail enhancement method for the target region as described in any one of claims 1-8.

10. A storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform an image detail enhancement method for a target region as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image detail enhancement method for the target region as described in any one of claims 1-7.

Images