Image enhancement method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and provides an image enhancement method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the edge detection based on a target image, and obtaining a contour feature; based on the contour features and the label features, performing image enhancement on the target image to obtain a target enhanced image; the label feature is constructed based on a target detection label carried by the target image, the target enhanced image and the target image carry the same target detection label, and the target detection label comprises coordinates and categories of detection frames corresponding to targets in the image, so that the defects that in a traditional scheme, the generated image is difficult to accurately control, and the image quality is poor are overcome. According to the method, the defects that the target detection task needs to be performed, and the generated image lacks authenticity and naturalness are overcome, high-quality and diversified image enhancement is realized, and the category and position of the object in the generated image can be accurately controlled, so that a large number of high-quality data sets can be provided for the target detection task, and the performance of a target detection model and the generalization ability are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image enhancement method, device, electronic device and storage medium. Background Art

[0002] In recent years, target detection technology has developed towards multi-scale, small target, and real-time, such as RetinaNet, FeaturePyramid Network, etc. Although the introduction of these target detection algorithms has improved the efficiency and accuracy of target detection, they still face problems such as data scarcity, category imbalance, and single scene, which seriously restricts the generalization ability of the model and the actual application effect.

[0003] Current object detection data enhancement methods, such as Simple Copy-Paste, have alleviated the problem of data scarcity to a certain extent and improved model performance, but their limitations are also obvious. Most of these methods rely on simple image processing techniques, such as cutting and pasting the target object into a new background environment. Although simple and fast, they often ignore the complex interaction between the target object and the background environment, resulting in the lack of authenticity and naturalness of the newly generated images.

[0004] In addition, these methods also have shortcomings in controlling data diversity. It is difficult to accurately control the diversity of generated images and cannot effectively simulate different lighting conditions, weather conditions, and background environments, thus limiting the scene adaptability of enhanced data. This results in the model's performance often being unsatisfactory when faced with complex and changing real-world scenarios. Summary of the invention

[0005] The present invention provides an image enhancement method, device, electronic device and storage medium to solve the problem that the data enhancement process in the prior art lacks authenticity and diversity, resulting in the generated images not being realistic and natural enough, the application scenarios being limited, and further affecting the generalization ability and application effect of the model, thereby realizing intelligent, refined and diversified image enhancement.

[0006] The present invention provides an image enhancement method, comprising: Determining a target image, wherein the target image has a target detection label; Perform edge detection based on the target image to obtain contour features; Based on the contour features and the label features, the target image is enhanced to obtain a target enhanced image; The target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0007] According to an image enhancement method provided by the present invention, the label feature is determined based on the following steps: Encoding the category of the detection frame of each target in the target detection label to obtain the category prompt word feature of each target; Embed the coordinates of the detection frame of each target in the target detection label to obtain the detection frame position features of each target; The category prompt word features and the detection frame position features of each target are fused to obtain the label features of the target image.

[0008] According to an image enhancement method provided by the present invention, embedding the coordinates of the detection frame of each target in the target detection label to obtain the position features of the detection frame of each target includes: Extracting the diagonal coordinates of the detection box of each target in the target detection label; Performing Fourier embedding on the diagonal coordinates to obtain detection frame position features of each target; The diagonal coordinates are the coordinates of two vertices on the diagonal line of the corresponding detection frame.

[0009] According to an image enhancement method provided by the present invention, the target image is enhanced based on the contour feature and the label feature to obtain a target enhanced image, including: Based on the contour feature, the label feature, and the multiple noise images, the target image is enhanced to obtain multiple initial enhanced images; each initial enhanced image carries the target detection label; each noise image is obtained by random sampling; Determine, based on the detection frame of each target in any initial enhanced image, an enhanced region corresponding to each target; Determining a generation score of any of the initial enhanced images based on the enhanced regions corresponding to the respective targets in the any of the initial enhanced images and the label features of the target images; Based on the generation scores of the initial enhanced images, the target enhanced image is obtained by screening the initial enhanced images.

[0010] According to an image enhancement method provided by the present invention, the target image is enhanced based on the contour feature and the label feature to obtain a target enhanced image, including: Inputting the contour feature and the label feature into an image enhancement model to obtain a target enhanced image output by the image enhancement model; The image enhancement model is obtained by training on the basis of the diffusion model using sample images and sample target detection labels of the sample images.

[0011] According to an image enhancement method provided by the present invention, the image enhancement model is trained based on the following steps: Based on the diffusion model, construct a first enhancement model and a second enhancement model, and based on the first enhancement model and the second enhancement model, construct an initial enhancement model; Based on the sample image, the sample target detection label of the sample image, and the sample prompt text of each target in the sample image, the initial enhancement model is trained to obtain the image enhancement model; Among them, the first enhancement model and the second enhancement model in the initial enhancement model are connected through a convolutional layer; the first enhancement model is constructed by adding layers to the attention mechanism in the diffusion model, and in the first enhancement model, except for the parameters of the gated self-attention layer added by the layer addition, other parameters are not updated during the training process.

[0012] According to an image enhancement method provided by the present invention, the layer addition is to add a gated self-attention layer between each self-attention layer and each cross-attention layer in the attention mechanism in the diffusion model, and the input of the gated self-attention layer includes the sample label feature corresponding to the sample target detection label and the sample contour feature corresponding to the sample image; The second enhancement model is constructed by removing the decoding layer based on the diffusion model.

[0013] The present invention also provides an image enhancement device, comprising: A determination unit, used to determine a target image, wherein the target image has a target detection label; A detection unit, used for performing edge detection based on the target image to obtain contour features; An enhancement unit, configured to perform image enhancement on the target image based on the contour feature and the label feature to obtain a target enhanced image; The target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described image enhancement methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image enhancement method as described above is implemented.

[0016] The image enhancement method, device, electronic device and storage medium provided by the present invention perform edge detection on a target image to obtain contour features, perform image enhancement based on the contour features and the label features of the target detection label carried by the target image, and obtain a target enhanced image with the same target detection label as the target image; the target detection label includes the coordinates and category of the detection frame of each target in the corresponding image, which overcomes the defects of the traditional scheme that it is difficult to accurately control the generated image and the lack of authenticity and naturalness of the generated image, not only achieves high-quality and diversified image enhancement, but also can accurately control the category and position of objects in the generated image, thereby providing a large number of high-quality data sets for target detection tasks, which in turn helps to improve the performance of the target detection model and enhance its generalization ability, so that it is more suitable for complex tasks and performs better in specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a schematic diagram of the process of the image enhancement method provided by the present invention; Figure 2 It is the overall framework diagram of the image enhancement method provided by the present invention; Figure 3 It is a structural schematic diagram of the initial enhancement model provided by the present invention; Figure 4 It is a schematic diagram of adding a layer in the attention mechanism provided by the present invention; Figure 5 is a schematic structural diagram of an image enhancement device provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In the field of target detection, with the continuous evolution of algorithms, the introduction of advanced models such as RetinaNet and Feature Pyramid Network has significantly improved the accuracy and efficiency of target detection, especially in processing multi-scale targets and small targets. In addition, it has also contributed to the improvement of real-time requirements, making target detection technology show a wide range of application potential in many fields such as autonomous driving, video surveillance, and medical image analysis.

[0021] However, despite the significant progress these algorithms have made in theory, they still face many challenges in practical applications, among which data scarcity, category imbalance, and single scene are particularly prominent. These problems seriously restrict the generalization ability of target detection models and affect the performance of models in different environments and conditions. In order to solve these problems, researchers continue to explore effective data enhancement methods in order to improve the performance of models by increasing the diversity and richness of training data.

[0022] Current target detection data enhancement methods, such as Simple Copy-Paste, have alleviated the problem of data scarcity to a certain extent and improved the performance of the model, but they still have many limitations. First, these methods mostly rely on simple image processing techniques, such as cutting and pasting, to generate new images by pasting the target object into a new background. Although this method is simple and fast, it ignores the complex interaction between the target object and the background environment, resulting in the generated images often lacking authenticity and naturalness, and it is difficult to effectively simulate the complex situations in real scenes. Secondly, these methods also have shortcomings in controlling data diversity, that is, it is difficult to accurately control the diversity of generated images, and it is impossible to effectively simulate different lighting conditions, weather conditions, background environments, etc., which limits the scene adaptability of the enhanced data, resulting in the model's performance often being unsatisfactory when facing complex and changeable actual scenes.

[0023] In this regard, the present invention provides an image enhancement method, which aims to overcome the limitations of current data enhancement methods. The contour features and label features of the image are respectively obtained through the target image and its corresponding target detection label, and image enhancement is performed based on the above to obtain a target enhanced image. In this way, the category and position of the target object in the generated image can be accurately controlled to achieve high-quality and diversified images, thereby providing a large amount of high-quality data sets for target detection tasks, thereby effectively improving the performance of the target detection model and enhancing its generalization ability, providing more solid data support for the application of target detection technology.

[0024] Figure 1 It is a flow chart of the image enhancement method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, determining a target image, the target image having a target detection label; Step 120, performing edge detection based on the target image to obtain contour features; Step 130, based on the contour features and the label features, the target image is enhanced to obtain a target enhanced image; Among them, the target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0025] Specifically, considering the two problems existing in the current target detection data enhancement method, namely, it is difficult to accurately control the generated image, the diversity of the generated image cannot be guaranteed, the different light background environments cannot be simulated, the scene adaptability is limited, and the generated image lacks authenticity and naturalness, in an embodiment of the present invention, it is proposed to perform image enhancement through the target image and the target detection label it carries, so that the category and position of the object in the generated image can be accurately controlled, and various real and complex scenes (such as lighting conditions, weather conditions, background environment, etc.) can be simulated, thereby realizing intelligent and diversified data enhancement of the original image, and obtaining real, diverse and natural generated images, which solves the problems of the current data enhancement method being difficult to accurately control and the generated images lacking authenticity and naturalness, and can provide a large number of high-quality data sets for target detection tasks, thereby significantly improving the performance and generalization ability of the target detection model.

[0026] It should be noted that the method provided by the present invention can not only be applied to target detection tasks to provide training data for target detection models, but can also be applied to other image perception tasks, such as semantic segmentation, image classification, target tracking, etc. It can provide data support for many types of image perception tasks and provide assistance for the continuous progress of image perception tasks.

[0027] Specifically, in the embodiment of the present invention, before performing image enhancement, it is first necessary to determine the image to be enhanced, which is referred to as the target image. The target image can be acquired by various image acquisition devices, such as ordinary cameras, wide-angle cameras, depth cameras, infrared cameras, multi-spectral cameras, etc.; it can also be obtained by searching and downloading on the Internet, and can also be generated by image generation algorithms, models, etc., which are not specifically limited in the embodiment of the present invention.

[0028] Here, the target image may be an image in various fields, such as security monitoring, medical imaging, industrial production, agricultural production, retail warehouses, etc. In the embodiment of the present invention, when acquiring the target image, the original image and its corresponding annotation information may be acquired first, and the annotation information may include the category of each target in the original image and the coordinates of the detection frame, and then the original image and its annotation information may be subjected to image transformation, such as random cropping, size transformation, etc., so as to obtain the transformed image and its corresponding annotation information, and the image and its annotation information at this time are the target image and its corresponding target detection label.

[0029] After obtaining the target image, in the embodiment of the present invention, the target image can be processed to extract the edge information of the object therein, thereby obtaining the contour feature. Here, specifically, the target image can be processed by a visual prior generator to extract the edge contour information, thereby obtaining the contour feature corresponding to the target image.

[0030] Among them, the visual prior generator can be an edge detector, such as HED (Holistically-Nested Edge Detection), and the process of contour feature extraction is the process of calculating the edge information of the object in the target image through the edge detection operator of the edge detector, so as to obtain the overall contour feature. Here, contour feature extraction through the edge detector can better balance the visual diversity and the quality of the bounding box, making the extracted contour features more obvious.

[0031] Furthermore, after obtaining the contour feature, in an embodiment of the present invention, image enhancement can be performed based on the contour feature and combined with the label feature to obtain a generated enhanced image, that is, an enhanced image corresponding to the target image, called a target enhanced image.

[0032] Specifically, here we can first draw the contour according to the contour features to draw an image contour that is consistent with the current contour features; then, we can assign colors and fill textures to each target in the contour according to the category and position of the target represented by the label features, so that we can get a preliminary generated image; after that, we can also perform enhancement processing on the preliminary generated image, such as color enhancement, contrast enhancement, detail enhancement, etc., to make it clearer, more vivid, more detailed, etc., and finally get a target enhanced image.

[0033] Among them, the label feature is constructed according to the target detection label corresponding to the target image. Since the target detection label includes the category of each target in the image and the coordinates of the detection frame, when constructing the label feature that can represent the information contained in the target detection label, features can be extracted from the two types of information respectively, and the overall label feature is constructed on this basis.

[0034] Specifically, the target image can be enhanced based on the label features and the contour features to obtain the generated target enhanced image. That is, the image generation is guided by the label features based on the contour features to generate an image with the same contour and the same object category and position as represented by the label features, but different style, environment, etc.

[0035] It should be noted that, although the target enhanced image obtained through the above image enhancement processing has the same contour features as the original target image and carries the same target detection label, the visual features of the two are obviously different. For example, it can be an image with different lighting conditions, weather conditions, background environment, etc., or an image of other scenes directly simulated, or a style transformation or migration is performed, etc., which is not specifically limited in the embodiments of the present invention.

[0036] In an embodiment of the present invention, images and target detection labels are used as input, and image enhancement processing is performed through deep learning technology to output a series of target enhanced images that keep the position and category of the original target unchanged but present a variety of target styles, environmental conditions, etc., thereby achieving intelligent and diversified image enhancement. It not only retains the key target information in the original target image, but can also simulate various complex real scenes, greatly improving the diversity, authenticity and naturalness of the generated images, and is a high-quality data reserve for target detection tasks.

[0037] In the embodiments of the present invention, it can flexibly adapt to different types of image inputs, and by optimizing the input images, an enhanced image can be obtained. It can be applied to multiple fields that require high-precision target detection, such as autonomous driving, security monitoring, medical image analysis, industrial quality inspection, intelligent agriculture, retail, etc., to provide high-quality data support.

[0038] In addition, the method provided by the embodiment of the present invention can also be combined with other advanced target detection technologies, such as multi-scale detection, feature pyramid network, attention mechanism, etc., to further improve detection performance and accuracy. For example, if the method provided by the present invention is integrated into the training process of the target detection task, the dynamic generation and update of training samples can be realized, so that the model can continuously learn and adapt to new scenes and target categories, thereby laying a solid foundation for the construction of a more intelligent and adaptable computer vision system.

[0039] The image enhancement method provided by the present invention performs edge detection on a target image to obtain contour features, performs image enhancement based on the contour features and label features of a target detection label carried by the target image, and obtains a target enhanced image having the same target detection label as the target image; the target detection label includes the coordinates and category of the detection frame of each target in the corresponding image, which overcomes the defects of the traditional scheme that it is difficult to accurately control the generated image and the lack of authenticity and naturalness of the generated image, not only realizes high-quality and diversified image enhancement, but also can accurately control the category and position of objects in the generated image, thereby providing a large number of high-quality data sets for target detection tasks, thereby helping to improve the performance of the target detection model and enhance its generalization ability, so that it is more suitable for complex tasks and performs better in specific tasks.

[0040] Based on the above embodiment, the tag features are determined based on the following steps: Encode the category of the detection box of each target in the target detection label to obtain the category prompt word features of each target; Embed the coordinates of the detection frame of each target in the target detection label to obtain the position features of the detection frame of each target; The category prompt word features and detection frame position features of each target are fused to obtain the label features of the target image.

[0041] Specifically, the determination of the label features of the target image can be achieved through the following steps: Since the target detection label is composed of a detection frame-category pair, and the category and detection frame represent two different types of information, the detection frame reflects the position of the target, and the category represents the type of object to which the target belongs. Therefore, in order to better reflect the target information, in an embodiment of the present invention, these two types of information can be processed separately to obtain the features corresponding to each of the two types of information, and then the overall features are constructed on this basis to obtain the final features, that is, the label features of the target image.

[0042] Specifically, here, the prompt word is first constructed based on the category of the detection box of each target in the target detection label to obtain the category prompt word feature. That is, for each target in the target image, its category is first obtained, and then the category of the target is encoded, such as using the multimodal model CLIP (Contrastive Language–Image Pre-training) to encode it, and encode it into features, thereby obtaining the category prompt word feature, which can be expressed as , Indicates In the target image The category cue word features of the target.

[0043] At the same time, the position features can be constructed based on the coordinates of the detection frame of each target in the target detection label to obtain the detection frame position features of each target. That is, for each target in the target image, the coordinates of the detection frame are first obtained, and then the coordinates are embedded to represent them as the embedding vector of the coordinates, thereby obtaining the detection frame position features of each target, which can be expressed as , Indicates In the target image The detection box position features of each target.

[0044] After that, the label features of the target image can be determined based on the category prompt word features and detection frame position features of each target in the target image; that is, the features obtained by extracting the two types of information in the target detection label can be fused to obtain the features at the overall level, that is, the category prompt word features and detection frame position features of each target in the target image are fused to obtain the label features of the target image.

[0045] Based on the above embodiment, the label feature of the target image can be expressed by the following formula:

[0046]

[0047] In the formula, Indicates The label features of the target image, Indicates The number of objects in the target image, Indicates The label features of the target, and Respectively represent In the target image The category cue word features of the target and the The detection box position features of each target, Multilayer Perceptron represents a multilayer perceptron.

[0048] In an embodiment of the present invention, on the basis of the target detection label, the overall label feature is obtained through prompt word construction, position feature extraction, and feature fusion, and on this basis, combined with the contour feature of the target image obtained by the visual prior generator, the target image is enhanced together to obtain multiple target enhanced images with the same contour feature and target detection label, thereby achieving high-quality data enhancement, being able to generate diversified images without changing key target information, and being able to simulate different lighting conditions, weather conditions, background environments, etc., greatly improving the scene adaptability of the enhanced data.

[0049] Based on the above embodiment, the coordinates of the detection frame of each target in the target detection tag are embedded to obtain the position features of the detection frame of each target, including: Extract the diagonal coordinates of the detection box of each target in the target detection label; Perform Fourier embedding on the diagonal coordinates to obtain the detection frame position features of each target; The diagonal coordinates are the coordinates of the two vertices on the diagonal line of the corresponding detection box.

[0050] Specifically, the above process of embedding the coordinates of the detection frame of each target in the target detection label to obtain the position features of the detection frame of each target may specifically include: First, the diagonal coordinates of the detection frame of each target in the target image can be determined according to the target detection label. The diagonal coordinates here are the coordinates of the two end points of the diagonal line of the detection frame. Since the target detection label already contains the coordinates of the detection frame, the coordinates of the upper left corner and the lower right corner can be directly obtained here, or the coordinates of the upper right corner and the lower left corner can be obtained and directly used as the diagonal coordinates of the detection frame of each target.

[0051] Then, the diagonal coordinates of each detection box can be Fourier embedded to represent them as an embedding vector of the coordinates, thereby obtaining the detection box position features of each target; that is, Fourier embedding can be performed on the basis of the diagonal coordinates of each detection box, and the embedding vector of each detection box position can be obtained after Fourier embedding representation.

[0052] Based on the above embodiment, step 130 includes: Based on the contour features, label features, and multiple noise images, the target image is enhanced to obtain multiple initial enhanced images; each initial enhanced image has a target detection label; each noise image is obtained by random sampling; Based on the detection frame of each target in any initial enhanced image, determining the enhanced region corresponding to each target; Determining a generation score of the initial enhanced image based on the enhanced regions corresponding to the targets in the initial enhanced image and the label features of the target image; Based on the generation scores of the initial enhanced images, the target enhanced image is obtained by screening the initial enhanced images.

[0053] Specifically, in the above step 130, the process of performing image enhancement on the target image according to the contour features and the label features to obtain the target enhanced image may include the following steps: Figure 2 It is the overall framework diagram of the image enhancement method provided by the present invention, such as Figure 2 As shown, after the contour features and label features of the target image are obtained according to the visual prior generator and the prompt word constructor, an input image can be provided when performing image enhancement. The input image can be a blank image or a noise image. Preferably, in the embodiment of the present invention, a noise image is selected as the input image, and the image enhancement process is performed on the basis of the noise image. The specific goal is to restore the noise image to a clear image that is consistent with the contour and target detection label of the target image, thereby generating a diverse target enhanced image.

[0054] Specifically, a noise image may be obtained first, and the noise image may be obtained by random sampling, specifically, random sampling from Gaussian distribution, standard normal distribution, etc., so as to obtain a noise image, and this may be used as input for image enhancement. Specifically, the noise image may be preprocessed first, for example, by reducing noise by Gaussian filtering, etc., to improve the visual effect; then, the contour features may be matched with the preprocessed noise image, such as by performing contour matching on the basis of the preprocessed noise image by image registration, feature point matching, etc.; then, color filling, texture replacement, etc. may be performed for the target in the matched contour according to the label features, and finally, enhancement processing such as color enhancement, contrast enhancement, detail enhancement, etc. may be performed to make it clearer, more vivid, more detailed, etc., to obtain a diverse target enhanced image, that is, an image with the same contour features as the target image, and consistent with the object category and position represented by the label features, but with different visual features.

[0055] However, it is worth noting that since the final image is generated based on the noise image, which is obtained by random sampling, when the noise obtained by sampling is different, even if the given conditions (contour features and label features) are the same, the final generated image must be different. In order to achieve diversified image enhancement, multiple noises are usually randomly sampled in the sampling process. Therefore, there are often multiple images generated, and the quality of these multiple images cannot be guaranteed one by one. They need to be further screened and confirmed. Therefore, the initially generated image can be called the initial enhanced image, which has the same target detection label as the target image.

[0056] Furthermore, after obtaining multiple initial enhanced images, in an embodiment of the present invention, these multiple initial enhanced images need to be screened to screen out poor quality images and obtain high-quality images to serve as reserve data for subsequent target detection tasks, thereby improving the performance and generalization ability of the target detection model, and further optimizing its performance in target detection tasks.

[0057] Specifically, for any image in the generated initial enhanced image, the initial enhanced image can first be cropped according to the detection frame of each target to crop out the image corresponding to each detection frame, that is, the enhanced area corresponding to each target can be cropped from the initial enhanced image according to the detection frame.

[0058] Subsequently, the quality of the generation of the target image in the image enhancement task can be scored based on the enhanced areas corresponding to the targets cropped from the initial enhanced image and the categories of the targets in the target detection labels, so as to judge whether the generated enhanced areas of the targets match and correspond to the originally recorded categories, thereby obtaining the similarity matching scores corresponding to the targets.

[0059] Preferably, the multimodal model CLIP can be used to perform similarity matching scoring, that is, similarity calculation can be performed based on the enhanced area and category prompt word features corresponding to each target, thereby obtaining a similarity matching score. , , Indicates The initial enhanced image corresponding to the target image The enhanced area corresponding to the target, Indicates the first The category cue word features of the target.

[0060] After that, the similarity matching scores of the targets in the initial enhanced image can be counted to obtain the image-level score. Here, the similarity matching scores of the targets in the initial enhanced image can be fused. The specific fusion method can be averaging, weighted summing, or other methods. The embodiment of the present invention does not make specific limitations. Finally, the score of the initial enhanced image can be obtained, that is, the generated score.

[0061] Furthermore, image screening can be performed according to the generated score to screen out high-quality images from all generated initial enhanced images and remove low-quality images.

[0062] Specifically, here, all the initial enhanced images can be sorted in order of generation scores from high to low (or from low to high) to obtain an image sequence; then, the inferior images can be filtered out from the image sequence according to a preset filtering ratio, and the high-quality images can be retained as the final target enhanced images. Here, specifically, the initial enhanced images in the last 70% or last 80% (or in the first 70% or first 80%) of the image sequence can be filtered out according to a filtering ratio such as 10:3 (0.3) or 5:1 (0.2), and the first 30% or first 20% (last 30% or last 20%) can be retained, and the retained images can be used as the final target enhanced images.

[0063] Based on the above embodiment, step 130 includes: Input the contour features and label features into the image enhancement model to obtain the target enhanced image output by the image enhancement model; Among them, the image enhancement model is trained on the basis of the diffusion model by applying sample images and sample target detection labels of the sample images.

[0064] Specifically, in step 130, the process of performing image enhancement on the target image according to the contour features and label features can be implemented through an image enhancement model. Specifically, the contour features and label features of the target image, as well as the noise image, can be input into the image enhancement model together, so that the image enhancement model, under the guidance of the input information, restores the noise to a clear image and outputs it, thereby obtaining a target enhanced image.

[0065] It is worth noting that in the embodiment of the present invention, the image enhancement model used is constructed on the basis of diffusion models (such as Stable Diffusion, Denoising Diffusion Probabilistic Models, etc.). After the initial model in the training process is obtained through the diffusion model, in order to better improve the image enhancement capability of the model and enable it to perform better in practical applications, in the embodiment of the present invention, the initial model needs to be trained, that is, the sample images collected in advance and the sample target detection labels corresponding to the sample images are applied, and the parameters of the initial model are iterated to obtain a trained image enhancement model, so as to better ensure the authenticity and diversity of the target enhanced images obtained in the end.

[0066] The sample images may include images in multiple scenes and multiple fields.

[0067] In an embodiment of the present invention, an image and a target detection label are taken as input, and a deep learning technology and an image enhancement model constructed on the basis of a diffusion model are used to output a series of target enhanced images that maintain the object positions and categories in the original image but present a variety of target enhanced images in terms of image style, environmental conditions, etc., thereby realizing intelligent and diversified image enhancement, which not only retains the key target information in the original image, but can also simulate various complex real scenes, greatly improving the diversity and authenticity of the generated images.

[0068] Based on the above embodiment, the image enhancement model is trained based on the following steps: Based on the diffusion model, a first enhancement model and a second enhancement model are constructed, and based on the first enhancement model and the second enhancement model, an initial enhancement model is constructed; Based on the sample images, sample target detection labels of the sample images, and sample prompt texts of each target in the sample images, an initial enhancement model is trained to obtain an image enhancement model; Among them, the first enhancement model and the second enhancement model in the initial enhancement model are connected through a convolutional layer; the first enhancement model is constructed by adding layers to the attention mechanism in the diffusion model, and in the first enhancement model, except for the parameters of the gated self-attention layer added by the layer addition, other parameters are not updated during the training process.

[0069] Specifically, the training process of the image enhancement model may include: First, two models, namely the first enhanced model and the second enhanced model, can be constructed based on the diffusion model. Here, the Stable Diffusion model is specifically selected as the basis, and the first enhanced model and the second enhanced model are constructed based on this.

[0070] Next, based on the first enhancement model and the second enhancement model, the initial model in the training process, that is, the initial enhancement model, can be constructed; here, specifically, a convolutional layer can be used to connect the first enhancement model and the second enhancement model, and the first enhancement model and the second enhancement model can be connected through multiple convolutional layers to construct the initial enhancement model.

[0071] It should be noted that most of the parameters in the first enhanced model here do not participate in the training, that is, they are not updated during the training process; the parameters in the second enhanced model participate in the training.

[0072] Figure 3 It is a structural schematic diagram of the initial enhancement model provided by the present invention, such as Figure 3As shown in Figure 1, the initial enhanced model retains two copies of the original diffusion model, one of which (the first enhanced model) freezes most of the parameters; the other copy (the second enhanced model) only contains the encoding layer and the intermediate layer in the original diffusion model, and the parameters are trainable; both are initialized to zero. Convolutional connections are made through Convolution connects the decoding layer in the first enhancement model and the encoding layer in the second enhancement model through another The convolution connects the middle layer in the first enhancement model and the middle layer in the second enhancement model, thus realizing the overall image construction. Figure 3 The red path in the middle figure is the path for gradient update during training, while the black path is the flow path during actual use. Figure 3 middle represents an image that is entirely noise, It means The image after one iteration.

[0073] However, it is worth noting that compared to the original diffusion model that can only receive text input and generates images based on text guidance, in the embodiment of the present invention, target detection labels and contour features are added to the input. And precisely because of the addition of input, it is also necessary to make structural improvements to the original diffusion model. This improvement specifically occurs in the first enhanced model. Specifically, on the basis of the original diffusion model, its attention mechanism is improved, and a gated self-attention layer is added to realize the integration of the information contained in the target detection label to obtain the first enhanced model. In addition, in the first enhanced model, except for the parameters of the newly added gated self-attention layer, the parameters of all other layers are frozen and do not participate in the update during the training process. That is, Figure 3 The partially trainable layers in represent the trainable parameters of the gated self-attention layer.

[0074] After the initial enhancement model is constructed, in an embodiment of the present invention, the initial enhancement model can be trained to obtain a trained image enhancement model.

[0075] Specifically, the initial enhancement model can be trained by applying sample images, sample target detection labels of sample images, and sample prompt texts of each target in the sample images; that is, model training is carried out on the basis of sample images, guided by sample label features of sample target detection labels and sample prompt texts, so that it has good image enhancement capabilities and can output high-quality images, and finally a trained image enhancement model can be obtained.

[0076] It should be noted that the sample prompt text is just a simple descriptive text. For example, when the sample image contains a target table, the sample prompt text can be a simple descriptive sentence such as "generate a table". The image enhancement process is more based on the sample target detection label and the visual features provided by the sample image. The sample prompt text only ensures that the model can follow the execution and avoid obvious errors.

[0077] Based on the above embodiment, the layer addition is to add a gated self-attention layer between each self-attention layer and each cross-attention layer in the attention mechanism in the diffusion model, and the input of the gated self-attention layer includes the sample label feature corresponding to the sample target detection label and the sample contour feature corresponding to the sample image; The second enhanced model is constructed by removing the decoding layer based on the diffusion model.

[0078] Figure 4 is a schematic diagram of the middle layer addition of the attention mechanism provided by the present invention, such as Figure 4 As shown in the figure, when improving the attention mechanism in the diffusion model, a gated self-attention layer is added to all the places in the original diffusion model where text information needs to be fused. That is, a gated self-attention layer is added between the self-attention layer and the cross-attention layer in the attention mechanism. The gated self-attention layer here is actually a copy of the cross-attention layer, except that the input text becomes a label; that is, the input of the gated self-attention layer includes the sample label feature corresponding to the sample target detection label and the sample contour feature corresponding to the sample image.

[0079] The second enhanced model is obtained by removing the decoding layer based on the diffusion model, that is, only the coding layer and the middle layer in the original diffusion model are retained, and the second enhanced model is obtained thereby.

[0080] Based on the above embodiments, see Figure 4 It can be seen that the gated self-attention layer can be expressed as:

[0081] In the formula, represents the sample contour feature, are model parameters, represents the sample label feature, represents the self-attention layer.

[0082] The image enhancement device provided by the present invention is described below. The image enhancement device described below and the image enhancement method described above can be referred to each other.

[0083] Figure 5 is a schematic diagram of the structure of the image enhancement device provided by the present invention, such as Figure 5 As shown, the device comprises: A determination unit 510, configured to determine a target image, wherein the target image has a target detection label; A detection unit 520, configured to perform edge detection based on the target image to obtain contour features; An enhancement unit 530, configured to perform image enhancement on the target image based on the contour feature and the label feature to obtain a target enhanced image; The target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0084] The image enhancement device provided by the present invention performs edge detection on a target image to obtain contour features, performs image enhancement based on the contour features and label features of a target detection label carried by the target image, and obtains a target enhanced image having the same target detection label as the target image; the target detection label includes the coordinates and categories of the detection frames of each target in the corresponding image, which overcomes the defects of the traditional scheme that it is difficult to accurately control the generated image, and the generated image lacks authenticity and naturalness, and not only achieves high-quality and diversified image enhancement, but also can accurately control the category and position of objects in the generated image, thereby providing a large number of high-quality data sets for target detection tasks, thereby helping to improve the performance of the target detection model and enhance its generalization ability, so that it is more suitable for complex tasks and performs better in specific tasks.

[0085] Based on the above embodiment, the device further includes a label feature determination unit, which is used to: Encoding the category of the detection frame of each target in the target detection label to obtain the category prompt word feature of each target; Embed the coordinates of the detection frame of each target in the target detection label to obtain the detection frame position features of each target; The category prompt word features and the detection frame position features of each target are fused to obtain the label features of the target image.

[0086] Based on the above embodiment, the tag feature determination unit is used to: Extracting the diagonal coordinates of the detection box of each target in the target detection label; Performing Fourier embedding on the diagonal coordinates to obtain detection frame position features of each target; The diagonal coordinates are the coordinates of two vertices on the diagonal line of the corresponding detection frame.

[0087] Based on the above embodiment, the enhancement unit 530 is used for: Based on the contour feature, the label feature, and the multiple noise images, the target image is enhanced to obtain multiple initial enhanced images; each initial enhanced image carries the target detection label; each noise image is obtained by random sampling; Determine, based on the detection frame of each target in any initial enhanced image, an enhanced region corresponding to each target; Determining a generation score of any of the initial enhanced images based on the enhanced regions corresponding to the respective targets in the any of the initial enhanced images and the label features of the target images; Based on the generation scores of the initial enhanced images, the target enhanced image is obtained by screening the initial enhanced images.

[0088] Based on the above embodiment, the enhancement unit 530 is used for: Inputting the contour feature and the label feature into an image enhancement model to obtain a target enhanced image output by the image enhancement model; The image enhancement model is obtained by training on the basis of the diffusion model using sample images and sample target detection labels of the sample images.

[0089] Based on the above embodiment, the device further includes a model training unit, which is used to: Based on the diffusion model, construct a first enhancement model and a second enhancement model, and based on the first enhancement model and the second enhancement model, construct an initial enhancement model; Based on the sample image, the sample target detection label of the sample image, and the sample prompt text of each target in the sample image, the initial enhancement model is trained to obtain the image enhancement model; Among them, the first enhancement model and the second enhancement model in the initial enhancement model are connected through a convolutional layer; the first enhancement model is constructed by adding layers to the attention mechanism in the diffusion model, and in the first enhancement model, except for the parameters of the gated self-attention layer added by the layer addition, other parameters are not updated during the training process.

[0090] Based on the above embodiment, the layer addition is to add a gated self-attention layer between each self-attention layer and each cross-attention layer in the attention mechanism in the diffusion model, and the input of the gated self-attention layer includes the sample label feature corresponding to the sample target detection label and the sample contour feature corresponding to the sample image; The second enhancement model is constructed by removing the decoding layer based on the diffusion model.

[0091] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image enhancement method, which includes: determining a target image, wherein the target image has a target detection label; performing edge detection based on the target image to obtain contour features; performing image enhancement on the target image based on the contour features and the label features to obtain a target enhanced image; wherein the target enhanced image and the target image have the same target detection label, wherein the target detection label includes the coordinates and categories of the detection boxes of each target in the corresponding image, and the label features are constructed based on the target detection label.

[0092] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image enhancement method provided by the above-mentioned methods, and the method includes: determining a target image, wherein the target image has a target detection label; performing edge detection based on the target image to obtain contour features; performing image enhancement on the target image based on the contour features and label features to obtain a target enhanced image; wherein the target enhanced image and the target image have the same target detection label, and the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image enhancement method provided by the above-mentioned methods, the method comprising: determining a target image, the target image having a target detection label; performing edge detection based on the target image to obtain contour features; performing image enhancement on the target image based on the contour features and label features to obtain a target enhanced image; wherein the target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement method, characterized in that: include: Determining a target image, wherein the target image has a target detection label; Perform edge detection based on the target image to obtain contour features; Based on the contour features and the label features, the target image is enhanced to obtain a target enhanced image; The target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

2. The image enhancement method according to claim 1, characterized in that: The label features are determined based on the following steps: Encoding the category of the detection frame of each target in the target detection label to obtain the category prompt word feature of each target; Embed the coordinates of the detection frame of each target in the target detection label to obtain the detection frame position features of each target; The category prompt word features and the detection frame position features of each target are fused to obtain the label features of the target image.

3. The image enhancement method according to claim 2, characterized in that: The embedding of the coordinates of the detection frame of each target in the target detection tag to obtain the position features of the detection frame of each target includes: Extracting the diagonal coordinates of the detection box of each target in the target detection label; Performing Fourier embedding on the diagonal coordinates to obtain detection frame position features of each target; The diagonal coordinates are the coordinates of two vertices on the diagonal line of the corresponding detection frame.

4. The image enhancement method according to any one of claims 1 to 3, characterized in that: The step of performing image enhancement on the target image based on the contour feature and the label feature to obtain a target enhanced image includes: Based on the contour feature, the label feature, and a plurality of noise images, the target image is enhanced to obtain a plurality of initial enhanced images; each initial enhanced image carries the target detection label; each noise image is obtained by random sampling; Determine, based on the detection frame of each target in any initial enhanced image, an enhanced region corresponding to each target; Determining a generation score of any of the initial enhanced images based on the enhanced regions corresponding to the respective targets in the any of the initial enhanced images and the label features of the target images; Based on the generation scores of the initial enhanced images, the target enhanced image is obtained by screening the initial enhanced images.

5. The image enhancement method according to any one of claims 1 to 3, characterized in that: The step of performing image enhancement on the target image based on the contour feature and the label feature to obtain a target enhanced image includes: Inputting the contour feature and the label feature into an image enhancement model to obtain a target enhanced image output by the image enhancement model; The image enhancement model is obtained by training on the basis of the diffusion model using sample images and sample target detection labels of the sample images.

6. The image enhancement method according to claim 5, characterized in that: The image enhancement model is trained based on the following steps: Based on the diffusion model, construct a first enhancement model and a second enhancement model, and based on the first enhancement model and the second enhancement model, construct an initial enhancement model; Based on the sample image, the sample target detection label of the sample image, and the sample prompt text of each target in the sample image, the initial enhancement model is trained to obtain the image enhancement model; Among them, the first enhancement model and the second enhancement model in the initial enhancement model are connected through a convolutional layer; the first enhancement model is constructed by adding layers to the attention mechanism in the diffusion model, and in the first enhancement model, except for the parameters of the gated self-attention layer added by the layer addition, other parameters are not updated during the training process.

7. The image enhancement method according to claim 6, characterized in that: The layer addition is to add a gated self-attention layer between each self-attention layer and each cross-attention layer in the attention mechanism in the diffusion model, and the input of the gated self-attention layer includes the sample label feature corresponding to the sample target detection label and the sample contour feature corresponding to the sample image; The second enhancement model is constructed by removing the decoding layer based on the diffusion model.

8. An image enhancement device, characterized in that: include: A determination unit, used to determine a target image, wherein the target image has a target detection label; A detection unit, used for performing edge detection based on the target image to obtain contour features; An enhancement unit, configured to perform image enhancement on the target image based on the contour feature and the label feature to obtain a target enhanced image; The target enhanced image and the target image have the same target detection label, the target detection label includes the coordinates and category of the detection box of each target in the corresponding image, and the label feature is constructed based on the target detection label.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image enhancement method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image enhancement method according to any one of claims 1 to 7 is implemented.

Images