Selective pass data enhancement method and device based on image saliency preservation

Through the significance object detection model and selective paste technology, the noise problem in data enhancement is solved, the model performance and retention of significance information is improved, and more efficient data enhancement effect is achieved.

CN118968209APending Publication Date: 2024-11-15ROCKET FORCE UNIV OF ENG
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410791660.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When existing data augmentation methods increase the diversity of training samples, they are prone to introduce noise, resulting in deterioration of model performance and difficult to effectively reduce the impact of noise.

Method used

Using a selective paste data enhancement method based on image significance maintenance, the significance prediction map of the image is detected through the significance target detection model, the significance score of the region is calculated, and the replication is carried out based on the region whose score is greater than the threshold to ensure the information of important regions. Retention and noise reduction.

Benefits of technology

It effectively reduces the introduction of data noise, improves the performance and generalization capabilities of machine learning models, and ensures the retention of significant information during image data enhancement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968209A_ABST
    Figure CN118968209A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a selective pass data enhancement method and device based on image saliency preservation, and is applied to the technical field of machine learning. The method comprises the following steps: acquiring an original image; noise adding and fuzzy processing are carried out to obtain a transformed image; inputting the original image into a pre-trained saliency target detection model, and outputting a saliency prediction map; performing region division on the saliency prediction map by taking a pixel point on the saliency prediction map as a preset-size cutting frame of a center to obtain a plurality of to-be-copied regions; calculating significance scores of the to-be-copied areas respectively, and determining the areas with the significance scores greater than a preset threshold value as target copy areas; copying an image in the same area as the target copying area in the original image to the transformed image; and taking the image obtained by restoring the fuzzy region of the copied image as enhanced data. Therefore, the saliency of the image can be used as prior information to guide the pass operation, and the introduction of noise data is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a selective paste data enhancement method and device based on image saliency preservation. Background Art

[0002] With the continuous development of artificial intelligence technology, image processing technology based on deep learning has become more mature and widely used. Models trained by data set-driven artificial intelligence technology often require huge data sets, but due to the limitations of time and money costs, they may often encounter situations where there are few images, poor image quality, and imbalanced categories, which brings various difficulties to image tasks. The quality and quantity of images in the data set greatly affect the generalization ability of deep learning models. Since deep learning network models have extremely strong learning capabilities, the models may often extract general features of data on image data sets as features for predicting a certain type of result, which often leads to good predictions on the training set, but high error rates on the test set and validation set, and low generalization ability of the model. An obvious trend has emerged in the era of data-driven artificial intelligence - fields with rich and cheap data are more likely to breed AI technology. However, most of the existing data sets in many fields currently have prominent problems such as small scale, uneven distribution, and difficulty in collection and labeling. It can be said that the lack or high cost of data is often the direct cause of the development of artificial intelligence.

[0003] In order to solve the problem of data scarcity, data augmentation technology came into being. Image-based data augmentation can increase the diversity of training samples, such as expanding the data set and enhancing data quality through basic image processing operations such as flipping and adding noise, or generating new samples based on existing data. The model is trained with the data set after the data augmentation method to achieve the effect of improving the robustness and generalization ability of the model. The essence of the data augmentation method is actually to make the data generate the value equivalent to a larger amount of data based on the existing limited data without actually collecting more data, that is, the process of generating incremental data according to the rules based on the existing data samples. The data augmentation method is not only an increase in the amount of data samples, but more of an "enhancement" of the characteristics of the data itself. Sample data is a sampling of the overall data. When the sample data volume is large enough, the distribution of the sample should be similar to the distribution of the population. However, due to objective reasons, the collected sample data is not complete. In this case, data enhancement methods can be used to generate new sample data that is more similar to the real data distribution. The deep learning network model has a very strong learning ability, so some useless information features learned will have a negative impact on the final result. The data enhancement technology can implement the pre-process of adding prior knowledge by imposing constraints on the data as needed, such as deleting or completing some information, to reduce the negative impact on the model performance of image processing tasks.

[0004] Data augmentation increases the effective data size and promotes diversity in training examples, but it inevitably introduces noise and ambiguity into the training process. Therefore, if the augmentation process is not properly tuned, the overall performance will deteriorate. Therefore, a method that can reduce data noise is urgently needed. Summary of the invention

[0005] The present disclosure provides a selective paste data enhancement method and device based on image saliency preservation.

[0006] According to a first aspect of the present disclosure, a selective paste data enhancement method based on image saliency preservation is provided. The method comprises:

[0007] Acquire an original image; add noise and blur processing to the original image to obtain a transformed image;

[0008] Inputting the original image into a pre-trained salient object detection model, and outputting a salient prediction map;

[0009] Using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center, dividing the saliency prediction map into regions to obtain a plurality of regions to be copied;

[0010] Calculating the significance scores of the to-be-copied areas respectively, and determining the to-be-copied areas whose significance scores are greater than a preset threshold as target copy areas;

[0011] Copying the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image;

[0012] The image obtained by restoring the blurred area in the copied image is used as the enhanced data.

[0013] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the salient object detection model is trained by the following steps:

[0014] Acquire a sample image set for training; the sample image set includes sample images and annotation information of target objects in the sample images;

[0015] Training a salient object detection model according to the sample image and the annotation information to obtain model parameters that enable a target loss function of the salient object detection model to meet a convergence condition;

[0016] The salient object detection model includes a feature extraction network, a first saliency extraction network, a second saliency extraction network and a detection network.

[0017] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the value of each pixel in the saliency prediction map is between 0 and 1;

[0018] The calculating the significance score of the area to be copied comprises:

[0019] Calculating the average significance of the region to be replicated;

[0020] The average significance is normalized to obtain a significance score of the region to be copied.

[0021] According to the above aspects and any possible implementation, an implementation is further provided, wherein the average significance is calculated by the following formula:

[0022]

[0023] In the formula, sod i Represents the sum of the saliency values ​​of the pixels in region i; num i Represents the number of pixels in area i; avg_sod i Represents the average significance in region i, where region i is the region to be replicated.

[0024] According to the above aspects and any possible implementation, an implementation is further provided, wherein the significance score is calculated by the following formula:

[0025]

[0026] Where, avg_sod i Represents the average significance of region i; avg_sod min Represents the minimum average significance of all regions in the significance prediction map; avg_sod max represents the maximum average significance in the region of the significance prediction map; score(i) represents the significance score of region i.

[0027] According to the above aspects and any possible implementation, an implementation is further provided, wherein the specific operation formula for copying the image in the same area as the target copy area in the original image to the same area as the target copy area in the transformed image is as follows:

[0028]

[0029] Where M(S) represents the binary mask of region S, the position of region S is 1, and the rest of the positions are 0; x represents the input image; x′ represents the transformed image; ⊙ represents the dot product; Represents the image after paste; where area S is the target copy area.

[0030] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the original image may be an image obtained by fusion of at least two images.

[0031] According to a second aspect of the present disclosure, a selective paste data enhancement device based on image saliency preservation is provided. The device comprises:

[0032] The image acquisition module is used to acquire the original image; add noise and blur to the original image to obtain a transformed image;

[0033] A saliency prediction module, used for inputting the original image into a pre-trained salient object detection model and outputting a saliency prediction map;

[0034] A copy region determination module, configured to divide the saliency prediction map into regions using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center, to obtain a plurality of regions to be copied;

[0035] The replication region determination module is further configured to respectively calculate the significance scores of the to-be-replicated regions, and determine the to-be-replicated regions whose significance scores are greater than a preset threshold as target replication regions;

[0036] A data enhancement module, used for copying the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image;

[0037] The data enhancement module is also used to restore the blurred area in the copied image to obtain an image as enhanced data.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0039] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0040] The embodiments of the present disclosure provide a selective paste data enhancement method and device based on image saliency preservation. After performing salient target detection processing on the original image, the saliency scores of one or more areas therein are calculated respectively, and the areas with the same positions as those with saliency scores greater than a preset threshold are copied to the image after the original image is transformed. In this way, the saliency information of the image is used as prior information to guide the paste operation, reduce the noise of machine learning samples, and improve the efficiency of machine learning.

[0041] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0043] Figure 1 A flowchart of a selective paste data enhancement method based on image saliency preservation according to an embodiment of the present disclosure is shown;

[0044] Figure 2 A block diagram of a selective paste data enhancement device based on image saliency preservation according to an embodiment of the present disclosure is shown;

[0045] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0047] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0048] This paper proposes a selective paste data enhancement algorithm based on image saliency preservation, which uses the image saliency information as prior information to extract key target information in the image, and pastes the extracted key information back to the corresponding area after image-level changes to the input image to ensure that the area with high importance score is always presented after data enhancement. The whole method is divided into 3 steps: 1. Use the saliency target detection model to detect the saliency prediction map of the input image; 2. Use the saliency prediction information to determine the score of each area; 3. Selective paste data enhancement based on saliency preservation. By introducing the saliency information of the image as a priori, the introduction of noise data is reduced, and the performance of the network model is further improved.

[0049] Among them, data enhancement is a method of expanding the data set using technical means. By transforming or modifying the original data, more new data samples are generated to improve the generalization ability and stability of the model. Data enhancement can effectively expand the size of the data set and improve the performance of the model without increasing the cost of additional annotation by using operations such as rotation, flipping, and cropping. In addition, data enhancement can also simulate the uncertainty and variability in real scenes by performing operations such as color transformation and adding noise to the image, making the model more adaptable to various actual situations. Therefore, data enhancement is one of the very important technologies in the field of machine learning and deep learning, and is widely used in image classification, object detection, speech recognition and other fields. In general, data enhancement operations include the following aspects:

[0050] (1) Image flipping: Flipping an image along the horizontal or vertical axis can generate a new image that is mirror-symmetrical to the original image. This helps the model learn features that are not affected by direction.

[0051] (2) Random Cropping: Randomly selecting an area on the image and cropping it enables the model to learn to focus on important parts of the image while also increasing the diversity of the image.

[0052] (3) Rotation transformation: Rotating the image around its center point by a certain angle can simulate the image conditions under different shooting angles and enhance the model's adaptability to different viewing angles.

[0053] (4) Changing image brightness and contrast: By adjusting the brightness and contrast of the image, different lighting conditions can be simulated to improve the robustness of the model to lighting changes.

[0054] (5) Adding noise: Adding random noise to the image, such as Gaussian noise and salt and pepper noise, helps the model learn more robust feature representations.

[0055] Figure 1A flowchart of a selective paste data enhancement method 100 based on image saliency preservation according to an embodiment of the present disclosure is shown. The method 100 includes:

[0056] Step 110, obtaining an original image; adding noise and blurring to the original image to obtain a transformed image.

[0057] In some embodiments, in the process of machine learning, when data enhancement is used to increase the diversity of training samples, more noise will be introduced into the expanded samples, reducing the performance of the model. Therefore, the expanded samples need to be further processed to reduce noise and improve model performance.

[0058] In some embodiments, the original image in the training sample is obtained, and / or the expanded sample image after expansion based on the original image. In order to further improve the processing efficiency of the present disclosure, the original image may be an image after fusion of at least two images. Since the significant difference of the fused image is more obvious, it will be more efficient to use the method described in the present disclosure in the data enhancement process. In which, the fused image can be obtained by further fusion processing of the image obtained when the data is enhanced by CutMix technology. CutMix is ​​an image enhancement technology that increases the diversity of the data set by exchanging parts of the two images and adjusting the labels accordingly. Specifically, CutMix randomly selects a picture, cuts off a part of the area, and then pastes the part to the same position of another picture. The adjustment of the label is allocated according to the proportion of the area of ​​the two pictures in the original image. This method can enable the model to be better generalized to various situations and improve the performance of the model.

[0059] At the same time, the original image can be the image obtained by further fusion processing when the image is data enhanced by the CutMix technique, mainly because the early work in the field of data enhancement focused more on data-independent techniques such as Dropout, SpatialDropout, and DropBlock. Dropout performs well on fully connected layers. However, it works poorly on convolutional layers. One of the reasons why dropout fails on CNN layers may be that the activation units in the convolutional layers are correlated, so although some activation units are removed, information can still flow through these layers. SpatialDropout solves this problem by removing the entire feature map from the convolutional layer. DropBlock further improves SpatialDropout by discarding random continuous feature blocks from the feature map instead of discarding the entire feature map in the convolutional layer. Recent work has shown that data-dependent regularizers can achieve better generalization for CNN models. Mixup is such a data-dependent regularizer that synthesizes additional training samples by fusing random input image pairs and their corresponding labels in linear proportions. Cutout removes continuous regions from the image in the input space. This noise encourages the network to learn the full context of the image instead of overfitting a small number of visual features. Unlike Cutout, CutMix randomly replaces an image region with a patch from another training image instead of completely inserting two selected images and mixing their labels according to the ratio of mixed pixels. This method can enjoy the Cutout feature without producing any blank image areas. However, the random selection process may crop out the area where the key object in the target image is located, and the enhanced image may not contain any information about the target object. For example, an image of a cat and an image of a dog form a mixed image, and the label of the mixed image is a mixture of the labels of cats and dogs at a pixel ratio. The mixed image does not contain the target information of the cat, but the mixed label of the mixed image encourages the model to learn the features of the two targets, forcing the model to treat the features learned from the area unrelated to the target as the representative features of the object during training, which obviously reduces the performance of the network. Therefore, the introduction of the saliency preservation method can ensure that the area where the key object in the target image is located will not be cropped out, and the noise can be well removed.

[0060] In some embodiments, in order to ensure that the saliency prediction map detected based on the original image and the saliency area re-determined according to the preset threshold and the saliency score greatly ensure that the saliency is not lost (for example, not cropped) during the data enhancement process, the original image is preferentially transformed, such as adding noise and blurring the noise-added area, etc.

[0061] Step 120: input the original image into a pre-trained salient object detection model, and output a salient prediction map.

[0062] In some embodiments, the salient target detection model is trained by the following steps: obtaining a sample image set for training; the sample image set includes sample images and annotation information of target objects in the sample images; training the salient target detection model according to the sample images and the annotation information to obtain model parameters that make the target loss function of the salient target detection model meet the convergence conditions; wherein the salient target detection model includes a feature extraction network, a first salient extraction network, a second salient extraction network and a detection network. The feature extraction network and multiple salient extraction networks can be convolutional neural networks, which can include but are not limited to VGG networks, Residual convolutional networks, Dense convolutional networks, SE convolutional networks, etc. The detection network can be a target detection network, which can include but are not limited to yolo networks, Faster RCNN networks, Mask RCNN networks, etc.

[0063] In practical applications, before machine learning training is performed, training data, i.e., a sample image set, may be determined first. In this embodiment, the training data may include sample images and annotation information of target objects in the sample images, wherein the annotation information of the target objects may be determined based on specific information of the detection results of the target objects that need to be obtained in practical applications. For example, if the position information of the target object in the target image needs to be obtained, the annotation information may include the position information of the target object in the corresponding sample image. When the annotation information of the target object includes the position information of the target object in the corresponding sample image, before obtaining the sample image set for training, at least one annotation area corresponding to the target object in the sample image needs to be determined. Specifically, the at least one annotation area may be an area drawn or framed by different annotation personnel based on experience for the target object in the sample image. The salient target detection model is trained based on the sample images in the sample image set and the annotation information of the target objects in the sample images, and the model parameters that make the target loss function of the salient target detection model meet the convergence conditions are obtained. The model parameters at this time are the parameters of the salient target detection model in this embodiment.

[0064] In some embodiments, the value of each pixel in the saliency prediction map is between 0 and 1, indicating the degree of saliency of the pixel. The saliency prediction map includes a saliency image region and a non-saliency image region. The saliency image region is the region where the target detection image is located, and the non-saliency image region is the region where the non-target image is located. For example, if the target object is a dog, the saliency image region is the region detected as a dog, displayed as 1 (white), and the non-saliency image region is the region detected as a non-dog, displayed as 0 (black).

[0065] Step 130 , dividing the saliency prediction map into regions using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center to obtain a plurality of regions to be copied.

[0066] In some embodiments, the preset size is less than or equal to the size of the original image; the area to be copied is any area divided in the saliency prediction map according to the preset size. For example, any pixel point a is determined as the center from the saliency prediction map, the preset size is k×k, a cropping frame is determined, the saliency prediction map is divided into regions according to the cropping frame, and the divided region is used as the area to be copied. It should be noted that the determined pixel point a as the center can be determined from the salient image area or from the non-salient image area. It should be noted that the preset size is not limited to the above examples, and can be k×k, k×g, etc.

[0067] Step 140 , respectively calculating the significance scores of the to-be-copied regions, and determining the to-be-copied regions whose significance scores are greater than a preset threshold as target copy regions.

[0068] In some embodiments, the calculating of the significance score of the area to be copied includes: calculating the average significance of the area to be copied; and normalizing the average significance to obtain the significance score of the area to be copied. For example, after calculating the average significance of the area (the area to be copied) corresponding to each pixel, assigning values ​​to each pixel in ascending order from 0, and then normalizing to obtain the significance score of the area to be copied.

[0069] The calculation of the average significance is performed by the following formula:

[0070]

[0071] In the formula, sod i Represents the sum of the saliency values ​​of the pixels in region i; num i Represents the number of pixels in area i; avg_sod iRepresents the average significance in region i, where region i is the region to be replicated.

[0072] The calculation of the significance score is performed by the following formula:

[0073]

[0074] Where, avg_sod i Represents the average significance of region i; avg_sod min Represents the minimum average significance of all regions in the significance prediction map; avg_sod max represents the maximum average significance in the region of the significance prediction map; score(i) represents the significance score of region i.

[0075] In some embodiments, the calculated significance score is used to further determine the area (target copy area) in each area (area to be copied) in the significance prediction map whose significance score is greater than a preset threshold, that is, the area whose significance score meets the requirements is determined as the key object to further ensure that the significant area will not be lost during the data enhancement process. In other words, during the image processing or data enhancement process, the main significant features of the image are ensured to be retained. Among them, the preset threshold can be set by the operator based on historical experience, or it can be flexibly adjusted by the operator based on the calculation result of the current significance score.

[0076] Step 150, copying the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image; and performing restoration processing on the blurred area in the copied image to obtain an image as enhanced data.

[0077] In some embodiments, the specific operation formula for copying the image in the same area as the target copy area in the original image to the same area as the target copy area in the transformed image is as follows:

[0078]

[0079] Where M(S) represents the binary mask of region S, the position of region S is 1, and the rest of the positions are 0; x represents the input image; x represents the transformed image; ⊙ represents the dot product; Represents the image after paste; where area S is the target copy area.

[0080] For example, through the calculation result of step 140, it is known that the significance score of region u in the area to be copied is greater than the preset threshold, then the image of the same area as region u is copied from the original image to the same area of ​​the transformed image, and then the blurred area in the transformed image is restored, so that the significant features of the image are retained, the noise is reduced, and the quality of the machine learning training samples is improved.

[0081] Overall, this solution can be implemented through the following algorithm process:

[0082]

[0083] Furthermore, the saliency-preserving selective paste method can also be applied to a special region-level enhancement method, CutMix, which selects regions with higher saliency scores in the original image to replace the corresponding regions in the target image. This can preserve the saliency information in the original image and alleviate the problem of key information loss caused by the CutMix method to a certain extent.

[0084] Therefore, by maintaining saliency, the noise problem caused by data enhancement can be solved more effectively, and superior performance is shown when processing image data.

[0085] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0086] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0087] Figure 2 FIG. 2 shows a block diagram of a selective paste data enhancement device 200 based on image saliency preservation according to an embodiment of the present disclosure. Figure 2 As shown, the device 200 includes:

[0088] The image acquisition module 210 is used to acquire an original image; add noise and blur to the original image to obtain a transformed image;

[0089] A saliency prediction module 220, configured to input the original image into a pre-trained salient object detection model and output a saliency prediction map;

[0090] A copy region determination module 230, configured to divide the saliency prediction map into regions using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center, to obtain a plurality of regions to be copied;

[0091] The replication region determination module 230 is further configured to respectively calculate the significance scores of the to-be-replicated regions, and determine the to-be-replicated regions with the significance scores greater than a preset threshold as target replication regions;

[0092] A data enhancement module 240 is used to copy the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image;

[0093] The data enhancement module 240 is further configured to use the image obtained by restoring the blurred area in the copied image as the enhanced data.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0095] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0096] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0097] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0098] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a selective paste data enhancement method based on image saliency preservation. For example, in some embodiments, the selective paste data enhancement method based on image saliency preservation may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the selective paste data enhancement method based on image saliency preservation described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform a selective paste data enhancement method based on image saliency preservation in any other appropriate manner (eg, by means of firmware).

[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A selective paste data enhancement method based on image saliency preservation, characterized in that: include: Get the original image; Adding noise and blurring to the original image to obtain a transformed image; Inputting the original image into a pre-trained salient object detection model, and outputting a salient prediction map; Using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center, dividing the saliency prediction map into regions to obtain a plurality of regions to be copied; Calculating the significance scores of the to-be-copied areas respectively, and determining the to-be-copied areas whose significance scores are greater than a preset threshold as target copy areas; Copying the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image; The image obtained by restoring the blurred area in the copied image is used as the enhanced data.

2. The method according to claim 1, characterized in that The salient object detection model is trained by the following steps: Acquire a sample image set for training; the sample image set includes sample images and annotation information of target objects in the sample images; Training a salient object detection model according to the sample image and the annotation information to obtain model parameters that enable a target loss function of the salient object detection model to meet a convergence condition; The salient object detection model includes a feature extraction network, a first saliency extraction network, a second saliency extraction network and a detection network.

3. The method according to claim 1, characterized in that The value of each pixel in the saliency prediction map is between 0 and 1; The calculating the significance score of the area to be copied comprises: Calculating the average significance of the region to be replicated; The average significance is normalized to obtain a significance score of the region to be copied.

4. The method according to claim 3, characterized in that The calculation of the average significance is performed by the following formula: In the formula, sod i Represents the sum of the saliency values ​​of the pixels in region i; num i Represents the number of pixels in area i; avg_sod i Represents the average significance in region i, where region i is the region to be replicated.

5. The method according to claim 4, characterized in that The calculation of the significance score is performed by the following formula: Where, avg_sod i Represents the average significance of region i; avg_sod min Represents the minimum average significance of all regions in the significance prediction map; avg_sod max represents the maximum average significance in the region of the significance prediction map; score(i) represents the significance score of region i.

6. The method according to claim 1, characterized in that The specific operation formula for copying the image in the same area as the target copy area in the original image to the same area as the target copy area in the transformed image is as follows: Where M(S) represents the binary mask of region S, the position of region S is 1, and the rest of the positions are 0; x represents the input image; x represents the transformed image; ⊙ represents the dot product; Represents the image after paste; where area S is the target copy area.

7. The method according to claim 1, characterized in that The original image may be an image obtained by fusion of at least two images.

8. A selective paste data enhancement device based on image saliency preservation, characterized in that: include: An image acquisition module, used for acquiring original images; Adding noise and blurring to the original image to obtain a transformed image; A saliency prediction module, used for inputting the original image into a pre-trained salient object detection model and outputting a saliency prediction map; A copy region determination module, configured to divide the saliency prediction map into regions using a cropping frame of a preset size with a pixel point on the saliency prediction map as the center, to obtain a plurality of regions to be copied; The replication region determination module is further configured to respectively calculate the significance scores of the to-be-replicated regions, and determine the to-be-replicated regions whose significance scores are greater than a preset threshold as target replication regions; A data enhancement module, used for copying the image of the same area as the target copy area in the original image to the same area as the target copy area in the transformed image; The data enhancement module is also used to restore the blurred area in the copied image to obtain an image as enhanced data.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Citation Information

Cited By

  • InSAR co-seismic deformation down-sampling method and system based on image saliency

    CN120314949A