Environment semantic enhanced remote sensing archaeological target detection data augmentation method and device

Abstract

The invention discloses an environment semantic enhanced remote sensing archaeological target detection data augmentation method and device. The method comprises the following steps: constructing a remote sensing archaeological original data set; constructing an environment instruction set based on the original data set, and generating a composite image with environment semantics by using a graph-to-graph diffusion model and the environment instruction set; performing random enhancement operation on the original image of the original data set to obtain an enhanced image, and constructing an augmented data set based on the composite image, the enhanced image, the fractal image and the original image; and performing performance evaluation on the augmented data set. According to the method, the quality of the remote sensing archaeological image data set is improved, the model overfitting condition is relieved, and the influence of abnormal data on the model performance is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing archaeology, and in particular relates to a method and device for augmenting remote sensing archaeology target detection data with enhanced environmental semantics. Background Art

[0002] Ancient tombs are one of the main research objects in the field of archaeology. Due to the different eras and cultural natures of the tombs, they present various forms on the surface, such as stone-walled tombs, stone pile tombs, and stone coffin tombs. Their common feature is that they often use the pile as a ground mark, and show soil, shadows, and crop marks on remote sensing images. Compared with traditional dragnet field archaeological surveys, remote sensing technology can significantly improve regional coverage and work efficiency. With the continuous increase in the amount and type of remote sensing data, the use of manual interpretation or semi-automatic remote sensing target detection methods has slow calculation speed, long time consumption, and cumbersome implementation steps. It is difficult to cope with a large amount of complex data. Archaeological target detection based on deep learning is a flexible and robust archaeological target automatic detection method that realizes the automatic detection of large-scale tomb relics. The training process of deep learning is essentially to fit the real data distribution through the existing data distribution. Therefore, whether the amount of data is sufficient determines the upper limit of the performance of the deep learning algorithm.

[0003] After thousands of years of environmental changes and human destruction, many ancient tombs have been buried underground and have not been discovered, or only some of the relics remain on the surface. It is difficult to distinguish between relics and non-relic targets. It is difficult to effectively extract these weak archaeological information on remote sensing images, which makes it difficult to break through the quantitative limitations of ancient tomb sample data. In most cases, after remote sensing extraction, ancient tomb targets still need to be confirmed by field investigation, especially in special areas such as alpine areas, desert areas, vegetation-covered areas and large-scale survey areas. Field investigation requires a lot of manpower and material resources. The cost of building ancient tomb datasets is high, which further limits the number of samples. When the number of samples is limited, it is usually difficult for the model to learn the real data distribution, resulting in poor model performance and limiting the automatic detection accuracy of archaeological targets. The small sample problem will have a certain impact on the recognition accuracy and generalization ability of the model.

[0004] For small sample data, enhancing or expanding it may be an effective solution. Its essence is to expand the distribution of original data with the help of auxiliary data or transformation rules, so as to obtain data with more complex structure or more quantity. Compared with single image processing operations, image hybrid enhancement is an important means to improve the robustness of the model. Data augmentation technology that introduces environmental semantic information is of great significance to improving the accuracy and reliability of target detection.

[0005] The complex environment will have a significant impact on the identification of ancient tomb targets. The difference in the number of samples between different environments will lead to bias in the training results of the model. It has good detection capabilities for background environments with relatively more sample data, but poor performance in environments with fewer samples. Most existing image enhancement technologies cannot simulate sample images in different environments, resulting in an uneven distribution of sample data and affecting the accuracy of the model. Conventional diffusion model image enhancement technology is suitable for the classification of medium or large targets. Since it will destroy the structure of small target samples, it may cause the model to be unable to accurately learn sample features, resulting in poor augmentation effects for small target data. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method and device for augmenting remote sensing archaeological target detection data with environmental semantics enhancement. On the one hand, by transforming and expanding the original data, the diversity and richness of the data are increased, and the generalization ability and robustness of the archaeological target detection model are improved; on the other hand, by aggregating the images generated by the diffusion model driven by environmental semantics prompt words, the relationship between the archaeological sites or relics and the surrounding environment is better understood, and the accuracy of archaeological target detection is improved. By increasing the robustness of the data set and the complexity of the environment, the recognition ability of the deep learning model in different environments can be effectively enhanced, the generalization ability of the model can be improved, and high-precision automatic detection of ancient tombs in complex environments can be achieved.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics, comprising the following steps:

[0009] Constructing original remote sensing archaeological data sets;

[0010] Based on the original data set, an environmental instruction set is constructed, and a synthetic image with environmental semantics is generated using the graph diffusion model and the environmental instruction set.

[0011] Perform random enhancement operations on the original images of the original data set to obtain enhanced images, and construct an augmented data set based on the synthetic images, enhanced images, fractal images and original images;

[0012] Performance evaluation is performed on the augmented dataset.

[0013] On the other hand, the present invention also provides a remote sensing archaeological target detection data augmentation device with enhanced environmental semantics, comprising:

[0014] The original data set construction unit is used to construct the original data set of remote sensing archaeology;

[0015] A synthetic image construction unit is used to construct an environment instruction set based on the original data set, and generate a synthetic image with environmental semantics using the graph diffusion model and the environment instruction set;

[0016] An augmented data set construction unit, used for performing a random augmentation operation on an original image of an original data set to obtain an enhanced image, and constructing an augmented data set based on the synthetic image, the enhanced image, the fractal image and the original image;

[0017] The performance evaluation unit performs performance evaluation on the augmented dataset.

[0018] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics.

[0020] The beneficial effects of the present invention are:

[0021] The present invention transforms the environmental semantics of small-target remote sensing archaeological data without destroying the target structural features; simulates target images in multiple environments to avoid sample imbalance problems; and augments a small amount of sample data to improve the performance and generalization ability of the deep learning model. The method improves the stability of the remote sensing archaeological image data set, alleviates the overfitting of the model, reduces the impact of abnormal data on the model performance, improves the robustness of the model, and improves the performance of the model in remote sensing archaeological target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flow chart of a method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics according to the present invention;

[0023] Figure 2 This is the principle diagram of image enhancement and fusion algorithm. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, the present invention provides a method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement, the method comprising the following steps:

[0026] Construction of the original data set of remote sensing archaeology: First, remote sensing satellite data is acquired. According to archaeological materials or survey data, the climate, geography, and cultural environment characteristics of the archaeological survey area are analyzed, and remote sensing data including effective sensor types and imaging time are obtained. The soil signs, crop signs, and shadow signs of the archaeological targets are visually interpreted, and the images interpreted with archaeological targets are cropped so that each image has a clear single or multiple archaeological targets, and they are cropped into images with the same length and width, generally set to 640 pixels. Then, the sample annotation tool is used to construct labels for the archaeological targets, requiring the label box to be as close as possible to the minimum circumscribed rectangle of the archaeological target. After that, the label box and the corresponding image are stored in the folder with the same name or index to complete the data set construction.

[0027] Remote sensing image generation based on diffusion model and environmental instructions: Visually interpret and summarize the environmental features involved in the original data set, such as desert, Gobi, ice and snow, water (rivers, lakes), forests, grasslands, farmland, residential areas, etc., and preset a set of environmental instructions that at least covers all environmental features of the data set, such as P={"Desert","Gobi", "ice and snow","water", "forest", "grassland", "farmland", "residential area"}. Select a pre-trained image-generated diffusion model, input the original image in the data set and the preset environmental instructions, and you can generate a synthetic image with environmental semantics.

[0028] Among them, the graph-generated graph diffusion model is mainly composed of two modules: the forward diffusion process and the reverse diffusion process. In the forward diffusion process, the model first gradually adds noise from the real data sample until the sample becomes pure noise close to uniform distribution. In the reverse diffusion process, the model estimates the conditional probability distribution of reverse diffusion by training a U-Net neural network, gradually removes noise, and restores it to the original format of the data sample. Through the trained graph-generated graph diffusion model, a pure noise sample can be generated into a new high-quality data sample through a step-by-step reverse diffusion process. The graph-generated graph diffusion model has achieved remarkable results in the field of image generation, and is also widely used in the fields of text generation, audio synthesis, and video generation. However, if the diffusion model is used directly for image enhancement, it is often difficult to obtain better results because the structure of the small target sample is destroyed. At present, there are few studies and applications of using diffusion models to enhance remote sensing archaeological data. The present invention utilizes the excellent performance of the diffusion model in the field of image generation, introduces environmental parameter prompt words, and performs environmental semantic enhancement on remote sensing archaeological image data without destroying the original sample structure and label position distribution, thereby improving the accuracy of the model.

[0029] Comprehensive image enhancement processing: Enhance the image based on the following rules: Definition is the format of the input image, h represents the height of the image, w represents the width of the image, and c represents the number of channels of the image; Represents the entire enhancement algorithm, O utput represents the final output aggregate image, G represents the synthetic image generation operation, F represents the fractal graphics fusion operation, A represents the random image enhancement operation, represents the weight coefficient in the image aggregation process, which can be written as: utput .

[0030] The enhanced image obtained by random image enhancement operation is defined as A= , where I represents the original input image and o represents a set of predefined image enhancement operations. The result of random sampling is set as:

[0031] O={posterize,translate_y,translate_x,equalize,shear_x,shear_y,solarize,contrast, brightness, sharpness}, k represents the number of executions, preset as interval The image after image enhancement can be expressed as .

[0032] The operation of generating a synthetic image through the diffusion model of the input image is defined as , where I represents the original input image and p is an instruction randomly selected from a predefined instruction set P. The instruction set P is defined as:

[0033] P={"River","Autumn","BuiltArea","Clouds","Crops","Desert","FloodedVegetation","Forest","Grassland","Rangeland","Snowy / Ice","Water"}. After processing, the synthetic image is obtained , the whole algorithm can be expressed as:

[0034] O utput ,

[0035] in, Representation vector , is a set of pre-set weights, which are set to , T represents the fusion vector , The result after the original image I undergoes random image enhancement operation A is the enhanced image . It represents the synthetic image generated by the original image I through the instruction, that is, With enhanced image The first fusion result of the image multiplication operation is defined as ,in Represents two images of the same size, , where α and β represent a set of corresponding random numbers obtained from Beta distribution, then , represents the second fusion result after fractal images randomly selected from the fractal graphics set are fused with the first fusion result, that is, , where f represents a fractal image. All fractal images are highly non-accidental graphics drawn by humans on the DeviantArt art website, that is, synthetic images, enhanced images, and fractal images are enhanced again using a superposition and fusion mechanism. Figure 2 As shown in Figure 2, a specific data enhancement process is given.

[0036] Combine the enhanced dataset generated by the above algorithm with the original dataset (i.e., the original image dataset) to form an augmented dataset, and organize the label information.

[0037] Among them, fractal graphics help to improve the security performance of image data. Combining fractal graphics and random image enhancement operations can improve the performance of the model in terms of accuracy and security. Through the image fusion structure designed by the present invention, combined with fractal graphics, random image enhancement results, and diffusion model generation results, the purpose of image augmentation is effectively achieved, so that the model can greatly improve the generalization ability of the model without introducing new real data, so that the model can still maintain good recognition performance in multiple environments and strong disturbances.

[0038] Performance evaluation: The performance of the augmented dataset is evaluated by using the target detection neural network model selected by the user. The class average precision, PR curve and other related indicators are calculated to compare the class average precision of heterogeneous datasets, such as the performance of the augmented dataset and the original dataset. According to the results, the generation parameters of the diffusion model, such as the number of iterations and noise intensity, are adjusted to improve the data quality and form a final dataset that can achieve better target recognition effect after augmentation.

[0039] In summary, the present invention generates synthetic images by introducing environmental prompt words based on the diffusion model, which can enhance the environmental semantics of the original remote sensing image without changing the characteristic structure and label position of the archaeological target, and improve the accuracy of the deep learning target detection model. On this basis, combined with random image enhancement, fusion of complex fractal graphics and other operations, more image features are generated while ensuring that the original label is not offset, and data augmentation of remote sensing archaeological images is achieved. The augmented data set obtained after processing has better security performance, covers a variety of environmental semantics, and greatly improves the generalization ability of the model.

[0040] On the other hand, the present invention also provides a remote sensing archaeological target detection data augmentation device with enhanced environmental semantics, which includes various units capable of executing various steps of the above method, specifically including:

[0041] The original data set construction unit is used to construct the original data set of remote sensing archaeology;

[0042] A synthetic image construction unit is used to construct an environment instruction set based on the original data set, and generate a synthetic image with environmental semantics using the graph diffusion model and the environment instruction set;

[0043] An augmented data set construction unit, used for performing a random augmentation operation on an original image of an original data set to obtain an enhanced image, and constructing an augmented data set based on the synthetic image, the enhanced image, the fractal image and the original image;

[0044] The performance evaluation unit performs performance evaluation on the augmented dataset.

[0045] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics.

[0047] Without loss of generality, the present invention also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0048] The present invention also provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor can implement the above-mentioned method.

[0049] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0050] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement, characterized in that: The steps include: Constructing original remote sensing archaeological data sets; Based on the original data set, an environmental instruction set is constructed, and a synthetic image with environmental semantics is generated using the graph diffusion model and the environmental instruction set. Perform random enhancement operations on the original images of the original data set to obtain enhanced images, and construct an augmented data set based on the synthetic images, enhanced images, fractal images and original images; Performance evaluation is performed on the augmented dataset.

2. The method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics according to claim 1, characterized in that: The construction of remote sensing archaeology original data set includes: Analyze the climate, geography, and cultural environment characteristics of the archaeological survey area based on archaeological materials or survey data, obtain remote sensing data including effective sensor types and imaging time, and visually interpret soil marks, crop marks, and shadow marks of archaeological targets; Crop the images interpreted with archaeological targets so that each image has a clear single or multiple archaeological targets; Use the sample annotation tool to construct labels for the cropped images with archaeological objects; Store the label boxes and the corresponding images in the folder with the same name or index to complete the construction of the original dataset.

3. The method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics according to claim 1, characterized in that: The method of generating a synthetic image with environmental semantics by using a graph-generated image diffusion model and an environmental instruction set includes: visually interpreting and summarizing environmental features involved in an original data set, constructing an environmental instruction set that at least covers all environmental features of the original data set, selecting a pre-trained graph-generated image diffusion model, inputting the original image in the original data set and preset environmental instructions, and generating a synthetic image with environmental semantics.

4. The method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement according to claim 3 is characterized in that: The diffusion model includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, noise is gradually added to the real data sample until the real data sample becomes pure noise with uniform distribution. In the reverse diffusion process, the U-Net neural network is used to estimate the conditional probability distribution of reverse diffusion, gradually remove the noise, and restore the original format of the real data sample.

5. The method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement according to claim 1, characterized in that: The enhanced image is a result randomly extracted from a predefined set of image enhancement operations; the fractal image is a highly non-accidental graphic drawn by humans from the DeviantArt art website.

6. The method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics according to claim 5, characterized in that: The constructing of the augmented data set based on the synthesized image, the enhanced image, the fractal image and the original image includes constructing a final output aggregate image based on the synthesized image, the enhanced image, the fractal image and the original image. utput : O utput ， in, Represents the weight coefficient in the image aggregation process, I represents the original image, and T represents the fusion vector , represents the enhanced image obtained after the original image I is subjected to random image enhancement operation, represents the first fusion result of the original image I generated by the instruction and the enhanced image, represents the second fusion result after the fractal image is fused with the first fusion result, represents the weight vector corresponding to the fusion vector T; Based on the multiple original images, multiple final output aggregated images are obtained to form an enhanced data set; The enhanced dataset and the original dataset constitute an augmented dataset.

7. The method for augmenting remote sensing archaeological target detection data with enhanced environmental semantics according to claim 1, characterized in that: The performance evaluation of the augmented data set includes: evaluating the performance of the augmented data set through the target detection neural network model selected by the user, calculating the class average accuracy and PR curve, comparing the class average accuracy of heterogeneous data sets, and adjusting the number of iterations and noise intensity of the graph-generated graph diffusion model according to the performance evaluation results.

8. A remote sensing archaeological target detection data augmentation device with environmental semantic enhancement, characterized in that: include: The original data set construction unit is used to construct the original data set of remote sensing archaeology; A synthetic image construction unit is used to construct an environment instruction set based on the original data set, and generate a synthetic image with environmental semantics using the graph diffusion model and the environment instruction set; An augmented data set construction unit, used for performing a random augmentation operation on an original image of an original data set to obtain an enhanced image, and constructing an augmented data set based on the synthetic image, the enhanced image, the fractal image and the original image; The performance evaluation unit performs performance evaluation on the augmented dataset.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Among them, when one or more programs are executed by the one or more processors, the one or more processors implement the method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement a method for augmenting remote sensing archaeological target detection data with environmental semantic enhancement as described in any one of claims 1-7.

Images