Image mosaic misalignment detection method based on satellite imagery and VGG model fine-tuning

Abstract

The embodiment of the present invention provides a method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning, the method comprising: preprocessing mosaic images of satellite images, performing misalignment judgment, and determining misalignment areas; marking the misalignment areas of the mosaic images, and dividing the marked mosaic images into a training set and a verification set; inputting the training set into a VGG model for training, calculating a loss function through a predicted image obtained through VGG model training and the training set, and training until the VGG model is stable; inputting the verification set into the trained VGG model, comparing the output image of the VGG model with the verification set, and evaluating the correctness of the model output result; when the correctness is less than a preset accuracy value, re-marking the erroneous image, iteratively training the VGG model through the re-marked erroneous image, and outputting the model weight of the VGG model; integrating the model weight into a preset detection script, and detecting the mosaic misalignment points of the original satellite image in real time.

Description

Technical Field

[0001] The present invention relates to the field of satellite image technology, and in particular to a method for detecting image mosaic misalignment points based on satellite imagery and VGG model fine-tuning. Background Art

[0002] Satellite image mosaicking is a technique that stitches together multiple image fragments taken from satellites into a large, continuous image. This process involves precisely aligning the individual image fragments to form a seamless or nearly seamless view of a large area of ​​the Earth's surface. The main purpose is to create a continuous map covering a wider area to provide large-scale information that cannot be covered by a single image.

[0003] Satellite image mosaicking is usually achieved through a series of processes including preprocessing, image alignment and registration, fusion and splicing, and post-processing. During the mosaicking process, due to sensor errors, inaccurate ground control points, projection and deformation, and differences in shooting angles, there will be misalignment of objects on both sides of the mosaicking line.

[0004] For the above problems, currently the dislocated parts are mainly detected step by step by manual inspection. However, manual inspection requires a lot of manpower and time, resulting in extremely low detection efficiency. Therefore, there is an urgent need for a detection method to replace manual inspection and realize rapid, large-scale and accurate detection of satellite image mosaics. Summary of the invention

[0005] In view of the problems existing in the prior art, an embodiment of the present invention provides a method and system for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning.

[0006] The embodiment of the present invention provides a method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning, the method comprising:

[0007] Acquire a mosaic image obtained by mosaicking the original satellite image, preprocess the mosaic image, and perform misalignment determination on the preprocessed mosaic image to determine a misalignment area in the mosaic image;

[0008] Detecting the mosaic line in the mosaic image, determining whether the mosaic line overlaps with the dislocation area, marking the mosaic image based on the determination result, and dividing the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlap area of ​​the mosaic line and the dislocation area as a dislocation label, and mark the non-overlap area as a non-dislocation label;

[0009] Input the training set into the VGG model for training, calculate the loss function through the predicted image obtained by the VGG model training and the training set, use the loss function as a training basis, and train until the VGG model is stable;

[0010] Input the validation set into the trained VGG model, compare the output image of the VGG model with the validation set, and evaluate the correctness of the model output result;

[0011] When the correctness is less than the preset accuracy value, an error image in the output image is obtained, the error image is re-labeled, the VGG model is iteratively trained through the re-labeled error image, and the correctness of the model output result is evaluated until the correctness of the model output result is greater than the preset accuracy value, and the model weight of the VGG model is output;

[0012] The model weights are integrated into a preset detection script, the detection script is encapsulated into a detection tool or a detection service, and the mosaic misalignment points of the original satellite image are detected in real time based on the detection tool or the detection service.

[0013] In one embodiment, the method further comprises:

[0014] Obtain the labeling result of the object frame in the mosaic line in the predicted image, compare it with the labeling result of the object frame in the mosaic image in the training set, and calculate the loss function based on the comparison result of the object frame.

[0015] In one embodiment, the method further comprises:

[0016] The number of misplaced labels and the number of non-misplaced labels in the mosaicked image are counted, and the image data in the mosaicked image are adjusted so that the number of misplaced labels and the number of non-misplaced labels are approximately equal. When dividing the training set and the validation set, the number of misplaced labels and the number of non-misplaced labels in the training set and the validation set are kept approximately equal.

[0017] In one embodiment, the misalignment determination is:

[0018] Differences in geometric accuracy, image feature point offset, incoherence in the seam area, spatial accuracy analysis, and image overlay errors.

[0019] In one embodiment, the method further comprises:

[0020] A detection process for mosaic dislocation points is obtained, and a detection script is written based on the detection process. The detection process includes: collecting original satellite images, image mosaicking, image preprocessing, input model training, and outputting detection results.

[0021] The embodiment of the present invention provides a system for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning, the system comprising:

[0022] An acquisition module is used to acquire a mosaicked image after mosaicking the original satellite image, preprocess the mosaicked image, and perform misalignment determination on the preprocessed mosaicked image to determine a misalignment area in the mosaicked image;

[0023] a marking module, for detecting mosaic lines in the mosaic image, determining whether the mosaic lines overlap with the misaligned regions, marking the mosaic image based on the determination result, and dividing the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlapped region of the mosaic line and the misaligned region as a misaligned label, and mark the non-overlapped region as a non-misaligned label;

[0024] A training module, used for inputting the training set into a VGG model for training, calculating a loss function through the predicted image obtained by the VGG model training and the training set, using the loss function as a training basis, and training until the VGG model is stable;

[0025] An evaluation module is used to input the verification set into the trained VGG model, compare the output image of the VGG model with the verification set, and evaluate the correctness of the model output result;

[0026] A weight module is used to obtain an error image in the output image when the correctness is less than a preset accuracy value, re-label the error image, iteratively train the VGG model through the re-labeled error image and evaluate the correctness of the model output result until the correctness of the model output result is greater than the preset accuracy value, and output the model weight of the VGG model;

[0027] The detection module is used to integrate the model weights into a preset detection script, encapsulate the detection script into a detection tool or a detection service, and detect the mosaic dislocation points of the original satellite image in real time based on the detection tool or the detection service.

[0028] In one embodiment, the system further comprises:

[0029] The calculation module is used to obtain the labeling result of the object frame in the mosaic line in the predicted image, compare it with the labeling result of the object frame in the mosaic image in the training set, and calculate the loss function based on the comparison result of the object frame.

[0030] In one embodiment, the system further comprises:

[0031] The statistical module is used to count the number of misplaced labels and the number of non-misplaced labels in the mosaic image, adjust the image data in the mosaic image so that the number of misplaced labels and the number of non-misplaced labels are close to each other, and keep the number of misplaced labels and the number of non-misplaced labels in the training set and the verification set close to each other when dividing the training set and the verification set.

[0032] An embodiment of the present invention provides an electronic device, including a processor and a memory;

[0033] The processor is connected to the memory;

[0034] The memory is used to store executable program code;

[0035] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in one or more embodiments.

[0036] An embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning are implemented.

[0037] In view of the above, in one or more embodiments of the present specification, the mosaic image of the satellite image is preprocessed, the misalignment judgment is performed, and the misalignment area is determined; the mosaic image misalignment area is marked, and the marked mosaic image is divided into a training set and a verification set; the training set is input into the VGG model for training, and the loss function is calculated by the predicted image obtained by the VGG model training and the training set, and the VGG model is trained until it is stable; the verification set is input into the trained VGG model, and the output image of the VGG model is compared with the verification set to evaluate the correctness of the model output result; when the correctness is less than the preset accuracy value, the erroneous image is re-marked, and the VGG model is iteratively trained with the re-marked erroneous image, and the model weight of the VGG model is output; the model weight is integrated into the preset detection script to detect the mosaic misalignment point of the original satellite image in real time. In this way, the above-mentioned model detection can replace manual detection, and realize rapid and large-scale accurate detection of satellite image mosaics. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 This is a flowchart of a method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning provided by an embodiment of this specification.

[0040] Figure 2 It is a structural schematic diagram of an image mosaic misalignment point detection system based on satellite imagery and VGG model fine-tuning provided by an embodiment of the present specification.

[0041] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0042] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed can be changed without departing from the scope of protection of the contents of this specification. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0043] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0044] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning, including:

[0045] Step S101, obtaining a mosaicked image after mosaicking the original satellite images, preprocessing the mosaicked image, and performing misalignment determination on the preprocessed mosaicked image to determine a misalignment area in the mosaicked image.

[0046] Specifically, satellite image mosaicking technology stitches multiple original satellite images to create a map or view of a larger range, which is a mosaicked image after image mosaicking. The mosaicked image is preprocessed, and the preprocessing step may include further processing such as cropping the size of the mosaicked image, adjusting the resolution, and the image format, so as to fit the model training step in the subsequent steps. In addition, in the subsequent model training step, the purpose of training is to determine the mosaicked line area in the mosaicked image, so the mosaicked image can also be directly processed in the preprocessing step to determine the image data of the mosaicked line part. The detailed steps include: determining the mosaicked line in the mosaicked image, generating an object frame of a fixed size (256x256 pixels) along the mosaicked line to intercept the mosaicked line in the mosaicked image, thereby simplifying the image size, wherein the overlap rate of the object frame can be set to 0.7, and the resolution and size can be set according to the details of the ground objects in the image. Then the object frame is overlaid on the mosaicked image to highlight the mosaicked line on the mosaicked image.

[0047] In addition, after preprocessing the mosaic image, the mosaic image can be judged for misalignment. The error judgment can be divided into five dimensions, including: geometric accuracy difference, image feature point offset, discontinuity of the seam area, spatial accuracy analysis, and image overlay error. Among them, the geometric accuracy difference is to quantify the inconsistency of the image by measuring the offset between the edges of key objects. If the deviation exceeds a certain threshold (such as a few pixels), it can be considered that a misalignment has occurred; the image feature point offset is to determine the degree of offset by calculating the Euclidean distance or other similarity measurement between the feature points of multiple preset images, and set a threshold to determine whether it constitutes a misalignment; the discontinuity of the seam area is to analyze the color distribution and texture features near the mosaic line to detect color and texture; the spatial accuracy analysis is to measure the spatial deviation of the key points near the mosaic line to determine the accuracy of the splicing. If the deviation exceeds the preset accuracy requirement, it indicates that there is a misalignment problem; the image overlay error is to check whether the objects in the overlapping area have double contours or are blurred, and further confirm whether there is a misalignment. After the mosaic image is misaligned, the misaligned area in the mosaic image is determined. The misaligned area is generated by map mosaicking and is usually distributed near the mosaicking line.

[0048] Step S102, detect the mosaic line in the mosaic image, determine whether the mosaic line overlaps with the misaligned area, mark the mosaic image based on the determination result, and divide the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlapping area of ​​the mosaic line and the misaligned area as a misaligned label, and the non-overlapping area as a non-misaligned label.

[0049] Specifically, the mosaic line of the original satellite image in the mosaic image is detected, and it is determined whether the mosaic line overlaps with the dislocated area, that is, whether the mosaic line has a dislocated area. When there is a dislocated area on the mosaic line, it is marked as a dislocated label on the mosaic image, such as "1", and when there is no dislocated area on the mosaic line, it is marked as a non-dislocated label on the mosaic image, such as "0". Among them, for the detection of mosaic line dislocation, the mosaic line in the mosaic image can be detected and compared with the dislocated area, and the dislocated mosaic line can be determined by the overlapping part; or the mosaic line can be directly covered on the mosaicked image to determine whether the two sides of the line where the mosaic line passes are dislocated. If dislocation occurs, the area is marked as a dislocated area. The specific detection method can be set according to the detection requirements of the original satellite image. For example, when there is a detection requirement for other areas of the mosaic image except the mosaic line, the former detection method can be used, or the latter detection method can be used to detect the dislocated area near the mosaic line. Then the marked mosaic image is divided into a training set and a validation set. Among them, data labeling can be done through software platforms, such as the LabelMe platform, to determine the category of images and make a data set. Specifically, you can create a project on the LabelMe platform and define two categories: "misalignment" and "no misalignment". Then set the numerical labels respectively: 1 for "misalignment" and 0 for "no misalignment". Upload all mosaic images to the LabelMe platform, and automatically generate a JSON format label file with the same name in the image path, with the "1 / 0" attribute corresponding to the Boolean value "True / False" representing whether the image is misaligned.

[0050] In addition, when allocating data for the training set and validation set, it is also necessary to adjust the class balance of the data, that is, in order to make the training samples for model training in the subsequent steps reasonable, the mosaic image needs to be adjusted. The adjustment step can count the image data (mosaic line object box data) of "no misalignment" (label 0) and "misalignment" (label 1), and then adjust the number of the two to make the numbers of the two close to equal, so as to maintain feature learning in the subsequent model training process. After adjusting the category ratio, divide the data set into a training set and a validation set, for example, divide the data set into 80% (training set) and 20% (validation set) to ensure that the category balance is maintained during the division, that is, the ratio of the two types of samples in the training set and validation set should be similar.

[0051] Step S103, inputting the training set into the VGG model for training, calculating the loss function by using the predicted image obtained by the VGG model training and the training set, using the loss function as a training basis, and training until the VGG model is stable.

[0052] Specifically, the training set is input into the VGG model for training. The VGG model is a deep convolutional neural network (CNN) mainly used for image classification and target recognition tasks. It extracts image features by stacking multiple small-size convolution kernels (3x3), which enhances the ability to capture image details. The format of the training set can be the JSON format label file generated by the LabelMe platform in the above step. The loss function in the training process can be calculated by the predicted image obtained in the training process and the input training set. The predicted image is the result of the classification prediction of each object box (for example, a 256x246 pixel object box) during the model training process, that is, the probability of whether each object box is misaligned or a binary classification label (1 means "misalignment occurs", 0 means "no misalignment occurs"). The training set is the true label of the corresponding object box in the annotated data set, that is, the pre-annotated "misalignment occurs" or "no misalignment occurs". The specific training process may include: in each training round, traversing the entire training set; for each batch of data, forward propagation calculates the prediction result; calculates the loss function value, and updates the model parameters through back propagation until the model tends to be stable. Among them, for the detection of model stability, mIoU (mean Intersection over Union) can be introduced as the main evaluation indicator. mIoU can reflect the accuracy of the model in pixel-level classification. When the accuracy curve of the model training reaches stability, it means that the model training has stabilized and the training is completed.

[0053] Step S104, input the verification set into the trained VGG model, compare the VGG model output image with the verification set, and evaluate the correctness of the model output result.

[0054] Specifically, after completing the model training, it is necessary to verify the generalization ability and robustness of the model, as well as optimize the shortcomings of the model. Input the validation set into the trained VGG model, output the output image corresponding to the validation set through the model, and compare the output image with the pre-marked misaligned part in the validation set. The comparison process includes: checking the images predicted by the model as "misaligned" one by one, comparing them with the validation set, and screening out the parts that are not actually misaligned but are mistakenly marked as misaligned by the model; and checking the image units predicted by the model as "not misaligned" one by one, comparing them with the validation set, and screening out the parts that are actually misaligned but not detected by the model. By comparing and finding the wrong images with the wrong output results of the model, the correctness of the model output results can be evaluated through the wrong images.

[0055] Step S105, when the correctness is less than the preset accuracy value, obtain the error image in the output image, re-label the error image, iteratively train the VGG model through the re-labeled error image and evaluate the correctness of the model output result until the correctness of the model output result is greater than the preset accuracy value, and output the model weight of the VGG model.

[0056] Specifically, when there are many erroneous images output by the model and the correctness of the output result is less than the preset accuracy value, the erroneous images with incorrect annotations are re-labeled, that is, 1 "misalignment occurs" is changed to 0 "no misalignment occurs", and 0 "no misalignment occurs" is changed to 1 "misalignment occurs". Then the VGG model is retrained with the modified erroneous images, wherein the retrained data set is not limited to the modified erroneous images, and can also include any untrained labeled mosaic images. After retraining, the correctness of the model output results is tested again, and compared with the correctness of the model in the last training result, and the above process is repeated, that is, each time based on the results predicted by the new data, the erroneous samples are filtered, the training set is updated, and the model is retrained. Then the convergence condition is set, for example, when the mIoU of the model on the validation set has not been significantly improved for several consecutive iterations, or when the correctness threshold of the predetermined output result is reached, the iteration is stopped, the training steps of the model are completed, and the optimal model weights of the VGG model are output.

[0057] Step S106, integrating the model weight into a preset detection script, encapsulating the detection script into a detection tool or a detection service, and detecting the mosaic misalignment points of the original satellite image in real time based on the detection tool or the detection service.

[0058] Specifically, the best model weights obtained after multiple iterations of optimization are loaded into the detection script, wherein the detection script is written based on the detection process by obtaining the detection process of the mosaic mismatch point, and the detection process is the whole process from the original satellite image to the output mosaic mismatch point, including: collecting the original satellite image, image mosaicking, image preprocessing, input model training, and output detection results, wherein the best model weights of the VGG model are used in the model training process, and then the entire script process is encapsulated into an independent tool or service, which can be a command line tool, a Web application or an API service. After the tool or service is deployed, the mosaic mismatch point of the original satellite image can be detected in real time.

[0059] The embodiment of the present invention provides a method for detecting the misaligned points of image mosaics based on satellite images and VGG model fine-tuning, preprocessing the mosaic images of satellite images, performing misalignment judgment, and determining the misaligned areas; marking the misaligned areas of the mosaic images, and dividing the marked mosaic images into a training set and a verification set; inputting the training set into the VGG model for training, calculating the loss function through the predicted images obtained by the VGG model training and the training set, and training until the VGG model is stable; inputting the verification set into the trained VGG model, comparing the output images of the VGG model with the verification set, and evaluating the correctness of the model output results; when the correctness is less than the preset accuracy value, re-marking the erroneous images, iteratively training the VGG model through the re-marked erroneous images, and outputting the model weights of the VGG model; integrating the model weights into the preset detection script, and detecting the mosaic misaligned points of the original satellite image in real time. In this way, the above-mentioned model detection can replace manual detection, and realize rapid and large-scale accurate detection of satellite image mosaics.

[0060] See also Figure 2 , Figure 2 Schematic diagram of a system for detecting dislocations of image mosaics based on satellite images and VGG model fine-tuning provided in an embodiment of the present application. Figure 2 As shown, the system comprises:

[0061] The acquisition module S201 is used to acquire a mosaicked image after mosaicking the original satellite image, preprocess the mosaicked image, and perform misalignment determination on the preprocessed mosaicked image to determine a misalignment area in the mosaicked image;

[0062] The marking module S202 is used to detect the mosaic line in the mosaic image, determine whether the mosaic line overlaps with the misaligned area, mark the mosaic image based on the determination result, and divide the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlapped area of ​​the mosaic line and the misaligned area as a misaligned label, and mark the non-overlapped area as a non-misaligned label;

[0063] A training module S203 is used to input the training set into a VGG model for training, calculate a loss function using the predicted image obtained by the VGG model training and the training set, and train the VGG model using the loss function as a training basis until the VGG model is stable;

[0064] Evaluation module S204, used for inputting the verification set into the trained VGG model, comparing the output image of the VGG model with the verification set, and evaluating the correctness of the model output result;

[0065] The weight module S205 is used to obtain an error image in the output image when the correctness is less than a preset accuracy value, re-label the error image, iteratively train the VGG model through the re-labeled error image and evaluate the correctness of the model output result until the correctness of the model output result is greater than the preset accuracy value, and output the model weight of the VGG model;

[0066] The detection module S206 is used to integrate the model weight into a preset detection script, encapsulate the detection script into a detection tool or a detection service, and detect the mosaic dislocation points of the original satellite image in real time based on the detection tool or the detection service.

[0067] In another embodiment, a system for detecting misaligned points of image mosaics based on satellite images and VGG model fine-tuning further includes:

[0068] The calculation module is used to obtain the labeling result of the object frame in the mosaic line in the predicted image, compare it with the labeling result of the object frame in the mosaic image in the training set, and calculate the loss function based on the comparison result of the object frame.

[0069] In another embodiment, a system for detecting misaligned points of image mosaics based on satellite images and VGG model fine-tuning further includes:

[0070] The statistical module is used to count the number of misplaced labels and the number of non-misplaced labels in the mosaic image, adjust the image data in the mosaic image so that the number of misplaced labels and the number of non-misplaced labels are close to each other, and keep the number of misplaced labels and the number of non-misplaced labels in the training set and the verification set close to each other when dividing the training set and the verification set.

[0071] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0072] Each processing unit and / or module of the embodiments of the present application may be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or may be implemented by software that executes the functions described in the embodiments of the present application.

[0073] See also Figure 3 , which shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, the electronic device can be used to implement Figure 1 The method in the embodiment shown. Figure 3As shown, the electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0074] The communication bus 302 is used to realize the connection and communication between these components.

[0075] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0076] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0077] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts of the entire electronic device 300, and executes various functions and processes data of the terminal 300 by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (ProgrammableLogic Array, PLA). The processor 301 can integrate one or a combination of a processor (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0078] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally be at least one storage device located away from the aforementioned processor 301. As Figure 3 As shown, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0079] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the interactive application based on image generation stored in the memory 305, and specifically perform the following operations: pre-process the mosaic image of the satellite image, perform misalignment judgment, and determine the misalignment area; mark the misalignment area of ​​the mosaic image, and divide the marked mosaic image into a training set and a verification set; input the training set into the VGG model for training, calculate the loss function through the predicted image obtained by the VGG model training and the training set, and train until the VGG model is stable; input the verification set into the trained VGG model, compare the output image of the VGG model with the verification set, and evaluate the correctness of the model output result; when the correctness is less than the preset accuracy value, re-mark the erroneous image, iteratively train the VGG model through the re-marked erroneous image, and output the model weight of the VGG model; integrate the model weight into the preset detection script to detect the mosaic misalignment point of the original satellite image in real time.

[0080] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0081] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0082] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0087] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct related hardware. The program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0088] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for detecting image mosaic misalignment points based on satellite images and VGG model fine-tuning, the method comprising: Acquire a mosaic image obtained by mosaicking the original satellite image, preprocess the mosaic image, and perform misalignment determination on the preprocessed mosaic image to determine a misalignment area in the mosaic image, wherein the misalignment determination includes: geometric accuracy difference, image feature point offset, discontinuity of a seam area, spatial accuracy analysis, and image superposition error; Detecting the mosaic line in the mosaic image, determining whether the mosaic line overlaps with the dislocation area, marking the mosaic image based on the determination result, and dividing the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlap area of ​​the mosaic line and the dislocation area as a dislocation label, and mark the non-overlap area as a non-dislocation label; Input the training set into the VGG model for training, calculate the loss function through the predicted image obtained by the VGG model training and the training set, use the loss function as a training basis, and train until the VGG model is stable; Input the validation set into the trained VGG model, compare the output image of the VGG model with the validation set, and evaluate the correctness of the model output result; When the correctness is less than the preset accuracy value, an error image in the output image is obtained, the error image is re-labeled, the VGG model is iteratively trained through the re-labeled error image, and the correctness of the model output result is evaluated until the correctness of the model output result is greater than the preset accuracy value, and the model weight of the VGG model is output; The model weights are integrated into a preset detection script, the detection script is encapsulated into a detection tool or a detection service, and the mosaic misalignment points of the original satellite image are detected in real time based on the detection tool or the detection service.

2. The method according to claim 1, characterized in that The predicted image obtained by the VGG model training and the loss function of the training set are calculated, including: Obtain the labeling result of the object frame in the mosaic line in the predicted image, compare it with the labeling result of the object frame in the mosaic image in the training set, and calculate the loss function based on the comparison result of the object frame.

3. The method according to claim 1, characterized in that The step of dividing the labeled mosaic image into a training set and a validation set comprises: The number of misplaced labels and the number of non-misplaced labels in the mosaicked image are counted, and the image data in the mosaicked image are adjusted so that the number of misplaced labels and the number of non-misplaced labels are approximately equal. When dividing the training set and the validation set, the number of misplaced labels and the number of non-misplaced labels in the training set and the validation set are kept approximately equal.

4. The method according to claim 1, characterized in that: The method further comprises: A detection process for mosaic dislocation points is obtained, and a detection script is written based on the detection process. The detection process includes: collecting original satellite images, image mosaicking, image preprocessing, input model training, and outputting detection results.

5. A detection system for image mosaic misalignment based on satellite images and VGG model fine-tuning, characterized in that: The system comprises: An acquisition module is used to acquire a mosaicked image after mosaicking the original satellite image, preprocess the mosaicked image, and perform misalignment determination on the preprocessed mosaicked image to determine the misaligned area in the mosaicked image. The misalignment determination includes: geometric accuracy difference, image feature point offset, discontinuity of the seam area, spatial accuracy analysis, and image superposition error. a marking module, for detecting mosaic lines in the mosaic image, determining whether the mosaic lines overlap with the misaligned regions, marking the mosaic image based on the determination result, and dividing the marked mosaic image into a training set and a validation set, wherein the marking is to mark the overlapped region of the mosaic line and the misaligned region as a misaligned label, and mark the non-overlapped region as a non-misaligned label; A training module, used for inputting the training set into a VGG model for training, calculating a loss function through the predicted image obtained by the VGG model training and the training set, using the loss function as a training basis, and training until the VGG model is stable; An evaluation module is used to input the verification set into the trained VGG model, compare the output image of the VGG model with the verification set, and evaluate the correctness of the model output result; A weight module is used to obtain an error image in the output image when the correctness is less than a preset accuracy value, re-label the error image, iteratively train the VGG model through the re-labeled error image and evaluate the correctness of the model output result until the correctness of the model output result is greater than the preset accuracy value, and output the model weight of the VGG model; The detection module is used to integrate the model weights into a preset detection script, encapsulate the detection script into a detection tool or a detection service, and detect the mosaic dislocation points of the original satellite image in real time based on the detection tool or the detection service.

6. The system according to claim 5, characterized in that The system further comprises: The calculation module is used to obtain the labeling result of the object frame in the mosaic line in the predicted image, compare it with the labeling result of the object frame in the mosaic image in the training set, and calculate the loss function based on the comparison result of the object frame.

7. The system according to claim 5, characterized in that The system further comprises: The statistical module is used to count the number of misplaced labels and the number of non-misplaced labels in the mosaic image, adjust the image data in the mosaic image so that the number of misplaced labels and the number of non-misplaced labels are close to each other, and keep the number of misplaced labels and the number of non-misplaced labels in the training set and the verification set close to each other when dividing the training set and the verification set.

8. An electronic device comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images