Ship target change detection method, device and equipment and storage medium

Abstract

The embodiment of the invention relates to the field of remote sensing image target change detection, and discloses a ship target change detection method, device and equipment and a storage medium, and the method comprises the steps: obtaining two adjacent time phase images of a port region; extracting a change region of interest image in the two adjacent time phase images; ship target detection is carried out on the change interest area image in the two adjacent time phase images, and a ship target detection frame of the two adjacent time phase images is obtained; calculating the intersection-to-union ratio between the ship target detection frames of the two adjacent time phase images; and determining a ship target change condition according to the ship target detection frames of the two adjacent time phase images and the intersection-to-union ratio. The problems of intelligent detection and positioning of ship changes in remote sensing images are solved, and the detection capability and the detection precision of ship targets are effectively improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of remote sensing image target change detection, and particularly to a ship target change detection method, device, equipment and storage medium. Background Art

[0002] Remote sensing image target change detection is a process of using multi-temporal images to discover the change situation of interested targets. With the continuous development of earth observation technology, a large number of multi-source multi-temporal remote sensing images covering the same area have been accumulated, which provides a very rich data source for analyzing the change situation of a specific area. Through target change detection and analysis, people can quickly and accurately master the change situation of interested targets such as airplanes and ships, which is of great significance for analyzing the battlefield situation and monitoring the dynamics of entering and leaving ports. Summary of the Invention

[0003] The purpose of the present invention is to provide at least a ship target change detection method, device, equipment and storage medium, which can at least solve the problem of intelligent detection and positioning of ship changes in remote sensing images, and effectively improve the detection ability and detection accuracy of ship targets.

[0004] To solve the above technical problems, the embodiments of the present invention provide a ship target change detection method, including:

[0005] Obtain adjacent two-temporal images of the port area;

[0006] Extract the change of interest area images in the adjacent two-temporal images;

[0007] Perform ship target detection on the change of interest area images in the adjacent two-temporal images to obtain the ship target detection frames of the adjacent two-temporal images;

[0008] Calculate the intersection over union between the ship target detection frames of the adjacent two-temporal images;

[0009] Determine the ship target change situation according to the ship target detection frames of the adjacent two-temporal images and the intersection over union.

[0010] In some optional embodiments, before extracting the change of interest area images in the adjacent two-temporal images, it further includes:

[0011] Preprocess the adjacent two-temporal images, and the preprocessing includes geometric registration and radiometric correction.

[0012] In some optional embodiments, the change of interest area includes the area where ship changes may occur within the port area, and the change of interest area is represented by vector information.

[0013] In some alternative embodiments, the method for detecting changes in ship targets further includes: establishing a region of interest in change based on preset conditions; the preset conditions include at least one of the following:

[0014] The region of interest in change contains the docking positions of the ship targets of interest;

[0015] The region of interest in change is delimited along the dock where the ships are docked;

[0016] The long side of the region of interest in change is greater than the length of the ship target docked at the dock;

[0017] The short side of the region of interest in change is greater than an integer multiple of the width of the ship target docked at the dock.

[0018] In some alternative embodiments, the pre-trained DDMNet model is used to detect ship targets in the images of the regions of interest in change in the two adjacent temporal images, and the detection frames of ship targets in the two adjacent temporal images are obtained.

[0019] In some alternative embodiments, the intersection over union (IoU) between the detection frames of ship targets in the two adjacent temporal images is calculated using the following formula;

[0020] α = S c / S d

[0021] In the formula, α represents the intersection over union, S c represents the overlapping area between the detection frames of ship targets in the two adjacent temporal images, and S d represents the total area after the superposition of the detection frames of ship targets in the two adjacent temporal images.

[0022] In some alternative embodiments, the detection frames of ship targets in the two adjacent temporal images include the first detection frame of ship targets in the previous temporal image and the second detection frame of ship targets in the current temporal image;

[0023] Based on the detection frames of ship targets in the two adjacent temporal images and the intersection over union, the change situation of ship targets is determined, including:

[0024] If the area of the first detection frame of ship targets is 0 and the area of the second detection frame of ship targets is greater than 0, then the ship target corresponding to the second detection frame of ship targets is a newly added ship target;

[0025] If the area of the first detection frame of ship targets is greater than 0 and the area of the second detection frame of ship targets is 0, then the ship target corresponding to the first detection frame of ship targets is a disappeared ship target;

[0026] If the areas of both the first ship target detection box and the second ship target detection box are greater than 0, and the overlapping area between the first ship target detection box and the second ship target detection box is 0, then the ship target corresponding to the first ship target detection box is a disappearing ship target, and the ship target corresponding to the second ship target detection box is a newly added ship target;

[0027] If the areas of both the first ship target detection box and the second ship target detection box and the overlapping area between the first ship target detection box and the second ship target detection box are greater than 0, and the intersection-over-union ratio between the first ship target detection box and the second ship target detection box is less than a set threshold, then the ship target corresponding to the first ship target detection box is a disappearing ship target, and the ship target corresponding to the second ship target detection box is a newly added ship target;

[0028] If the areas of both the first ship target detection box and the second ship target detection box and the overlapping area between the first ship target detection box and the second ship target detection box are greater than 0, and the intersection-over-union ratio between the first ship target detection box and the second ship target detection box is not less than a set threshold, then both the ship target corresponding to the first ship target detection box and the ship target corresponding to the second ship target detection box are unchanged ship targets.

[0029] An embodiment of the present invention further provides a ship target change detection device, including:

[0030] An image acquisition module, configured to acquire adjacent two-phase images of a port area;

[0031] An image extraction module, configured to extract a change region of interest image from the adjacent two-phase images;

[0032] A target detection module, configured to perform ship target detection on the change region of interest image in the adjacent two-phase images to obtain ship target detection boxes of the adjacent two-phase images;

[0033] An intersection-over-union ratio calculation module, configured to calculate the intersection-over-union ratio between the ship target detection boxes of the adjacent two-phase images;

[0034] A change determination module, configured to determine the ship target change situation according to the ship target detection boxes of the adjacent two-phase images and the intersection-over-union ratio.

[0035] An embodiment of the present invention further provides an electronic device, including:

[0036] At least one processor; and,

[0037] A memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned ship target change detection method.

[0039] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned ship target change detection method is implemented.

[0040] This method is directed to remote sensing images of port areas. On the basis of establishing change regions of interest, ship detection is performed on the images of the change regions of interest, effectively reducing background interference. By calculating the intersection-over-union ratio of ship detection frames on new and old phase images, the change detection of ship targets can be automatically, quickly, and accurately achieved, and then the effective distinction between newly added ships and disappeared ships can be realized. It avoids the limitations brought by manually designed features in traditional methods, and thus can obtain more accurate target change detection results, solves the problem of intelligent detection and positioning of ship changes in remote sensing images, and effectively improves the detection ability and detection accuracy for ship targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] One or more embodiments are illustrated by way of example in the accompanying drawings, and these exemplary illustrations do not constitute a limitation on the embodiments.

[0042] Figure 1 It is a schematic diagram of the first strategy of the target change detection method for two-phase images in the prior art;

[0043] Figure 2 It is a schematic diagram of the second strategy of the target change detection method for two-phase images in the prior art;

[0044] Figure 3 It is a schematic flowchart of a ship target change detection method provided by an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the establishment of a change region of interest provided by an embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of the difference in the sampling methods of standard convolution and deformable convolution provided by an embodiment of the present invention;

[0047] Figure 6 It is a schematic flowchart of the ship target change detection process provided by an embodiment of the present invention;

[0048] Figure 7 It is a schematic diagram of a ship target change detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present invention, many technical details are provided to help readers better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present invention can still be implemented. The following division of each embodiment is for convenience of description and should not constitute any limitation to the specific implementation manner of the present invention. Each embodiment can be combined and cross-referenced with each other on the premise of not being contradictory.

[0050] Remote sensing image target change detection is a process of using multi-temporal images to discover changes in targets of interest. With the continuous development of earth observation technology, a large amount of multi-source and multi-temporal remote sensing images covering the same area have been accumulated, which provides a very rich data source for analyzing the changes in a specific area. Through target change detection and analysis, people can quickly and accurately grasp the changes in targets of interest such as airplanes and ships, which is of great significance for analyzing battlefield situations and monitoring the dynamics of ships entering and leaving ports.

[0051] How to improve the intelligent level of ship change detection and positioning in remote sensing images and enhance the detection ability and detection accuracy for ship targets has become an urgent problem to be solved.

[0052] In the existing two-temporal image target change detection methods, according to the sequence of target detection and change detection, there are mainly two solution strategies:

[0053] The first strategy is as Figure 1 shown. After preprocessing the input two-temporal images, the change detection method (such as the difference method) is used to discover the change area, then the target type corresponding to the change area is judged, the target types of interest are retained, and finally the change situation of the target is analyzed. This strategy is severely affected by the change detection link, and the obtained results may be relatively fragmented, and the complete target change range cannot be obtained. The change detection link is easily affected by the change situations of complex ground objects, such as different seasons, shooting times, and shooting angles, which makes the originally unchanged ground objects easily detected as changed ground objects. In some cases, some targets may move a small distance, and this strategy may also misdetect them as changed targets. Therefore, the detection ability and detection accuracy of this strategy are relatively low. In addition, this strategy can only detect changed targets (such as newly added targets or disappeared targets), but cannot obtain unchanged targets.

[0054] The second strategy is as Figure 2As shown in the figure, based on the preprocessing of two-phase images, target detection is performed on the two images respectively to extract the position information of the targets of interest. Then, by synthesizing the target detection results of the two phases, it is analyzed and judged which targets are newly added targets, which targets are disappeared targets, and which targets are unchanged targets. This strategy is severely affected by the accuracy of the target detection algorithm. The error of target detection will be magnified exponentially in the target change detection results (because target detection needs to be performed on the two images separately). For example, if the correct rate of the target detection algorithm is 90%, then the correct rate of the target change detection may only be 80%. The error in the target detection link will be magnified exponentially. Since the images of the two phases need to be detected separately, the error in the change inspection link is the product of the errors of the previous two target detections, resulting in limited accuracy of the final target change detection. Therefore, the detection ability and detection accuracy of this strategy are relatively low.

[0055] In addition, in the target detection link of the existing target change detection methods, since manually designed features are mainly used for ship detection, the accuracy and intelligence level of the target detection link are not high and it is not suitable for popularization and application.

[0056] To solve the above technical problems, the present invention proposes a ship target change detection method, device, equipment and storage medium. The following specifically describes the implementation details of the embodiments of the present invention. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing the solution.

[0057] Example 1

[0058] A ship target change detection method provided in this embodiment can be applied to an electronic device with communication, computing and data storage capabilities. Its specific process can be as Figure 3 shown, including:

[0059] Step 101, obtain adjacent two-phase images of the port area.

[0060] In some specific implementations, before extracting the change interest area images in the adjacent two-phase images, preprocess the adjacent two-phase images. Through multi-phase image preprocessing, the problems of inaccurate image position matching and large radiation value differences are reduced in advance to ensure the comparability of different images in terms of spatial position and radiation value. The preprocessing includes geometric registration and radiation correction.

[0061] In one example, taking the new-phase image as the reference image and the old-phase image as the correction image, use the Registration module in ENVI5.3 software to generate matching points and perform geometric registration processing on the multi-phase images. The radiation correction is completed by the histogram matching method.

[0062] Using the general remote sensing software ENVI for image registration and radiometric correction, multi-temporal image preprocessing can be conveniently achieved, ensuring the comparability of images taken at different times in terms of spatial position and radiometric values. The results after preprocessing can be directly saved and output following the ENVI software. The preprocessing operation has strong compatibility, is easy to implement, and is applicable to the processing of different types of satellite images.

[0063] Step 102: Extract the changed region of interest images in two adjacent temporal images.

[0064] The changed region of interest includes the areas within the port region where ship changes may occur, and the changed region of interest is represented by vector information.

[0065] In specific implementation, the changed region of interest is mainly established by analyzing the ship docking rules in the target port region. Since ships are a type of target with a large aspect ratio, the changed region of interest can be represented as a polygon vector, such as a rectangular vector. Taking the rectangular vector as an example, the corner coordinates of the rectangular vector can be arranged in a clockwise order for representation, as Figure 4 shown. Based on analyzing the ship docking rules in the port region, the changed region of interest can be established with the new temporal image as the background, and the vector information of the changed region of interest is transmitted to the old temporal image.

[0066] When determining the specific position and size of the region of interest, it can be determined in combination with the common types of docked ships in this port region, mainly considering the following principles:

[0067] a. The position of the changed region of interest is selected at the docking position of the ship target of interest, such as the common docking wharf for warships;

[0068] b. The angle of the changed region of interest is delimited along the wharf, that is, the long side of the region of interest is basically consistent with the outer edge direction of the wharf;

[0069] c. Considering that the docking position of each ship may not be exactly the same, the long side of the changed region of interest is slightly larger than the length of the ships commonly docked at this wharf;

[0070] d. Considering the possible situation of multiple ships docking side by side, the narrow side of the changed region of interest is 2 to 3 times the width of the ships commonly docked at this wharf.

[0071] Therefore, the method of this embodiment further includes:

[0072] Establishing the changed region of interest based on preset conditions; the preset conditions include at least one of the following:

[0073] The changed region of interest contains the docking position of the ship target of interest;

[0074] The changed region of interest is delimited along the wharf of the docked ship target;

[0075] The long side of the variable region of interest is greater than the length of the ship target docked at the pier;

[0076] The short side of the variable region of interest is greater than an integer multiple of the width of the ship target docked at the pier, for example, 2 to 3 times.

[0077] By analyzing the ship docking rules at the port to delimit the variable region of interest, the interference of the complex background of the remote sensing image can be better reduced. This method is applicable to the change detection task of medium and large ships.

[0078] In this embodiment, all the images covered by the variable region of interest are extracted, which can lay a foundation for subsequent ship detection. Since the vector of the variable region of interest is established on the new-phase image, it is necessary to transfer this vector to the old-phase image after the registration of the new and old-phase images is completed. Suppose there are N ship variable regions of interest in a certain port, then the total number of region-of-interest images obtained from the new and old-phase images is 2N.

[0079] Considering that the input image during deep learning training is rectangular, when determining the boundary of the variable region-of-interest image, the extreme values of the corner points of the variable region of interest can be taken, that is: the upper-left image coordinates (X 1 , Y 1 ) and the lower-right image coordinates (X 2 , Y 2 ), which can be respectively expressed as Equation (1) and Equation (2). The gray values of the pixels outside the boundary of the variable region of interest are uniformly filled with 0, that is: represented by black.

[0080] (X 1 , Y 1 ) = [min(x i ), min(y j )] (1)

[0081] (X 2 , Y 2 ) = [max(x i ), max(y j )] (2)

[0082] In the formula, i, j ∈ [1, 2, 3, 4].

[0083] By designing the above specific principles for establishing the area of interest in changes, the range of the area of interest in changes is particularly enlarged in a certain proportion. It not only considers the general rules of ship docking in the port but also accommodates some occasional situations, such as the phenomenon of multiple ships docking side by side. The method for establishing the area of interest in changes of ships is simple and direct, easy to operate, and since the port terminal changes less frequently, it does not need to be updated frequently in the subsequent target change detection problems of other images, making the whole method convenient to apply, with strong practicability and accuracy. Since the two-phase images have been geometrically registered, that is, the corresponding relationship of all pixel points has been constructed, the area of interest constructed on one-phase image can be directly transferred to the other-phase image, and then the local image corresponding to the area of interest can be extracted.

[0084] Step 103: Detect ship targets in the area-of-interest images of two adjacent-phase images to obtain the ship target detection frames of two adjacent-phase images.

[0085] Deep learning technology has achieved very remarkable results in many fields such as remote sensing image target detection and text recognition. Models such as Faster Region-based Convolution Networks (Faster R-CNN) and You Only LookOnce (YOLO) have better realized target detection. However, due to the lack of full utilization of the multi-scale features of remote sensing images, the detection ability for ship-like targets is limited. In related methods, the Dual-resolution Deformable Multihead Network (DDMNet) architecture is proposed. The introduction of deformable convolution improves the detection ability for oriented targets such as ships, and better realizes the detection and positioning of ships in remote sensing images. On this basis, DDMNet introduces the ideas of deformable convolution and dilated convolution, designs a deformable feature fusion module based on dilated convolution, further improves the perception effect of the network architecture for target direction and shape changes, and realizes the fast and accurate detection of ship targets in remote sensing images. The difference in the sampling methods between standard convolution and deformable convolution is as Figure 5 shown, where a represents standard convolution, b represents conventional deformable convolution, c represents adapting to target scale changes, and d represents adapting to target rotation changes. Taking the standard 3×3 convolution kernel as an example, it can be found that the receptive field of deformable convolution is not a regular square, but can adjust the receptive field range according to target scale, rotation, etc., and is more suitable for the detection of ship-like targets.

[0086] In some specific implementations, a pre-trained DDMNet model is used to detect ship targets in the images of the regions of interest in two adjacent temporal images, and the ship target detection frames for the two adjacent temporal images are obtained. In other specific implementations, it can also be replaced by other similar deep learning object detection models. The effect of object detection depends on the coverage ability and the number of samples of the training samples used for different types of ships.

[0087] In one example, the FGSD2021 dataset is used in the training of the DDMNet model. This dataset annotates 20 ship classes, covers multiple ports around the world, and contains a total of 636 images with a resolution of about 1 meter. Among them, there are 424 training images and 212 test images, which are more suitable for the model training of this method. Using the DDMNet model trained with the FGSD2021 dataset to detect ships in the above-mentioned images of the regions of interest, since the FGSD2021 dataset covers data from multiple ports around the world and has a relatively comprehensive coverage of various medium and large-sized ships, it is more suitable for dealing with the ship target change detection problem of this method. At the same time, the DDMNet model introduces the idea of deformable convolution and has good adaptability for ship target detection.

[0088] Step 104, calculate the intersection over union (IoU) between the ship target detection frames of two adjacent temporal images.

[0089] Specifically, the ship target detection frames of two adjacent temporal images include the first ship target detection frame of the previous temporal image and the second ship target detection frame of the current temporal image.

[0090] After using the DDMNet model to complete the ship detection of the images of the regions of interest in the old and new temporal phases, it is necessary to calculate the intersection over union (IoU) of the detection frames in the old and new temporal phases, and then determine whether there is a change in the ship at this position. Let the area of the detection frame K 1 in the old temporal phase be S 1 , and the area of the detection frame K 2 in the new temporal phase be S 2 . Then, the intersection over union α of the two mainly uses the overlapping area S c of the detection frames in the old and new temporal phases and the total area S d after the detection frames are superimposed for calculation, as shown in Equation (3).

[0091] In a specific implementation, to calculate the intersection over union between the ship target detection frames of two adjacent temporal images, the following calculation formula is adopted;

[0092] α = S c / S d (3)

[0093] In the formula, α represents the intersection over union, and S cRepresents the overlapping area between the ship target detection frames of adjacent two-phase images, S d Represents the total area after superimposing the ship target detection frames of adjacent two-phase images.

[0094] Step 105: Determine the change situation of the ship target according to the ship target detection frames of adjacent two-phase images and the intersection over union.

[0095] In specific implementation, by setting a certain threshold δ, it is judged whether the detection frame changes. According to the area size and comparison situation of different detection frames, determine the change situation of the ship target according to the ship target detection frames of adjacent two-phase images and the intersection over union. The discrimination rules include:

[0096] If the area of the first ship target detection frame is 0 and the area of the second ship target detection frame is greater than 0, that is: S 1 = 0, S 2 > 0, then the ship target corresponding to the second ship target detection frame K 2 is a newly added ship target;

[0097] If the area of the first ship target detection frame is greater than 0 and the area of the second ship target detection frame is 0, that is: S 1 > 0, S 2 = 0, then the ship target corresponding to the first ship target detection frame K 1 is a disappearing ship target;

[0098] If the areas of both the first ship target detection frame and the second ship target detection frame are greater than 0, and the overlapping area between the first ship target detection frame and the second ship target detection frame is 0, that is: S 1 > 0, S 2 > 0, S c = 0, then the ship target corresponding to the first ship target detection frame K 1 is a disappearing ship target, and the ship target corresponding to the second ship target detection frame K 2 is a newly added ship target;

[0099] If the areas of the first ship target detection frame, the second ship target detection frame, and the overlapping area between the first ship target detection frame and the second ship target detection frame are all greater than 0, that is S 1 > 0, S 2 > 0, S c > 0, and the intersection over union between the first ship target detection frame K 1 and the second ship target detection frame K 2 is less than the set threshold, that is α < δ, then the ship target corresponding to the first ship target detection frame K 1 is a disappearing ship target, and the ship target corresponding to the second ship target detection frame K 2 is a newly added ship target;

[0100] If the areas of the first ship target detection box, the second ship target detection box, and the overlapping area between the first ship target detection box and the second ship target detection box are all greater than 0, that is, S 1 > 0, S 2 > 0, S c > 0, and the intersection over union between the first ship target detection box and the second ship target detection box is not less than the set threshold, that is, α ≥ δ, then the ship target corresponding to the first ship target detection box K 1 and the ship target corresponding to the second ship target detection box K 2 are both unchanged ship targets.

[0101] By designing the above calculation formula for the intersection over union, and at the same time giving the discrimination rules for newly added ship targets, disappeared ship targets, and unchanged ship targets, it can take into account special situations such as small displacements of ships and side-by-side parking, so as to more accurately obtain the ship change results. The output result is the target change information in vector form, including three categories: newly added, disappeared, and unchanged targets. It can not only detect the change information of a single ship, but also detect all the changed ships in the case of multiple ships side by side, with strong applicability and great potential value in practical applications.

[0102] This method is aimed at remote sensing images of port areas. On the basis of establishing the region of interest for change, the DDMNet model is used to detect ships in the region of interest image of change, effectively reducing background interference. By calculating the intersection over union of ship detection boxes on the old and new phase images, the effective distinction between newly added ships and disappeared ships can be achieved. The process is shown as Figure 6 shown, including multi-temporal image preprocessing, establishing the region of interest for change, extracting the region of interest image, DDMNet model target detection, intersection over union calculation and change analysis, which can automatically, quickly, and accurately realize the change detection of ship targets. If directly performing deep learning ship detection on the original two-temporal images and then comparing the target change situations, it will cause the error of ship detection to be transmitted to the final target change detection. However, this method reduces the research scope of ship target change detection to the region of interest for change, effectively reducing the influence of the complex background of remote sensing images. By introducing a deep learning model for ship detection, it avoids the limitations brought by manually designed features in traditional methods, and thus can obtain more accurate target change detection results.

[0103] In addition, this method has strong scalability. By establishing regions of interest for change covering more ports, rapid monitoring of ship targets in multiple regions can be realized. The method process supports full automation and can be effectively applied to ship target change detection tasks at the global scale.

[0104] Example 2

[0105] Another embodiment of the present invention relates to a ship target change detection device. The implementation details of the ship target change detection device in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution. The schematic diagram of the ship target change detection device in this embodiment can be as follows Figure 7 shown, including an image acquisition module 201, an image extraction module 202, a target detection module 203, an intersection over union calculation module 204, and a change determination module 205.

[0106] The image acquisition module 201 is used to acquire adjacent two-phase images of the port area;

[0107] The image extraction module 202 is used to extract the images of the regions of interest in the adjacent two-phase images;

[0108] The target detection module 203 is used to detect ship targets in the images of the regions of interest in the adjacent two-phase images to obtain the ship target detection frames in the adjacent two-phase images;

[0109] The intersection over union calculation module 204 is used to calculate the intersection over union between the ship target detection frames in the adjacent two-phase images;

[0110] The change determination module 205 is used to determine the ship target change situation according to the ship target detection frames in the adjacent two-phase images and the intersection over union.

[0111] In some specific implementations, before extracting the images of the regions of interest in the adjacent two-phase images, preprocessing is performed on the adjacent two-phase images. Through multi-phase image preprocessing, the problems of inaccurate image position matching and large radiation value differences are reduced in advance to ensure the comparability of different images in terms of spatial position and radiation value. The preprocessing includes geometric registration and radiometric correction.

[0112] In one example, taking the new-phase image as the reference image and the old-phase image as the correction image, the Registration module in ENVI5.3 software is used to generate matching points and perform geometric registration processing on the multi-phase images, and the radiometric correction is completed using the histogram matching method.

[0113] Using the general remote sensing software ENVI for image registration and radiometric correction can more conveniently implement multi-phase image preprocessing, ensure the comparability of images taken at different times in terms of spatial position and radiation value, and the results after preprocessing can be directly saved and output relying on the ENVI software. The preprocessing operation has strong compatibility, is easy to implement, and is applicable to the processing of different types of satellite images.

[0114] The regions of interest in the change include the regions in the port area where ship changes may occur, and the regions of interest in the change are represented by vector information.

[0115] In specific implementation, the region of interest of change is mainly established by analyzing the docking rules of ships in the target port area. Since ships belong to a type of target with a large aspect ratio of length to width, the region of interest of change can be represented as a polygon vector, such as a rectangular vector. Taking the rectangular vector as an example, the corner coordinates of the rectangular vector can be arranged in a clockwise order for representation, as shown in Figure 4 shown. Based on the analysis of the docking rules of ships in the port area, the region of interest of change can be established with the new-phase image as the background, and the vector information of the region of interest of change is transmitted to the old-phase image.

[0116] In specific implementation, the image extraction module 202 is further configured to: establish a region of interest of change based on a preset condition; the preset condition includes at least one of the following:

[0117] The region of interest of change includes the docking position of the ship target of interest;

[0118] The region of interest of change is delimited along the dock of the ship target at berth;

[0119] The long side of the region of interest of change is greater than the length of the ship target docked at the dock;

[0120] The short side of the region of interest of change is greater than an integer multiple of the width of the ship target docked at the dock, for example, 2 to 3 times.

[0121] By analyzing the docking rules of port ships to delimit the region of interest of change, the interference of the complex background of remote sensing images can be better reduced, which is applicable to the change detection task of medium and large ships.

[0122] All the images covered by the region of interest of change extracted in this embodiment can lay a foundation for subsequent ship detection. Since the region of interest of change vector is established on the new-phase image, it is necessary to transmit this vector to the old-phase image after the registration of the new and old-phase images is completed. Suppose there are N ship regions of interest of change in a certain port, then the total number of region of interest images obtained on the new and old-phase images is 2N.

[0123] Considering that the input image during deep learning training is rectangular, when determining the boundary of the region of interest of change image, the extreme values of the corner points of the region of interest of change can be taken, that is: the upper left corner image coordinates (X 1 , Y 1 ) and the lower right corner image coordinates (X 2 , Y 2 ), which can be respectively expressed as formula (1) and formula (2). The gray values of the pixels outside the boundary of the region of interest of change are filled with 0 uniformly, that is: represented by black.

[0124] (X 1 , Y 1 ) = [min(x i ), min(yj )] (1)

[0125] (X 2 ,Y 2 ) = [max(x i ), max(y j )] (2)

[0126] where i, j ∈ [1, 2, 3, 4].

[0127] By designing the above specific principles for establishing the area of interest in change, the range of the area of interest in change is particularly enlarged in a certain proportion. It not only considers the general rules of ship docking in the port but also accommodates some accidental situations, such as the phenomenon of multiple ships docking side by side. The establishment of the area of interest in ship change is simple and direct, easy to operate, and since the port terminal changes less frequently, it does not need to be updated frequently in the subsequent target change detection problems of other images, making the application convenient and having strong practicability and accuracy. Since the two-phase images have been geometrically registered, that is, the corresponding relationship of all pixel points has been constructed, the area of interest constructed on one-phase image can be directly transferred to the other-phase image, and then the local image corresponding to the area of interest can be extracted.

[0128] In some specific implementations, the pre-trained DDMNet model is used to detect ship targets in the area of interest images of adjacent two-phase images, and the ship target detection frames of the adjacent two-phase images are obtained. In some other specific implementations, other similar deep learning target detection models can also be used instead. The effect of target detection depends on the coverage ability and the number of samples of the training samples for different types of ships.

[0129] In an example, the FGSD2021 dataset is used in the training of the DDMNet model. The dataset labels up to 20 ship classes, covers multiple ports around the world, and contains a total of 636 images with a resolution of about 1 meter, including 424 training images and 212 test images, which is more suitable for the model training of this device. Using the DDMNet model trained with the FGSD2021 dataset to detect ships in the above area of interest images, since the FGSD2021 dataset covers data from multiple ports around the world and has a relatively comprehensive coverage of various medium and large ships, it is more suitable for dealing with the ship target change detection problem of this device. At the same time, the DDMNet model introduces the idea of deformable convolution and has good adaptability for ship target detection.

[0130] The ship target detection boxes in two adjacent temporal images include the first ship target detection box in the previous temporal image and the second ship target detection box in the current temporal image. After using the DDMNet model to complete the ship detection in the region of interest images of the old and new temporal phases, it is necessary to calculate the intersection over union (IoU) of the detection boxes in the old and new temporal phases, and then determine whether there is a change in the position of the ship. Let the detection box K in the old temporal phase 1 have an area of S 1 , and the detection box K in the new temporal phase 2 have an area of S 2 . Then, the intersection over union α of the two is mainly calculated using the ratio of the overlapping area S c of the detection boxes in the old and new temporal phases to the total area S d after the detection boxes are superimposed, as shown in Equation (3).

[0131] In the specific implementation, to calculate the intersection over union between the ship target detection boxes in two adjacent temporal images, the following calculation formula is used;

[0132] α = S c / S d (3)

[0133] In the formula, α represents the intersection over union, S c represents the overlapping area between the ship target detection boxes in two adjacent temporal images, and S d represents the total area after the ship target detection boxes in two adjacent temporal images are superimposed.

[0134] In the specific implementation, by setting a certain threshold δ, it is determined whether the detection box has changed. According to the sizes of different detection boxes and comparison situations, based on the ship target detection boxes and intersection over union in two adjacent temporal images, the change situation of the ship target is determined, and the discrimination rules include:

[0135] If the area of the first ship target detection box is 0 and the area of the second ship target detection box is greater than 0, that is: S 1 = 0, S 2 > 0, then the ship target corresponding to the second ship target detection box K 2 is a newly added ship target;

[0136] If the area of the first ship target detection box is greater than 0 and the area of the second ship target detection box is 0, that is: S 1 > 0, S 2 = 0, then the ship target corresponding to the first ship target detection box K 1 is a disappeared ship target;

[0137] If the areas of both the first ship target detection box and the second ship target detection box are greater than 0, and the overlapping area between the first ship target detection box and the second ship target detection box is 0, that is: S1 > 0, S 2 > 0, S c = 0, then the ship target detection box K 1 corresponding ship target is a disappearing ship target, and the second ship target detection box K 2 corresponding ship target is a newly added ship target;

[0138] If the area of the first ship target detection box, the area of the second ship target detection box, and the overlapping area between the first ship target detection box and the second ship target detection box are all greater than 0, that is, S 1 > 0, S 2 > 0, S c > 0, and the intersection-over-union ratio between the first ship target detection box K 1 and the second ship target detection box K 2 is less than the set threshold, that is, α < δ, then the ship target corresponding to the first ship target detection box K 1 is a disappearing ship target, and the ship target corresponding to the second ship target detection box K 2 is a newly added ship target;

[0139] If the area of the first ship target detection box, the area of the second ship target detection box, and the overlapping area between the first ship target detection box and the second ship target detection box are all greater than 0, that is, S 1 > 0, S 2 > 0, S c > 0, and the intersection-over-union ratio between the first ship target detection box and the second ship target detection box is not less than the set threshold, that is, α ≥ δ, then the ship target corresponding to the first ship target detection box K 1 and the ship target corresponding to the second ship target detection box K 2 are both unchanged ship targets.

[0140] By designing the above calculation formula for the intersection-over-union ratio and giving the discrimination rules for newly added ship targets, disappearing ship targets, and unchanged ship targets, it can take into account special situations such as small displacements of ships and side-by-side parking, so as to more accurately obtain the ship change results. The output result is the vector target change information, including three categories of newly added, disappearing, and unchanged targets. It can not only detect the change information of a single ship, but also detect all the changed ships in the case of multiple ships side by side, with strong applicability and great potential value in practical applications.

[0141] This device aims at the remote sensing images of the port area. On the basis of establishing the region of interest for change, it uses the DDMNet model to detect ships in the images of the region of interest for change, effectively reducing the background interference. By calculating the intersection-over-union ratio of the ship detection boxes on the old and new phase images, it can effectively distinguish newly added ships from disappearing ships. The process is shown as Figure 6As shown in the figure, it includes multi-temporal image preprocessing, establishing regions of interest (ROIs) for change, extracting ROI images, object detection using the DDMNet model, calculating the intersection over union (IoU), and change analysis, which can automatically, quickly, and accurately achieve change detection of ship targets. If directly performing deep learning ship detection on the original two-temporal images and then comparing the object changes, it will cause the error of ship detection to be transmitted to the final object change detection. However, narrowing the research scope of ship target change detection to the ROIs for change effectively reduces the influence of the complex background of remote sensing images. By introducing a deep learning model for ship detection, it avoids the limitations brought by manually designed features in traditional methods, and thus can obtain more accurate object change detection results.

[0142] In addition, this device has strong scalability. By establishing ROIs for change covering more ports, it can achieve rapid monitoring of ship targets in multiple regions. The process supports full automation and can be effectively applied to ship target change detection tasks on a global scale.

[0143] It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or can be implemented as a combination of multiple physical units. In addition, to highlight the innovative part of the present invention, units not closely related to solving the technical problems proposed by the present invention are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0144] Example 3

[0145] An embodiment of the present invention also provides an electronic device, including:

[0146] At least one processor; and,

[0147] A memory communicatively connected to the at least one processor; wherein,

[0148] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the ship target change detection method of the above embodiment.

[0149] Among them, the memory and the processor are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna. Further, the antenna also receives data and transmits the data to the processor.

[0150] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when executing operations.

[0151] Example 4

[0152] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-mentioned ship target change detection method.

[0153] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned embodiment methods can be completed by instructing relevant hardware through a program. The program is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0154] Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made in form and details without departing from the spirit and scope of the present invention.

Claims

1. A ship target change detection method, characterized in that: include: Obtain two adjacent temporal images of the port area; Extracting the changed interest area image from the two adjacent time phase images; Performing ship target detection on the images of the changed interest area in the two adjacent time phase images to obtain the ship target detection frames of the two adjacent time phase images; Calculating the intersection-and-union ratio between the ship target detection frames of the two adjacent phase images; The change of the ship target is determined according to the ship target detection frames of the two adjacent time phase images and the intersection-and-union ratio.

2. The ship target change detection method according to claim 1, characterized in that: Before extracting the changed interest area image from the two adjacent time phase images, the method further includes: The two adjacent phase images are preprocessed, and the preprocessing includes geometric registration and radiation correction.

3. The ship target change detection method according to claim 1, characterized in that: The change interest area includes an area in the port area where a ship change may occur, and the change interest area is represented by vector information.

4. The ship target change detection method according to claim 1, characterized in that: Also includes: A changing interest area is established based on a preset condition; the preset condition includes at least one of the following: The changing interest area includes the docking position of the ship target of interest; The changing area of ​​interest is defined along the dock where the ship target is docked; The long side of the changing interest area is greater than the length of the ship target docked at the pier; The short side of the changing interest area is greater than an integral multiple of the width of the ship target docked at the pier.

5. The ship target change detection method according to claim 1, characterized in that: The pre-trained DDMNet model is used to perform ship target detection on the images of the changed interest areas in the two adjacent time phase images to obtain the ship target detection frames of the two adjacent time phase images.

6. The ship target change detection method according to claim 1, characterized in that: The intersection-and-union ratio between the ship target detection frames of the two adjacent phase images is calculated using the following calculation formula: α=S c / S d In the formula, α represents the intersection-over-combination ratio, S c represents the overlapping area between the ship target detection frames of the two adjacent phase images, S d It represents the total area after the ship target detection frames of the two adjacent phase images are superimposed.

7. The ship target change detection method according to claim 1, characterized in that: The ship target detection frames of the two adjacent time phase images include a first ship target detection frame of the previous time phase image and a second ship target detection frame of the current time phase image; Determining the change of the ship target according to the ship target detection frames of the two adjacent time phase images and the intersection-and-union ratio includes: If the area of ​​the first ship target detection frame is 0, and the area of ​​the second ship target detection frame is greater than 0, the ship target corresponding to the second ship target detection frame is a newly added ship target; If the area of ​​the first ship target detection frame is greater than 0 and the area of ​​the second ship target detection frame is 0, the ship target corresponding to the first ship target detection frame is the disappeared ship target; If the area of ​​the first ship target detection frame and the area of ​​the second ship target detection frame are both greater than 0, and the overlapping area of ​​the first ship target detection frame and the second ship target detection frame is 0, then the ship target corresponding to the first ship target detection frame is the disappeared ship target, and the ship target corresponding to the second ship target detection frame is the newly added ship target; If the area of ​​the first ship target detection frame, the area of ​​the second ship target detection frame, and the overlapping area of ​​the first ship target detection frame and the second ship target detection frame are all greater than 0, and the intersection and union ratio between the first ship target detection frame and the second ship target detection frame is less than the set threshold, then the ship target corresponding to the first ship target detection frame is the disappeared ship target, and the ship target corresponding to the second ship target detection frame is the newly added ship target; If the area of ​​the first ship target detection frame, the area of ​​the second ship target detection frame, and the overlapping area of ​​the first ship target detection frame and the second ship target detection frame are all greater than 0, and the intersection and union ratio between the first ship target detection frame and the second ship target detection frame is not less than the set threshold, then the ship target corresponding to the first ship target detection frame and the ship target corresponding to the second ship target detection frame are both unchanged ship targets.

8. A ship target change detection device, characterized in that: include: An image acquisition module is used to acquire two adjacent temporal phase images of the port area; An image extraction module, used for extracting the changed interest area image in the two adjacent time phase images; A target detection module is used to perform ship target detection on the images of the changed interest area in the two adjacent time phase images to obtain the ship target detection frame of the two adjacent time phase images; An intersection-and-union ratio calculation module is used to calculate the intersection-and-union ratio between the ship target detection frames of the two adjacent time phase images; The change determination module is used to determine the change of the ship target according to the ship target detection frame of the two adjacent time phase images and the intersection-and-union ratio.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the ship target change detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the ship target change detection method according to any one of claims 1 to 7 is implemented.

Images