Parallel incremental deep learning method for ocean remote sensing image analysis

Abstract

The present invention discloses a parallel incremental deep learning method for marine remote sensing image analysis, and relates to the field of marine remote sensing technology. Based on focusing on the similarity of images, this method proposes an easily transplantable training and reasoning acceleration framework based on incremental cropping, which can achieve the effect of accelerating network calculations by sharing the same area calculations. In addition, since the shared benefits generated by the shared common operations of the existing incremental acceleration technology are obviously very sensitive to the similarity of the input sequence, and the similarity depends to a large extent on the input order, this scheme explores a new sorting method that perceives semantic differences, applies it to periodic video sequences and image sequences, and conducts research on various performance optimization technologies based on this.

Description

Technical Field

[0001] The present invention relates to the field of ocean remote sensing technology, and in particular to a parallel incremental deep learning method for ocean remote sensing image analysis. Background Art

[0002] The total area of ​​the world's oceans is about 360 million square kilometers, accounting for about 71% of the earth's surface area. The oceans provide rich resources and information for human activities. A slight fluctuation in ocean temperature can cause dramatic changes in weather and climate around the world; the movement of ocean currents affects the transport of water vapor in the atmosphere, thereby affecting the distribution of rainfall. Special ocean detection technology can capture this information in a timely manner.

[0003] Marine remote sensing technology is an effective means of detecting the ocean, including information carriers such as light and electricity and acoustic wave remote sensing technology. Marine remote sensing images often contain a lot of useful information, but also relatively redundant data. This redundancy may come from the slow changes in the marine environment, the similarity of images under specific weather conditions, or the overlap of information due to the working principle of the sensor. Extracting effective information from these images quickly and efficiently has become an increasingly important topic. Today, the most efficient way is to use deep neural networks to process this data.

[0004] Deep learning (DL) is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Deep learning is based on the intrinsic laws and representation levels of deep neural networks to learn sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Among them, the development of convolutional neural networks is one of the most mature technologies at present, and has been successfully applied to various image and video tasks, such as image classification and target detection. However, with the rapid growth of data volume and model parameters, traditional training and reasoning processes have encountered computing performance bottlenecks and cannot meet the growing computing needs. Especially when processing redundant information such as video data, the high dimension of video data and the larger data volume make this computing performance bottleneck more obvious. Traditional convolutional neural networks treat different frames in the video as separate images and process them separately, ignoring the inherent similarities between frames, resulting in a large amount of computational redundancy, resulting in low efficiency and long time consumption of convolutional neural networks in video tasks. As a result, a series of lightweight incremental training and reasoning frameworks have been derived, the most representative of which is the Skip Convolutions proposed by Habibian et al. in 2021. Instead of viewing the video as a static sequence of images, the Skip Convolution represents it as a series of residual frames defined by the input frame and intermediate features, and then uses the large amount of redundancy in the video stream to save computational operations. In order to efficiently calculate on the residual frame, the Skip Convolution reformulates the standard convolution, and each layer is coupled with a binary gate to determine whether a residual value has an important influence on the model prediction. The residual values ​​of the foreground area need to participate in the convolution calculation, and the residual values ​​of the background area can be safely skipped. These gates can be implemented as an effective network jointly trained with the convolution kernel, or simply skip the residual according to the size of the residual. Depending on the need for effective implementation on the hardware platform, the gate function can also incorporate block sparse structures. Skip convolution establishes a new state-of-the-art technology by effectively utilizing the temporal redundancy in the video. However, the Skip Convolution still uses the original incremental design, that is, sparse increments are used as convolution input, which makes it less portable. Summary of the invention

[0005] The purpose of the present invention is to address the above-mentioned shortcomings and propose an easily transplantable training acceleration framework based on incremental cropping, which can accelerate network training by sharing the same area calculations of similar frames. It explores a new sorting method for perceptual semantic differences and applies it to periodic video sequences and image sequences. Based on this, a parallel incremental deep learning training and reasoning method for marine remote sensing image analysis is developed with a variety of performance optimization technologies.

[0006] The present invention specifically adopts the following technical solutions:

[0007] The parallel incremental deep learning training and reasoning method for ocean remote sensing image analysis includes the following steps:

[0008] 1.1, an easily portable incremental training framework based on data reorganization; a framework that uses the similarity between images to reduce redundant calculations in similar areas, and is directly implemented based on the native PyTorch framework. The framework uses data separation and reorganization, and uses the cropped training samples or features as the input of the convolution operation;

[0009] 1.2. Incremental reasoning acceleration technology based on data semantic reordering. First, a new sorting mechanism that perceives semantic differences is explored, and an occlusion technology is designed to exclude the changed areas of the image and focus on the unchanged parts. Two optimization methods, reducing the search space and reusing intermediate results, are proposed to accelerate the sorting mechanism.

[0010] Preferably, the easily portable incremental training framework based on data reorganization includes the following steps:

[0011] 2.1 High-quality feature pruning based on loop-connected branch analysis

[0012] The threshold filtering method is used to calculate the average value of the difference between each sample and the benchmark. Only the area exceeding the threshold is classified as the candidate incremental area. Then, the connected branch processing loop is entered. Each time the loop is entered, each connected branch is calculated and marked first, that is, the pixels with overlapping areas are marked with the same number. In the calculation of connected branches, the 4-neighborhood pixel adjacent definition method is used, that is, if there are elements above, below, left, and right of the current pixel, they are considered connected. The operator for connected branch calculation is a matrix with a cross area number of 1, that is, when calculating the connected branch of a pixel, if there are pixels located in the four directions of its top, bottom, left, and right, the two pixels can be judged as adjacent. If it is the first time to enter the loop, it is also necessary to filter the incremental area of ​​the small area.

[0013] 2.2, Incremental training process of feature cropping and result concatenation

[0014] Describe the incremental training framework from two aspects

[0015] (1) Determine the incremental position: Taking the first input sample of each batch as the benchmark, calculate the change area of ​​the current batch that meets the threshold condition, cyclically calculate the connected branches and integrate the change area into a reasonable incremental area, and record the relative incremental position;

[0016] (2) Incremental training process: For the convolution results of the incremental area, each input sample or feature map is calculated separately; for the convolution results of the non-incremental area, each input sample or feature map directly copies the calculation result of the first input sample or feature map.

[0017] Preferably, the incremental reasoning acceleration technology based on data semantic reordering includes the following steps:

[0018] 3.1 Incremental Reasoning Mechanism Enhanced by Semantic Reordering

[0019] (1) Periodic semantic analysis and recognition

[0020] A new semantic order is used as the input of CNN incremental reasoning, and a lighter global sorting mechanism is adopted to reduce redundant calculations in the CNN incremental reasoning process;

[0021] (2) Semantic similarity evaluation based on occlusion;

[0022] 3.2, retrieval space pruning optimization based on cycle jumping and multiple occlusion calculation redundancy based on result sharing.

[0023] The present invention has the following beneficial effects:

[0024] This scheme proposes a new semantic difference-aware sorting mechanism and designs an occlusion technique to improve sorting efficiency and effect by excluding image change areas and focusing on unchanged parts. This mechanism is compatible with all existing incremental reasoning work.

[0025] The improved convolutional neural network can extract features more effectively and significantly reduce the amount of calculation, thereby effectively accelerating the speed of training and reasoning. This comprehensive solution not only achieves significant advantages in performance, but also makes the data in the training and reasoning process more refined and more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the connected branch calculation operator;

[0027] Figure 2 This is a schematic diagram of the connected branch marking method;

[0028] Figure 3 It is a diagram of the incremental training process;

[0029] Figure 4Ocean remote sensing images collected by satellite periodic motion;

[0030] Figure 5 To reconstruct the specific location diagram of similar images in the semantic order;

[0031] Figure 6 Schematic diagram of similarity evaluation based on occlusion frame;

[0032] Figure 7 Schematic diagram of similarity calculation for redundancy / reuse between different occlusion boxes. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0034] The parallel incremental deep learning training and reasoning method for ocean remote sensing image analysis includes the following steps:

[0035] 1.1, an easily portable incremental training framework based on data reorganization; a framework that uses the similarity between images to reduce redundant calculations in similar areas, and is directly implemented based on the PyTorch native framework. The framework uses data separation and reorganization, and uses the cropped training samples or features as the input of the convolution operation, avoiding the problem of repeated convolution of redundant features, and has high availability and easy portability.

[0036] An easily portable incremental training framework based on data reorganization

[0037] Existing incremental training works perform convolution operations on the complete incremental images and control whether the data should skip calculations through a gating function. The complexity of this implementation and the non-disclosure of the code limit its usability. This paper focuses on the portability of the code and proposes an incremental training framework based on data reorganization.

[0038] High-quality feature pruning based on loop-connected branch analysis

[0039] High-quality cropping of relative incremental positions is an important part of the incremental training framework. It is the basis and prerequisite of the incremental training process. Obtaining relatively accurate incremental regions is crucial to the efficiency and effect of incremental training. In a given batch of training input samples, the first sample is generally used as the benchmark for calculating the incremental region, and then the differences between the remaining samples in the batch and the benchmark are compared. In theory, all parts with differences can be regarded as incremental regions in the current batch, but some inconspicuous differences are often not caused by the moving subject of the video, but by other factors such as subtle changes in light. In response to the above problems, this scheme adopts a threshold filtering method to calculate the average value of the difference between each sample and the benchmark, and only the area exceeding the threshold is designated as a candidate incremental region. After that, it enters the loop of connected branch processing. Each time it enters the loop, it first calculates and marks each connected branch, that is, marks pixels with overlapping areas with the same number. In the calculation of connected branches, the 4-neighborhood pixel adjacent definition method is used, that is, if there are elements above, below, left, and right of the current pixel, they are considered connected. For example Figure 1 As shown in the figure, the operator for connected branch calculation is a matrix with a cross area number of 1, which means that when calculating the connected branch of a pixel, if there are pixels in the four directions of the top, bottom, left, and right, the two pixels can be judged as adjacent. If it is the first time entering the loop, it is also necessary to filter the incremental area of ​​the small area, because it has little impact on the final output result.

[0040] As shown in Table 1, the input is a batch of input samples. The first row first calculates the overall incremental map in the batch; the second to fourth rows filter out pixels smaller than the threshold based on the incremental map; the fifth row starts to enter the connected branch loop; the label function in the seventh row calculates the connected branches and marks the connected pixels with the same number, such as Figure 2 As shown, there are two connected components, represented by red lines. The pixels in one connected area are marked as 1, and the other is marked as 2. The number of digital categories in the figure is the number of connected components. After the connected branch calculation is completed, the eighth line of Algorithm 4-1 uses a rectangular frame to turn it into a regular area that can be used as the input for convolution calculation. The ninth and tenth lines determine whether it is the first time to enter the loop. If so, only the larger increment frame is taken. Here, a specific number can be taken, or a certain proportion of the total number of increment frames can be taken. The eleventh line sets the pixels of the current increment frame to 1, that is, the pixels in the frame are considered as new increments. The twelfth and thirteenth lines are the judgment of the end condition of the loop.

[0041] Table 1

[0042]

[0043] 2.2, Incremental training process of feature cropping and result concatenation

[0044] From the working principle of jump convolution mentioned in the background technology section, it can be known that when the input of the convolution layer is incremental, in order to ensure the effectiveness of the calculation, the standard convolution needs to be reformulated. And the convolution layer needs to be coupled with a binary gate to determine whether an incremental data needs to participate in or skip the convolution calculation. The incremental training method of jump convolution is a more direct idea, with sparse increments as the convolution body. However, this training method is not easy to migrate to different convolutional network implementations and lacks open source code. Therefore, the research on incremental training that is easy to transplant is an important topic for the widespread application of incremental training ideas in processing similar image sequence tasks. The scheme considers using the inherent similarity between video input samples, and analyzing the change area of ​​the samples in the batch relative to the first sample in the batch through the incremental data between the input samples in the batch, that is, the area where different input samples produce different results after the convolution operation. The convolution calculation results need to be updated in the changed area, and the convolution calculation results do not need to be updated in the non-changed area. Based on the above research, the incremental training framework can be described from two aspects.

[0045] (1) Determine the incremental position: Taking the first input sample of each batch as the benchmark, calculate the change area of ​​the current batch that meets the threshold condition, cyclically calculate the connected branches and integrate the change area into a reasonable incremental area, and record the relative incremental position;

[0046] (2) Incremental training process: For the convolution results of the incremental area, each input sample or feature map is calculated separately; for the convolution results of the non-incremental area, each input sample or feature map directly copies the calculation result of the first input sample or feature map.

[0047] The incremental training process can be divided into two parts: feature cropping and result splicing. Feature cropping is to convert the obtained relative incremental position into the absolute incremental position of the corresponding input sample or feature map according to the size of the input sample or feature map, and then crop it on the input sample or feature map as the input of the convolution operation. It is worth noting that for the first input sample or feature map in the batch, the complete data is used as the input of the convolution layer without cropping, and the complete convolution layer output is obtained after the convolution operation. After other input samples or feature maps obtain the incremental convolution output, the output of each incremental area is replaced with the corresponding position of the first convolution layer output to obtain the complete output of each input sample or feature map. Figure 3 As shown, there are two relative incremental positions, the starting point is , width and height are The rectangle and starting point are , width and height are A rectangle, where , , , The parameters corresponding to the second relative increment position are all decimals between 0 and 1. , Gao Wei ), calculated in the input sample or feature map to The absolute position on is the number of images in the current batch, the absolute position corresponding to the first relative position is the starting point Width and height are Next, to Cut out the absolute increment area , ...... , ,in Indicates the incremental region cropped from the first relative incremental position corresponding to the second input sample or feature in a batch. , ...... , They are used as inputs for convolution calculations to obtain corresponding outputs. , ...... , It can be seen that the above convolution process is different from the skip convolution. Instead of taking the increment as the input of the convolution layer, the input samples or features cropped according to the absolute increment area are directly used as the input of the convolution calculation. This avoids the redefinition of the standard convolution and the use of the gating function, and does not require changes to the underlying deep learning code framework, so it is easier to port to different implementations of different convolutional neural networks.

[0048] Another important process of incremental training is result concatenation, such as Figure 3 As shown, the first sample or feature within the batch There is no need to crop the incremental area, which can be directly used as the input of the convolution operation to obtain the complete convolution output. The convolution results of other samples or features do not need to be calculated and can be directly replaced The complete output of the remaining samples or features can be obtained ~ It can be seen that the convolution process of the above incremental training method is very close to the native method. The convolution content is part of the input image or feature map, and the final complete output can be obtained by simply splicing the convolution results. It is a training process that is easy to implement in different convolutional neural networks.

[0049] 1.2, Incremental reasoning acceleration technology based on data semantic reordering; First, a new sorting mechanism that perceives semantic differences is explored, and an occlusion technology is designed to exclude the image change area and focus on the unchanged part. Two optimization methods, namely reducing the search space and reusing intermediate results, are proposed to accelerate the sorting mechanism, including the following steps:

[0050] Incremental reasoning mechanism enhanced by semantic reordering

[0051] (1) Periodic semantic analysis and recognition

[0052] A new semantic order is used as the input of CNN incremental reasoning, and a lighter global sorting mechanism is adopted to reduce redundant calculations in the CNN incremental reasoning process.

[0053] Currently, existing incremental reasoning designs all use the natural input order based on time monitoring to capture the similarity between two consecutive images. This is usually reasonable when the data is extracted from fixed positions, and some images with very high content similarity may be distributed in non-continuous time instances. Therefore, directly using the natural order of images as input actually ignores the content of the image and cannot group high-similarity images into adjacent inputs, which ultimately leads to very low actual similarity scores of the inputs, severely limiting the benefits of incremental reasoning designs.

[0054] This scheme adopts a new semantic order as the input of CNN incremental reasoning, and no longer uses the natural order based on monitoring time. The new order places images with high similarity in adjacent positions, which makes the similarity between consecutive images higher than the natural order. Using this order as the input of the network can improve the gain of incremental design and speed up the reasoning process. If we follow the most intuitive sorting idea, we can design a sorting mechanism to calculate the similarity of all pairs of images and reorganize these images into an optimal order. However, the computational complexity of this method is very large, and the time complexity is ,in is the number of images, and this time overhead is very expensive. Therefore, a more lightweight global sorting mechanism is proposed to reduce the redundant calculation of the CNN incremental reasoning process.

[0055] The proposed sorting mechanism is particularly suitable for situations where the network input is periodic, such as ocean remote sensing images collected by periodically rotating satellites. Figure 4 As shown, the curves represent the time trajectories of a continuously rotating satellite at a fixed angle, while the points represent the images obtained under these trajectories. According to the regularity of periodic similar images caused by camera rotation, the sorting method is divided into two stages. The first stage searches for a given image by traversing all input images. Similar images of to ensure Compare Adjacent images in their original natural order and More similar, that is ,in express and The similarity between them, and then similar images Insert to image In addition, for image sequences such as remote sensing image datasets, there is no inherent similarity between adjacent images. We give a fixed threshold based on context information to search for similar images. In the second stage, similar images of subsequent images are inferred based on semantic similarity cycles. For example, image The similar images searched in the first stage are and , then the image Similar images of and The position of is inferred, that is, , , and , this periodic prior knowledge can be learned by sampling in the original natural image sequence. However, these candidate images are only predicted and need to be further verified through detailed calculation and comparison. For example, if and The similarity score between them satisfies: and ,So yes Similar images. It is worth noting that the above mechanism is also applicable to regular video stream data generated by fixed acquisition viewpoints. The sampling-based prior knowledge learning process can detect whether the original sequence can provide the maximum similarity score. If not, it can find a suitable semantic cycle to guide the following similarity search.

[0056] Given an image, in order to identify the specific locations of its candidate similar images, we compare the similarity scores of the candidate images with the adjacent images in the original natural order. Figure 5 As shown, the image within the known candidate window , and and To determine whether they are and (or and ), we need to calculate the similarity score and make multiple comparisons. For example, and After comparison, if Closer to , it should be placed and Between. From Figure 5 It can be seen that and lie in and between, lie in and In addition, we need to calculate and If the former is greater than the latter, it means More similar to , so it should be placed Then slide the candidate window to the next one containing , and candidate window.

[0057] (2) Semantic similarity evaluation based on occlusion;

[0058] Similarity evaluation is a key issue to ensure the quality of output sequence, which is related to the performance gain of incremental reasoning. There are many methods to calculate image similarity, such as MSE (mean square error), SSIM (structural similarity), histogram distance and deep learning. Among them, the method of measuring the similarity of corresponding pixels between two images by MSE has high computational efficiency and can perceive operations such as scaling, rotation, and cropping. As shown in formula 4-2-1, It represents the square mean of the difference between the corresponding pixels of the two images, which can reflect the similarity between the two images at a fine-grained level. and represents a pair of images, represents the number of pixels in the image, and Respectively and No. pixels. The smaller it is, the higher the similarity between the two images is, and it can be used to measure the similarity between images.

[0059] Formula (4-2-1)

[0060] However, in the incremental inference scenario, directly using MSE may produce false positive similarity evaluation results. Both MSE and incremental inference are concerned with the fine-grained pixel-level similarity evaluation of a given pair of images. However, the former actually focuses on the average numerical difference based on formula (4-2-1), while the latter is very sensitive to how many pixels are exactly the same, so the intermediate results previously calculated from these same pixels can be reused to prune redundant calculations. When many pixels in two images are slightly different, we can obtain a lower MSE score, but the amount of computation that can be saved in inference is negligible. Figure 6 Figure Figure 6 shows a specific example. Although there are only 12 different pixels between image (I−1) and I, due to the significant differences between the pixels (the pixel values vary between 0.0 and 0.5 in regions R1 and R2), the resulting MSE score is large, up to 7.5 points. In contrast, all pixels of image I and (I + 1) are different, indicating that there is no redundant calculation that can be pruned during the incremental inference process, that is, in our semantic scenario, their similarity is zero. However, due to the small differences between the pixels (0.0 vs. 2.5, 5.0 vs. 2.5), MSE the score M(I−1, I) may be greater than M(I, I + 1), and then we wrongly infer that Sim(I−1, I) < Sim(I, I+1). In addition, a video contains thousands of images, each with hundreds of thousands or even more pixel points, and plus an image will participate in the similarity calculation multiple times. Although using MSE the method of measuring picture similarity is lightweight compared with other methods, the running time cost of evaluation is still very high.

[0061] To alleviate the above problems, an occlusion-based image similarity extraction method is proposed. As Figure 3 shown at the bottom, some pixel points are covered by gray boxes, and only the uncovered pixels participate in the MSE calculation to reduce the probability of false positives in the comparison results and at the same time reduce the running cost. The similarity calculation between images is shown in formula (4-2-2), where represents the entire image, and represents the region excluded from the box. Similar to shown in equation 4-2-1, the variant metric is inversely proportional to the similarity. Assume an image with a resolution of 704*480 is set with an occlusion box of size 584*360. Compared with the method of using all pixels to calculate MSE , this occlusion method can reduce 62% of the floating-point operations and at the same time reduce the probability of false positive comparisons.

[0062] Formula (4-2-2)

[0063] It is worth noting that in the verification stage, we further eliminated the interference of the foreground area on the similarity calculation. When the conditions are not met, similarity extraction can be calculated based on multiple occlusion boxes of the same size but different positions. Only when the similarity scores calculated based on all occluded frames do not meet the conditions of similar images, it is determined that the assumed image is not a similar image. Figure 7 As shown in the figure, three occlusion frames are set, frame 1 is in the middle, frame 2 is in the lower left corner, and frame 3 is in the lower right corner. First, frame 1 is used to verify the similarity. If the verification meets the condition of similar images, frame 2 and frame 3 are no longer used. When frame 1 fails to cover all foreground areas and the verification result does not meet the condition, frame 2 is used to test the similarity. If the judgment condition of similar images is still not met at this time, the calculation based on frame 3 is continued. , the judgment result at this time is taken as the final result, and no other calculations are performed.

[0064] Retrieval space pruning optimization based on cycle jumping and multiple occlusion calculation redundancy elimination based on result sharing.

[0065] According to the knowledge of periodic similar images, there must be no similar images to a given image in a certain image range, and traversing all images to form a candidate set will lead to many useless calculations and comparisons. Therefore, an image skipping technique is proposed to optimize the time complexity.

[0066] Initially, the image skipping technique still finds the first similar image to a given image by traversing all images, but then, it uses the number of traversed images as a rotation cycle. When searching for other similar images, it jumps directly to the position of the next cycle within a certain range. This greatly reduces the time complexity from Down to ,in is the total number of images, is the number of images in a rotation cycle (cycle size). Figure 4 As shown, in the first stage of sorting, Locations found Similar images of .use As prior knowledge, for the next similar image , will be Search is performed within, and so on. is the number of allowed error images, which means we need to carefully compare each image in the candidate set.

[0067] In the image sequence, the period size may vary, which makes The value will fluctuate, use constant and The setting will obviously have a negative impact on finding similar images. We dynamically adjust and To adapt to this change, we first set the value of The initial value is determined by the camera's rotation speed. and the rate at which images are captured We use To initialize ,in As the weight representing the error allowed by the rotation period. Here, according to our experiment, , the user can also manually specify other values. and The information of is not available, we can initialize it with a random number After initialization, we can find the appropriate value through the binary adjustment strategy. The adjustment is based on The positions of similar images in yes If the similar image is strictly in the middle of the range, then ,in is the adjustment step size, usually set to 2 (users can also specify other values). If no similar images are found within this range, then .For example, The initial value of is 10. If In the If a location is found, Adjust to 5. If in range If no similar images are found in Expanded to 20. After expansion, if Similar images are found at positions, which means that the period has changed. In this case, we To adjust Value, where is relative to , so that similar images are in the middle of the range. Adjust After that, The value is reset to the initial value.

[0068] Multiple occlusion calculation redundancy elimination based on result sharing

[0069] The calculation of is based on multiple occlusion boxes. Since the areas excluded by different boxes may overlap, there is redundant calculation. Figure 6 As shown, when the calculation is based on Box 1 When the area arrive The results involved. For the , and The pixels in are calculated repeatedly, resulting in redundant calculations. , more redundant calculations are performed, namely , and Calculations included.

[0070] To address this problem, a method of reusing calculated values ​​is proposed. In the process of calculating If the same operation is required when the same operation is performed, the partial result related to the corresponding pixel is directly reused. Otherwise, the calculation result is recorded for possible reuse in the future. The reuse sharing benefit is obviously sensitive to the order in which different occlusion boxes are used. Usually, the middle box has a high probability of covering the foreground area and can get the correct comparison relatively quickly. Therefore, it is used as the first occlusion box. According to our tests, for the other boxes, the effect is good when they are set in four directions: lower left corner, lower right corner, upper left corner, and upper right corner. In addition, the distance between the next box in any direction and the first center box is very important, which will affect the coverage and sharing benefit of the foreground area. Next, we use Figure 1 Let's take an example to illustrate how to identify a reasonable distance. Assume is the maximum distance allowed, i.e. the next box is exactly at a corner (e.g. Figure 6 Also, assuming that the image Not satisfied with The decision condition based on the center box of similar images is , then use formula (4-3) to calculate the distance to the next box The main idea behind this strategy is that if the currently discovered image and Not very similar, then the next box should be far away from the first box to increase the probability of finding a similar image. Once the distance is determined, the next key question is how to organize the order of use of different occluded boxes in order to maximize the shared benefit. The solution is a greedy strategy: if the shared benefits of the two boxes are the same, then randomly select the next box, otherwise select the box with the largest overlapping area with the currently used box. In this way, the redundant calculations involved in the next box can be reduced.

[0071] Formula (4-3).

[0072] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A parallel incremental deep learning method for ocean remote sensing image analysis, characterized in that: The following steps are involved: 1.1, an easily portable incremental convolution framework based on data reorganization; a framework that uses the similarity between images to reduce redundant calculations in similar areas, and is directly implemented based on the native PyTorch framework. The framework uses data separation and reorganization, and uses the cropped training samples or features as the input of the convolution operation; 1.

2. Incremental reasoning acceleration technology based on data semantic reordering. First, we explored a new sorting mechanism that perceives semantic differences, designed an occlusion technique to exclude image change areas and focus on unchanged parts, and proposed two optimization methods to accelerate the sorting mechanism: reducing the search space and reusing intermediate results. The incremental reasoning acceleration technology based on data semantic reordering includes the following steps: 3.1 Incremental Reasoning Mechanism Enhanced by Semantic Reordering (1) Periodic semantic analysis and recognition A new semantic order is used as the input of CNN incremental reasoning, and a lighter global sorting mechanism is adopted to reduce redundant calculations in the CNN incremental reasoning process; According to the regularity of periodic similar images caused by camera rotation, the sorting method is divided into two stages: the first stage searches for a given image by traversing all input images. Similar images of to ensure Compare Adjacent images in their original natural order and More similar, that is ,in express and The similarity between them, and then similar images Insert to image In the second stage, similar images of subsequent images are inferred based on the semantic similarity cycle. The similar images searched in the first stage are and , then the image Similar images of and The position of is inferred, that is, , , and , these candidate images need to be further verified through detailed calculation and comparison. and The similarity score between them satisfies: and ,So yes similar images; given an image, in order to identify the specific location of its candidate similar images, the similarity scores of the candidate images are compared with the adjacent images in the original natural order. , and and Similarly, in order to determine whether they are and ,or and The specific position between the two images requires calculating the similarity score and performing multiple comparisons. For example, and After comparison, if Closer to , it should be placed and between; and lie in and By calculating and If the former is greater than the latter, it means More similar to , so it should be placed , and then slide the candidate window to the next one containing , and Candidate window of (2) Semantic similarity evaluation based on occlusion; The similarity calculation between images is shown in formula (4-2-2), where represents the entire image, Indicates the area that is excluded from the box: Formula (4-2-2) 3.2, retrieval space pruning optimization based on cycle jumping and multiple occlusion calculation redundancy elimination based on result sharing; Image skipping technology is used to optimize time complexity, including: using the number of traversed images as a rotation cycle; when searching for other similar images, it will directly jump to the position of the next cycle within a certain range, and the time complexity is reduced from Down to ,in is the total number of images, is the number of images in a rotation cycle; in the first stage of sorting, Locations found Similar images of ,use As prior knowledge, for the next similar image , will be Search is performed within, and so on, where is the number of error images allowed; dynamically adjusted in the image sequence and The values ​​of are first set according to the displacement between consecutive images in the original natural order The initial value is determined by the camera's rotation speed. and the rate at which images are captured Decide, use To initialize ,in As the weight of the error allowed for the rotation period; if and The information is not available, initialize with a random number ;After initialization, the appropriate value is found through the binary adjustment strategy. The adjustment is based on The positions of similar images in yes If the similar image is strictly in the middle of the range, then ,in is the adjustment step size. If no similar image is found within this range, then , After expansion, if Similar images are found at positions, which means that the cycle has changed. To adjust Value, where is relative to The offset of , so that similar images are in the middle of the range; adjust After that, The value is reset to the initial value; Multiple occlusion calculation redundancy based on result sharing, including: The calculation is based on multiple occlusion boxes. Since the areas excluded by different boxes may overlap, the method of reusing the calculated values ​​is adopted in the calculation. In the process of calculating If the same operation is required, the partial result related to the corresponding pixel is directly reused, otherwise, the calculation result is recorded for possible reuse in the future; the middle box is used as the first occlusion box, and the other boxes are set at the lower left corner, lower right corner, upper left corner and upper right corner. The distance recognition between the next box in any direction and the first center box includes: Assume is the maximum distance allowed, that is, the next box is exactly at a corner. At the same time, assuming that the image Not satisfied with The decision condition based on the center box of similar images is , then use formula (4-3) to calculate the distance to the next box ; After determining the distance, organize the order of using different occlusion boxes. If the shared benefits of two boxes are the same, then randomly select the next box, otherwise select the box with the largest overlapping area with the currently used box. In this way, the redundant calculations involved in the next box can be reduced; Formula (4-3).

2. The parallel incremental deep learning method for ocean remote sensing image analysis according to claim 1, characterized in that: An easily portable incremental training framework based on data reorganization The following steps are involved: 2.1 High-quality feature pruning based on loop-connected branch analysis The threshold filtering method is used to calculate the average value of the difference between each sample and the benchmark. Only the area exceeding the threshold is classified as the candidate incremental area. Then, the connected branch processing loop is entered. Each time the loop is entered, each connected branch is calculated and marked first, that is, the pixels with overlapping areas are marked with the same number. In the calculation of connected branches, the 4-neighborhood pixel adjacent definition method is used, that is, if there are elements above, below, left, and right of the current pixel, they are considered connected. The operator for connected branch calculation is a matrix with a cross area number of 1, that is, when calculating the connected branch of a pixel, if there are pixels located in the four directions of its top, bottom, left, and right, the two pixels can be judged as adjacent. If it is the first time to enter the loop, it is also necessary to filter the incremental area of ​​the small area. 2.2, Incremental training process of feature cropping and result concatenation Describe the incremental training framework from two aspects (1) Determine the incremental position: Taking the first input sample of each batch as the benchmark, calculate the change area of ​​the current batch that meets the threshold condition, cyclically calculate the connected branches and integrate the change area into a reasonable incremental area, and record the relative incremental position; (2) Incremental training process: For the convolution results of the incremental area, each input sample or feature map is calculated separately; for the convolution results of the non-incremental area, each input sample or feature map directly copies the calculation result of the first input sample or feature map.

Images