Method and device for visual perception of a ship under adverse weather conditions

Abstract

This invention provides a method and apparatus for ship visual perception under adverse weather conditions. The method includes: constructing adversarial ship visual data under adverse weather conditions; fusing initial ship visual perception image data with the constructed adversarial data to construct comprehensive visual perception image data; using a ship visual perception model as the model basis for adversarial training to generate an adaptive ship visual perception model; and using the obtained adaptive ship visual perception model, inputting the navigation image to be perceived, and using a feature activation algorithm to output interpretable visual perception results. The ship visual perception method of this invention can maintain stable and interpretable ship detection and classification under various weather conditions, which is of great significance for realizing reliable autonomous navigation of intelligent ships.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent ship situation awareness, in particular to a ship visual perception method and device under adverse weather conditions. BACKGROUND

[0002] Nowadays, ship visual perception technology is mainly widely used in marine video monitoring. High-precision and real-time ship visual perception technology has become an important part of ship safety traffic service, and is also a key technology for the development of intelligent ship situation awareness.

[0003] The ship visual perception technology currently applied to intelligent ship autonomous navigation can accurately and timely detect and classify target ships, but in response to complex weather conditions, most ship visual perception methods will miss and misclassify target ships, which will pose a safety hazard to subsequent decision-making for autonomous navigation.

[0004] The most advanced existing ship visual perception technology is often based on deep learning algorithms, but as a "black box" model, the decision-making results of deep learning are not transparent, which makes the visual perception model based on deep learning not explainable in real sea scene applications. Therefore, how to still accurately and timely detect and classify ships in captured navigation images under adverse weather conditions, while providing an explanation of the perception results that the crew can understand, is a major technical problem that needs to be solved. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a ship visual perception method and device under adverse weather conditions, which can provide accurate and real-time explainable visual perception of moving ships under various weather conditions, and provide technical support for the instability and untrustworthiness of existing ship visual perception based on deep learning.

[0006] To solve the above problems, the technical scheme of the present application is as follows:

[0007] A ship visual perception method under adverse weather conditions, comprising the following steps:

[0008] Constructing ship visual confrontation data under adverse weather conditions;

[0009] Fusing the constructed ship visual confrontation data with initial ship visual perception image data to construct comprehensive visual perception image data, using a ship visual perception model as the model basis for adversarial training, and generating an adaptive ship visual perception model;

[0010] By obtaining the adaptive ship visual perception model, inputting the navigation image to be perceived, and using the feature activation algorithm, an interpretable visual perception result is output.

[0011] Preferably, the step of constructing ship visual confrontation data under adverse weather conditions specifically comprises: using an existing ship visual perception image dataset, adopting a ship visual perception model based on deep learning, using a target disappearance confrontation attack on images in the dataset according to low-visibility images appearing in fog and haze, inputting the ship visual perception model, and iteratively modifying the input image according to the confidence score transformation of the target detector of the perception model until the target ship is divided into the background area to generate low-visibility ship visual confrontation data.

[0012] Preferably, the step of constructing ship visual confrontation data under adverse weather conditions specifically further comprises: for low signal-to-noise ratio images appearing under rain and snow weather, causing ship misrecognition, using an expected transformation confrontation attack on specific regions of the images in the dataset, performing random projection, rotation, and scaling feature transformation, and the expected transformation confrontation attack calculation process is represented as: Where the equal sign right side T represents the feature transformation distribution; M represents the specific region of the attack; d represents the distance between the adversarial data x adv and the original data x, inputting the ship visual perception model, and using the gradient descent method to construct an adversarial patch in the specific region to generate strong misleading ship visual confrontation data.

[0013] Preferably, the step of constructing ship visual confrontation data under adverse weather conditions specifically further comprises: fusing the low-visibility ship visual confrontation data and the strong misleading ship visual confrontation data to construct the ship visual confrontation data under adverse weather conditions.

[0014] Preferably, the step of fusing the constructed ship visual confrontation data and the initial ship visual perception image data to construct comprehensive visual perception image data, using the ship visual perception model as the model basis for adversarial training, and generating an adaptive ship visual perception model specifically comprises: using an adversarial Logit pairing training method to perform adversarial training of the ship visual perception model, adding the cross entropy of the Logit layer of the initial ship visual perception data and the Logit layer of the ship visual confrontation data in the model loss function, improving the similarity of the model learning the ship visual confrontation data and the initial ship visual perception data, and the loss function of the adversarial training is: Loss AT = Loss ori (ω, x, y) + H(ω, x, x'), where Loss oriLet ω represent the original loss function of the ship visual perception model, x represent the network weights of the ship visual perception model, y represent the ground truth of the target ship label, x' represent the input ship visual adversarial data, and H represent the cross-entropy function.

[0015] Preferably, the step of generating an adaptive ship visual perception model by fusing initial ship visual perception image data with the constructed ship visual adversarial data, constructing comprehensive visual perception image data, and using a ship visual perception model as the model basis for adversarial training, further includes: using a defined loss function to fine-tune the model network parameters through gradient descent, iteratively optimizing until the target ship detection accuracy reaches above a threshold, thereby obtaining a highly robust adaptive ship visual perception model.

[0016] Preferably, the step of inputting the navigation image to be perceived through the obtained adaptive ship visual perception model, and outputting an interpretable visual perception result using a feature activation algorithm specifically includes: inputting the navigation image to be perceived through the obtained adaptive ship visual perception model, calculating the global features of the navigation perception result using a fractionally weighted feature activation algorithm, and providing a visual interpretation result of the image region with the largest feature contribution in the navigation image.

[0017] Preferably, the step of taking the obtained adaptive ship visual perception model, inputting the navigation image to be perceived, and using a feature activation algorithm to output interpretable visual perception results specifically includes:

[0018] Using the trained adaptive ship visual perception model, the input is the navigation image to be perceived, and the output is a multi-scale feature map.

[0019] Image upsampling and feature normalization are performed on the multi-scale feature maps to obtain the multi-scale mask and the normalized feature vector, respectively.

[0020] The generated multi-scale mask is input again into the ship visual perception model, and the multi-scale feature vector is output. The similarity between the normalized feature vector and the multi-scale feature vector is calculated using the feature similarity algorithm.

[0021] The similarity is used to generate the increase in channel direction similarity, which measures the importance of each feature to ship perception.

[0022] The system outputs interpretable visual perception results through a linear combination of multi-scale feature maps and similarity increments.

[0023] Furthermore, the present invention also provides a ship visual perception device under adverse weather conditions, the device including a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the ship visual perception method under adverse weather conditions as described above by executing the executable instructions.

[0024] Compared with the prior art, the present invention has the following advantages and effects:

[0025] 1. This invention can design low-visibility and highly interfering ship visual adversarial data for existing ship visual perception models, effectively testing the stability and robustness of ship visual perception models.

[0026] 2. The adversarial training method proposed in this invention effectively enhances the accuracy of ship visual perception under adverse weather conditions, further ensuring the safety and reliability of maritime navigation.

[0027] 3. This invention addresses the unknowability of ship visual perception results by utilizing a feature activation algorithm to achieve visual interpretation of the perception results, which helps improve the credibility of deep learning-based ship visual perception models for seafarers and captains. Attached Figure Description

[0028] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0029] Figure 1 A flowchart of a ship visual perception method under adverse weather conditions provided in an embodiment of the present invention;

[0030] Figure 2 A detailed flowchart of the ship visual perception method under adverse weather conditions provided in this embodiment of the invention;

[0031] Figure 3 This is a schematic diagram illustrating the specific implementation process of ship visual interpretable perception provided in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0033] Specifically, the present invention provides a method for visual perception of ships under adverse weather conditions, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:

[0034] S1: Constructing visual countermeasure data for ships under adverse weather conditions;

[0035] Specifically, to address the problem that adverse weather conditions such as fog, low lighting, rain, and snow can prevent ship visual perception models from accurately perceiving target ships, a hybrid perturbation adversarial attack algorithm is employed to construct ship visual adversarial data under adverse weather conditions. The hybrid perturbation adversarial attack algorithm includes a target disappearance adversarial attack algorithm and an expectation transformation adversarial attack algorithm.

[0036] The specific implementation process is as follows:

[0037] Using existing ship visual perception image datasets, and selecting the most advanced deep learning-based ship visual perception model, this study employs a target vanishing adversarial attack on images in the dataset based on low visibility images caused by weather conditions such as fog and low lighting. By inputting the images into the ship visual perception model, the system iteratively modifies the input images based on the confidence scores of the target detector in the perception model until the target ship is classified as a background region, thereby generating low visibility ship visual adversarial data.

[0038] To address the issue of ship misidentification caused by low signal-to-noise ratio images appearing under rain and snow weather conditions, an expectation transform adversarial attack is applied to specific regions of the images in the dataset. This involves performing feature transformations such as random projection, rotation, and scaling. The calculation process of the expectation transform adversarial attack can be represented as follows:

[0039]

[0040] Where T on the right side of the equals sign represents the feature transformation distribution; M represents the specific region of attack; and d represents the adversarial data x. adv The distance from the original data x. By inputting the ship visual perception model, adversarial patches are constructed in specific areas using the gradient descent method to generate highly misleading ship visual adversarial data.

[0041] Finally, low-visibility ship visual countermeasures data and highly misleading ship visual countermeasures data are fused together to construct ship visual countermeasures data under adverse weather conditions.

[0042] S2: For the constructed ship visual adversarial data, the initial ship visual perception image data is integrated to construct comprehensive visual perception image data. The ship visual perception model is used as the model basis for adversarial training to generate an adaptive ship visual perception model.

[0043] Specifically, an adversarial Logit pairing training method is adopted to conduct adversarial training of the ship visual perception model. By adding the cross-entropy of the Logit layer of the initial ship visual perception data and the Logit layer of the adversarial ship visual data to the model loss function, the similarity between the adversarial ship visual data and the initial ship visual perception data is improved. The loss function of this adversarial training is as follows:

[0044] Loss AT =Loss ori (ω,x,y)+H(ω,x,x′)#(2)

[0045] Among them, Loss ori Let ω represent the original loss function of the ship visual perception model, x represent the network weights of the ship visual perception model, y represent the ground truth of the target ship label, x' represent the input ship visual adversarial data, and H represent the cross-entropy function.

[0046] The model network parameters are tuned using the defined loss function and gradient descent method. The optimization is iterated until the target ship in the ship visual adversarial data reaches an accuracy threshold above 98%. Finally, a highly robust adaptive ship visual perception model is obtained.

[0047] S3: Using the obtained adaptive ship visual perception model, input the navigation image to be perceived, and use the feature activation algorithm to output interpretable visual perception results.

[0048] Specifically, by using the obtained adaptive ship visual perception model, the navigation image to be perceived is input, and a fractionally weighted feature activation algorithm is used to calculate the global features of the navigation perception result, and provide the visual interpretation results of the image region with the largest feature contribution in the navigation image.

[0049] Figure 3 This is a schematic diagram illustrating the specific implementation process of ship visual interpretable perception provided in an embodiment of the present invention, such as... Figure 3 As shown, the specific implementation process is as follows:

[0050] Using the adaptive ship visual perception model trained in step S2, input the navigation image to be perceived and output a multi-scale feature map.

[0051] Image upsampling and feature normalization are performed on the multi-scale feature maps to obtain the multi-scale mask and the normalized feature vector, respectively.

[0052] The generated multi-scale mask is input again into the ship visual perception model, and the multi-scale feature vector is output. The similarity between the normalized feature vector and the multi-scale feature vector is calculated using a feature similarity algorithm. The feature similarity algorithm is usually, but not limited to, the cosine similarity algorithm.

[0053] The similarity is used to generate the increase in channel direction similarity, which measures the importance of each feature to ship perception.

[0054] The system outputs interpretable visual perception results through a linear combination of multi-scale feature maps and similarity increments.

[0055] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for visual perception of a ship under adverse weather conditions, characterized by, The method includes the following steps: Constructing ship visual adversarial data under adverse weather conditions specifically includes: utilizing existing ship visual perception image datasets, employing a deep learning-based ship visual perception model, and applying a target disappearance adversarial attack to the images in the dataset based on low visibility images caused by fog, haze, and low lighting. By inputting the image into the ship visual perception model, the input image is iteratively modified according to the confidence score transformation of the target detector in the perception model until the target ship is classified as a background area, thereby generating low visibility ship visual adversarial data. To address the issue of ship misidentification caused by low signal-to-noise ratio images appearing under rain and snow conditions, an expectation transform adversarial attack is applied to specific regions of the images in the dataset. This involves random projection, rotation, and scaling feature transformations. The calculation process for the expectation transform adversarial attack is represented as follows: , where the right side of the equals sign Indicates the characteristic transformation distribution; Indicates the specific area being attacked; Representing adversarial data Compared with the original data By inputting the distance into the ship's visual perception model, the gradient descent method is used to construct adversarial patches in specific areas to generate highly misleading ship visual adversarial data. By fusing low-visibility ship visual countermeasures data with highly misleading ship visual countermeasures data, ship visual countermeasures data under adverse weather conditions can be constructed. For the constructed ship visual adversarial data, the initial ship visual perception image data is integrated to construct comprehensive visual perception image data. The ship visual perception model is used as the model basis for adversarial training to generate an adaptive ship visual perception model. By using the obtained adaptive ship visual perception model, the input navigation image to be perceived is used, and the feature activation algorithm is employed to output interpretable visual perception results.

2. The method for visual perception of a ship in adverse weather conditions according to claim 1, characterized in that, The step of fusing the initial ship visual perception image data with the constructed ship visual confrontation data to construct comprehensive visual perception image data, adopting the ship visual perception model as the model basis for the confrontation training, and generating the adaptive ship visual perception model specifically includes: adopting the confrontation Logit pairing training method to perform the confrontation training of the ship visual perception model, adding the cross entropy of the Logit layer of the initial ship visual perception data and the Logit layer of the ship visual confrontation data in the model loss function, improving the similarity of the model learning the ship visual confrontation data and the initial ship visual perception data, and the loss function of the confrontation training is: wherein, represents the original loss function of the ship visual perception model, represents the network weight of the ship visual perception model, represents the input initial ship visual perception data, represents the target ship label true value, represents the input ship visual confrontation data, and H represents the cross entropy function.

3. The method for visual perception of a ship in adverse weather conditions according to claim 2, characterized in that, The steps for generating an adaptive ship visual perception model by fusing initial ship visual perception image data with the constructed ship visual adversarial data and using a ship visual perception model as the model basis for adversarial training specifically include: using a defined loss function to fine-tune the model network parameters through gradient descent, iterating until the target ship detection accuracy reaches above a threshold, thereby obtaining a highly robust adaptive ship visual perception model.

4. The method for visual perception of a ship in adverse weather conditions according to claim 1, characterized in that, The steps of using the obtained adaptive ship visual perception model, inputting the navigation image to be perceived, and using a feature activation algorithm to output interpretable visual perception results specifically include: using the obtained adaptive ship visual perception model, inputting the navigation image to be perceived, using a fractionally weighted feature activation algorithm to calculate the global features of the navigation perception results, and providing visual interpretation results of the image region with the largest feature contribution in the navigation image.

5. The method for visual perception of a ship in adverse weather conditions according to claim 1, characterized in that, The steps of obtaining an adaptive ship visual perception model, inputting the navigation image to be perceived, using a feature activation algorithm, and outputting interpretable visual perception results specifically include: Using the trained adaptive ship visual perception model, the input is the navigation image to be perceived, and the output is a multi-scale feature map. Image upsampling and feature normalization are performed on the multi-scale feature maps to obtain the multi-scale mask and the normalized feature vector, respectively. The generated multi-scale mask is input again into the ship visual perception model, and the multi-scale feature vector is output. The similarity between the normalized feature vector and the multi-scale feature vector is calculated using the feature similarity algorithm. The similarity is used to generate the increase in channel direction similarity, which measures the importance of each feature to ship perception. The system outputs interpretable visual perception results through a linear combination of multi-scale feature maps and similarity increments.

6. A visual perception device for a vessel in adverse weather conditions, characterized in that The apparatus comprises a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the method of visual perception of a ship in adverse weather conditions as claimed in any one of claims 1 to 5 by executing the executable instructions.

Images