An evaluation method for intelligent ship collision avoidance algorithms

Abstract

This invention provides an evaluation method for intelligent ship collision avoidance algorithms, comprising the following steps: constructing a high-dimensional parameter space related to the collision avoidance task based on the expected operating conditions, traffic constraints, rule constraints, and task requirements of the intelligent ship collision avoidance algorithm under test; generating candidate test scenarios through spatial sampling methods, and combining these with a screening method to constrain the effectiveness of the candidate test scenarios, thereby obtaining a set of effective test scenarios; conducting batch testing on the set of effective test scenarios to obtain test results under different scenario parameter combinations, different traffic interaction conditions, and different operating states; fitting the performance of the intelligent ship collision avoidance algorithm under the expected operating conditions based on the test results, and using this fitting to evaluate the intelligent ship collision avoidance algorithm. This method improves the coverage of test samples and the interpretability of evaluation results in complex encounter spaces, and enhances the ability to identify high-risk low-value areas, locally vulnerable areas, and boundary-sensitive areas.

Description

Technical Field

[0001] This invention belongs to the field of testing and evaluation technology for intelligent navigation and collision avoidance decision-making of ships, and relates to an evaluation method for intelligent ship collision avoidance algorithms. Background Technology

[0002] With the development of intelligent shipping and autonomous surface vessel technology, autonomous collision avoidance algorithms have become a core component of intelligent ship autonomous navigation systems. Compared to traditional manned vessels, autonomous collision avoidance algorithms can reduce human decision-making errors to a certain extent, improve navigation safety and operational efficiency, and therefore have received extensive research and rapid development in recent years. To ensure these algorithms have a foundation for engineering applications and certification testing, in addition to studying the collision avoidance capabilities of the algorithms themselves, it is also necessary to establish corresponding testing and evaluation methods to determine the applicability, performance level, and potential risks of the algorithms under different operating conditions. The background section of this paper also clearly points out that with the advancement of MASS Code, the Operational Design Domain (ODD) has become an important basis for describing the capability boundaries of autonomous systems, organizing tests, and conducting certification verification.

[0003] Currently, research on autonomous collision avoidance algorithms for ships mainly focuses on two technical directions. The first is algorithm design methods oriented towards collision avoidance decision-making, such as rule-based, path planning, geometric analysis, optimization computation, or intelligent learning methods, used to generate avoidance maneuvers for ships in encounter situations. The second is testing and evaluation methods oriented towards algorithm verification. These typically involve setting up several encounter scenarios in a simulation platform to observe whether the algorithm can complete collision avoidance, and analyzing the algorithm's performance using indicators such as collision risk, nearest encounter distance, nearest encounter time, rule compliance, or navigation efficiency. Existing research has shown that the evaluation of autonomous collision avoidance algorithms for ships involves not only safety but also often multiple aspects such as rule compliance, task efficiency, and maneuvering rationality.

[0004] However, existing technologies still have significant limitations.

[0005] First, existing testing methods generally rely on a small number of manually set typical scenarios or limited discrete samples for verification. While these methods are convenient for demonstrating the algorithm's performance under specific conditions, the limited number of test samples and insufficient parameter coverage make it difficult to reflect the overall distribution characteristics of ship encounter relationships under continuously changing conditions. They also struggle to answer questions about the algorithm's overall performance in a wider range of scenarios, where the boundaries lie, and in which regions vulnerabilities exist. The paper also explicitly points out that if testing is conducted based solely on a small number of fixed or typical scenarios, it is difficult to fully evaluate the algorithm's overall performance within a continuous operating range, and especially difficult to verify its true boundaries.

[0006] Secondly, existing test scenario organization methods often lack a unified parameterized representation. Many solutions treat scenarios merely as a few independent cases, failing to organize environmental conditions, ship relative positions, relative bearings, headings, speeds, traffic density, and interactive behaviors into a computable, coverable, and scalable high-dimensional parameter space. Without this parameterized organization, testing often struggles to balance scenario coverage and representativeness, easily leading to problems such as excessive local samples, insufficient coverage of key risk areas, difficulty in reproducing test results, and challenges in cross-sectional comparisons.

[0007] Furthermore, existing scenario generation methods often rely heavily on experience or randomness. Some methods depend on experts selecting classic encounter situations based on nautical experience, suitable for demonstrative verification but struggling to cover complex, continuously changing multi-parameter scenario spaces. While other methods can automatically generate scenarios, without effective constraints and filtering, they can easily produce a large number of samples with unreasonable initial configurations, insufficient risk significance, or low testing value, resulting in wasted simulation resources and hindering the formation of a high-quality test scenario library. This paper summarizes the problem as: how to form a set of scenarios that is representative, interpretable, and computationally cost-effective within a given operational boundary through reasonable parameter organization, sampling, and filtering.

[0008] Furthermore, most existing algorithm evaluation methods remain at the level of directly interpreting the results of tested samples. Evaluation is typically based only on indicators such as collision performance, DCPA, TCPA, pass distance, and rule compliance in a single or limited number of scenarios. This approach essentially answers "how the algorithm performs in these specific scenarios," but struggles to further address "how the algorithm's overall safety, stability, economy, environmental friendliness, and applicability are across the entire parameter space." Especially when testing costs are high and the number of scenarios is limited, existing methods struggle to identify hazardous areas, performance degradation zones, and potential failure boundaries from a finite set of discrete samples. In particular, there is a lack of a backtracking fitting method and evaluation method specifically for collision avoidance algorithms.

[0009] Furthermore, from an engineering application perspective, existing technologies have a narrow understanding of "failure." Many solutions only consider collisions or direct hazardous outcomes as algorithm failures, without fully considering that even if a collision does not occur, the algorithm may still suffer from rule violations, insufficient passage space, severe maneuvering fluctuations, excessively long detours, increased energy consumption, decreased environmental friendliness, and excessive sensitivity to local parameter disturbances. For intelligent ship collision avoidance algorithms, these issues also affect the algorithm's practical application value, operational reliability, and certification acceptability. Therefore, a technical solution is needed that can comprehensively evaluate from multiple dimensions, including rules, safety, economy, environmental friendliness, maneuvering quality, and robustness.

[0010] In summary, existing intelligent ship collision avoidance algorithm evaluation technologies have at least the following shortcomings: First, scenario construction relies heavily on typical cases or discrete samples, making it difficult to cover the continuously changing and complex encounter space; second, test results are mostly based on direct analysis at the case level, making it difficult to form a judgment on the overall performance distribution, danger zones, and applicable boundaries, and test results cannot be extrapolated back to the ODD level; third, the evaluation dimensions are relatively singular, and cannot take into account multiple requirements such as rule compliance, safety, navigation economy, environmental friendliness, and robustness, and cannot use only indicators in discrete test scenarios for comprehensive algorithm evaluation.

[0011] Existing research and publicly available information indicate that current research in the field of autonomous ship collision avoidance mainly focuses on the design and optimization of the collision avoidance decision-making algorithm itself, as well as simulation verification under a few typical scenarios. Regarding algorithm evaluation, existing solutions primarily rely on single experimental results, verification under limited typical operating conditions, or analysis of single indicators. Overall, current technologies lack a systematic evaluation method for intelligent ship collision avoidance algorithms that can construct a high-dimensional parameter space around its operating conditions and integrate scenario generation, scenario selection, batch simulation, overall evaluation, and multi-dimensional failure identification.

[0012] Therefore, there is an urgent need to propose an evaluation method for intelligent ship collision avoidance algorithms, so as to achieve systematic, scalable and engineering support for the tested algorithms from scenario construction, batch testing to overall evaluation and failure identification. Summary of the Invention

[0013] To address the current limitations of existing research on intelligent ship collision avoidance algorithms, which largely focuses on algorithm design while supporting evaluation methods generally remain at the level of verification using a few typical scenarios, discrete samples, or single-indicator analysis. While these methods can demonstrate whether an algorithm possesses basic collision avoidance capabilities under specific conditions, they struggle to cover the continuously changing and complex encounter space, reveal the algorithm's overall performance distribution, hazardous areas, and applicable boundaries over a wider operating range, and fail to provide sufficient support for engineering applications, market access reviews, and certification testing. Especially when the algorithm faces different target ship behavior patterns, the test results under the same initial scenario often show significant differences. Traditional scenario-by-scenario and case-by-case analysis methods are difficult to effectively utilize such group-structured test data, resulting in the difficulty of extrapolating test results to higher-level operational scope evaluation. Furthermore, existing technologies suffer from problems such as scenario construction relying on typical cases or discrete samples, difficulty in covering continuously changing and complex encounter spaces, test results remaining at the case level and failing to form an overall performance distribution, danger zone and applicable boundary judgment, and a single evaluation dimension that fails to consider rule compliance, safety, navigation economy, environmental friendliness, and robustness. The technical solution adopted in this invention is: an evaluation method for intelligent ship collision avoidance algorithms, including the following steps:

[0014] Based on the expected operating conditions, traffic constraints, rule constraints, and task requirements of the tested intelligent ship collision avoidance algorithm, as well as the different decisions of other ships, a high-dimensional parameter space related to the collision avoidance task is constructed. Within the high-dimensional parameter space, candidate test scenarios are generated through spatial sampling, and the validity of the candidate test scenarios is constrained by a screening method to obtain a set of valid test scenarios. The intelligent ship collision avoidance algorithm under test is connected to a unified simulation test environment, and batch tests are carried out in the set of effective test scenarios to obtain the test results of the intelligent ship collision avoidance algorithm under different scenario parameter combinations, different traffic interaction conditions and different operating states. A proxy model is established to fit the batch test results and to evaluate the continuous performance distribution in a high-dimensional parameter space.

[0015] Furthermore, it also includes establishing a surrogate model based on the batch test results to perform surrogate evaluation of the continuous performance distribution in the high-dimensional parameter space, thereby achieving fitting, estimation, or extrapolation of the continuous performance distribution in the high-dimensional parameter space. Furthermore, based on the results of batch testing or proxy evaluation, a multi-dimensional comprehensive evaluation mechanism for intelligent ship collision avoidance algorithms is constructed to evaluate the tested intelligent ship collision avoidance algorithms. The evaluation dimensions of the multi-dimensional comprehensive evaluation mechanism for intelligent ship collision avoidance algorithms include at least: compliance with international maritime collision avoidance rules, navigation safety, navigation economy, environmental friendliness, maneuverability, and robustness and stability under complex interaction conditions. Based on the evaluation results of each evaluation dimension, a comprehensive evaluation index is output.

[0016] Furthermore, the agent model is improved through three optimization methods: decoupling of scenario and strategy in group testing, enhancement of ship features, and small value sensitive weighting.

[0017] Furthermore, the process of group testing through scenario and strategy decoupling is as follows: The proxy model takes scene input parameters as input and outputs the minimum minimum distance between ships when they meet (DCPA) and the minimum encounter distance. In each fixed initial scenario, combined samples can be formed by traversing the intelligent ship collision avoidance algorithms carried by the two target ships. These samples share the same initial scenario parameters. The difference between the decision combinations of the current ship and the target ship is represented by the continuous sub-response as the sum of the basis term determined by the scenario and the offset term induced by the strategy combination, i.e.:

[0018] In the formula, g(s) represents the scene basis term determined by the initial scene parameters. This represents the policy offset term induced by the target ship algorithm combination in this scenario; The first Let the fixed initial scene be denoted as The corresponding intelligent ship collision avoidance algorithm combination sample set for the target ship is denoted as:

[0019] For a certain continuous sub-response Place it in the scene The following is a combination of intelligent ship collision avoidance algorithms. Corresponding observations To decouple the scene basis from the policy offset, we first construct group mean labels within each scene group:

[0020] In the formula, Indicates a fixed initial scene Below, the average response level of the continuous sub-response; Further define the offset within the group:

[0021] In the formula, The description indicates that, under this fixed scenario, the target ship's continuous response is generally at an average level. This describes a scenario where, due to changes in the combination of intelligent ship collision avoidance algorithms for both the ship and the target ship, the response shifts relative to the scenario baseline.

[0022] Furthermore, the method also includes constructing mean labels for group decomposition based on each scene group, and constructing two sub-models for each continuous sub-response of the ship, including: a scene basis model and an intra-group offset model; First, the scene-based model learns the mapping from scene parameters to group means by taking only fixed initial scene features as input:

[0023] In the formula, The scene basis model is defined as follows. The learning objective of the scene basis model is to estimate the basic level of continuous response in a scene by fixing only the scene parameters such as initial distance, relative bearing, heading, and speed. Secondly, the intra-group offset model uses both scene features and the features of the intelligent ship collision avoidance algorithms carried by the two target ships as input to learn the mapping from the complete sample to the intra-group offset, that is:

[0024] In the formula, This represents the intra-group offset model, which requires explicit use of information from other ship collision avoidance algorithms.

[0025] For the output of each sub-response, the final predicted value of the sub-response output is written as:

[0026] for Each output value is obtained through training. The regression sub-models make a joint prediction.

[0027] Furthermore, the process of enhancing ship features is as follows: By incorporating relevant geometric mechanistic features related to the minimum DCPA, including the difference in heading between the two ships and the ship itself, speed difference, speed ratio, azimuth trigonometric components, relative position Cartesian components, and the interaction term between the default DCPA and relative motion, the prediction accuracy of the surrogate model is improved.

[0028] Furthermore, the expression for the small-value sensitive weighting is as follows: For any sub-response objective First, calculate the 10th percentile of the sub-response target on the training set. Then define the sample weights as follows:

[0029] In the formula, This is the small-value enhancement coefficient.

[0030] Furthermore, the method also includes boundary sample enhancement, whereby when the difference between the sub-responses of the two ships is less than a difference threshold, for the two output quantities... If its child response gap When the threshold is less than 1, it indicates that the contributions of the two ship pairs to the sub-response are similar, and the test scenario is near the boundary of the risk-dominant relationship. set up Let represent the 10th percentile of the corresponding gap in the training set. Then, the boundary sample weights are written as:

[0031] In the formula, The boundary enhancement coefficient, taking into account both small value enhancement and boundary enhancement, is used to determine the final sample weights:

[0032] Under weighted training, the objective function of the surrogate model can be written as:

[0033] In the formula, Let be the sample loss function. For the first Regularization terms for base learners.

[0034] This invention provides an evaluation method for intelligent ship collision avoidance algorithms. The method takes the Operational Design Domain (ODD) as the test object, abstracts the ODD into a high-dimensional parameter space, and generates a test scenario library through a space-filling sampling method and a screening mechanism. Furthermore, it uses a surrogate model to fit the algorithm's performance in the ODD and explore failure situations, thereby providing effectiveness proof and feedback for intelligent ship collision avoidance algorithms and supporting intelligent ship testing and certification methods.

[0035] This invention proposes an evaluation method for intelligent ship collision avoidance algorithms. The method constructs a high-dimensional parameter space oriented towards collision avoidance tasks, based on the expected operating conditions, traffic interaction relationships, rule constraints, and task requirements of the algorithm under test. Candidate test scenarios are then generated within this high-dimensional parameter space, and a set of effective scenarios with coverage and test relevance is formed through screening to support batch simulation testing. In this way, test objects that traditionally rely on a few typical cases can be transformed into continuous, scalable, and organized parameterized test objects, enabling batch evaluation of complex encounter relationships within a unified framework.

[0036] This invention further constructs a unified testing and evaluation process, performing batch testing on the collision avoidance algorithm under test within a valid set of test scenarios. This yields multi-dimensional test results, including trajectory evolution, collision risk changes, clearance safety margin, rule compliance, handling quality, navigation economy, environmental friendliness, and robustness. Unlike existing technologies, this invention does not limit testing to the direct statistical analysis of discrete scenario results. Instead, it utilizes a surrogate evaluation mechanism to elevate finite discrete test results to a characterization of continuous performance distribution in a high-dimensional parameter space, thereby extending the evaluation from specific scenario results to the overall operational scope.

[0037] In the proxy assessment mechanism, this invention addresses the characteristic of multiple target ship behavior combinations forming grouped test data within the same basic scenario. It proposes an intra-group offset decomposition fitting approach, decomposing the test results into a basic risk component determined by the scenario itself and an offset component induced by different interaction strategies. This decomposition method distinguishes between common features within the same basic scenario and additional changes brought about by different strategy combinations, thereby more accurately identifying risk sources and improving the proxy assessment's ability to express both overall structure and local changes. Compared to directly fitting all samples uniformly, this decomposition better reflects the different impacts of "scenario difficulty" and "interaction behavior differences" on the test results. Furthermore, considering the clear physical meaning of nautical indicators in collision avoidance assessments, the invention enhances the expression of risk features related to nearest encounter distance, relative motion relationships, relative positions, and relative navigation states. This allows the proxy model to more directly utilize encounter geometry and relative motion laws, improving its ability to represent the risk formation mechanism. This feature enhancement avoids over-reliance on indirect learning of original variables in the surrogate evaluation process, enhancing the model's ability to identify overall risk distribution, danger zones, and local performance degradation areas. Addressing the issue that danger zone samples are typically few in number but crucial to algorithm evaluation results, a small-value-sensitive weighting approach is proposed. This assigns higher weights to high-risk small-value samples, boundary-sensitive samples, or samples exhibiting local mutations, making the surrogate evaluation process more attentive to changes in danger zones, performance drop intervals, and near boundaries. This method significantly improves the fitting accuracy to high-risk small-value regions, enabling the evaluation to not only describe the average performance in general areas but also more sensitively identify potential weaknesses, local vulnerabilities, and applicability boundaries of the algorithm. Experimental results also show that this improved model performs better in high-risk small-value regions.

[0038] Furthermore, the failure or risk identification of this invention is not limited to a single collision result, but can be comprehensively judged by combining compliance with international maritime collision avoidance rules, safety, maneuverability, navigation economy, environmental friendliness, maneuverability, and sensitivity to scene disturbances and interactive changes. In other words, even if the algorithm does not directly cause a collision, if there are rule violations, insufficient maneuverability, significant redundancy in bypassing, increased energy consumption, unstable maneuvering, abnormal concentration of local risks, or excessive sensitivity to minute parameter changes, these situations can be included in the high-risk, degradation, or failure identification scope of this invention. Through this multi-dimensional and open evaluation framework, this invention can achieve a more comprehensive and engineering-application-oriented evaluation of intelligent ship collision avoidance algorithms.

[0039] In summary, this invention, by constructing a high-dimensional parameter space scene set, conducting batch simulation tests, and combining intra-group offset decomposition, risk feature enhancement, and a proxy evaluation method with small-value sensitivity weighting, achieves an improvement from discrete test results to overall performance distribution, hazardous area identification, and applicable boundary analysis. This method not only improves the coverage of test samples and the interpretability of evaluation results in complex encounter spaces, but also enhances the ability to identify high-risk small-value areas, locally vulnerable areas, and boundary-sensitive areas. It provides a systematic technical path for the research, optimization, performance comparison, access review, and certification testing of intelligent ship collision avoidance algorithms. This method also includes: by constructing a high-dimensional parameter space and combining it with spatial sampling and filtering methods, it breaks through the dependence of traditional typical scenario testing on discrete cases and improves the test object's coverage of continuous and complex encounter spaces; by introducing a proxy evaluation mechanism, it realizes the extension from finite discrete test results to overall performance distribution, hazardous areas, and applicable boundary analysis, enabling test results to be extrapolated to a higher level of operational scope evaluation; by establishing a multi-dimensional comprehensive evaluation mechanism that takes into account rule compliance, safety, navigation economy, greenness, and robustness, it overcomes the problem that existing technologies have a single evaluation dimension and cannot fully reflect the engineering application value of collision avoidance algorithms; fourthly, this invention forms a complete evaluation chain from scenario construction, batch testing, overall evaluation to boundary identification, which is applicable to the testing and evaluation of collision avoidance algorithms for different types of intelligent ships and has strong versatility, scalability, and engineering application value. Compared with the prior art, the present invention has the following advantages: First, this invention constructs a high-dimensional parameter space for intelligent ship collision avoidance tasks and generates an effective set of test scenarios by combining spatial sampling and screening methods. This changes the existing technology's reliance on a small number of typical cases or discrete samples for evaluation. It can more fully cover the continuously changing and complex encounter space, improve the representativeness, coverage, and credibility of the test objects, and at the same time reduce the occupation of test resources by invalid and redundant samples.

[0040] Secondly, by establishing a technical process that combines batch simulation testing with proxy evaluation, this invention extends the results from finite discrete test results to the overall performance distribution in a high-dimensional parameter space. It can identify the risk clustering areas, local vulnerable areas, performance degradation areas, and applicable boundaries of the tested algorithm under different operating conditions, so that the test conclusions are no longer limited to the performance of a single scenario, but can be extrapolated to a higher level of overall operating range evaluation.

[0041] Furthermore, this invention introduces optimization ideas such as scenario and strategy decoupling group modeling, nautical feature enhancement, and small value sensitivity weighting in the agent evaluation process. This can improve the agent model's ability to express key risk information, boundary-sensitive information, and high-risk small value regions, thereby improving the overall evaluation accuracy, information content, and ability to identify local weak links, making the evaluation results more interpretable and practically valuable in engineering.

[0042] Furthermore, this invention constructs a multi-dimensional comprehensive evaluation mechanism that considers rule compliance, safety, navigation economy, environmental friendliness, maneuverability, robustness, and stability. This overcomes the problem of the single evaluation dimension in existing technologies and can more comprehensively reflect the overall performance of intelligent ship collision avoidance algorithms in practical applications. It is also helpful in discovering potential problems that cannot be identified by relying solely on whether a collision occurs or not.

[0043] Finally, this invention can comprehensively identify failure, degradation, high risk, and boundary-sensitive states, providing a basis for the access review, scope declaration, certification testing organization, and subsequent optimization of intelligent ship collision avoidance algorithms. It has strong versatility, scalability, and engineering application value. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the method in this application; Figure 2 These are the results of the ablation experiment: (a) is the target ship DCPA Low-10% MAE, (b) is the target ship DCPAL Low-5% MAE, (c) is the target ship's minimum passing distance Low-10% MAE, and (d) is the target ship's minimum passing distance Low-5% MAE. Figure 3 The results show the performance of the intelligent ship collision algorithm in the slice, where (a) both target ships are in normal V0, (b) both target ships are equipped with normal V0 and speed and term maintenance respectively, (c) both target ships are equipped with normal V0 and compression algorithm respectively, (d) both target ships are in normal V0 and speed and term maintenance, (e) both target ships are equipped with speed and term maintenance and compression algorithm respectively, and (d) both target ships are in compression algorithm. Figure 4 This is a margin failure, showing the performance of the wall-collision algorithm of the intelligent ship under test in the slice; Figure 5The images show the performance of the smart ship collision algorithm under test in the slice, where (a) is the worst DCPA and (b) is the minimum passing distance. Figure 6 This is a diagram showing the performance of the smart ship's collision algorithm in the slice during DCPA cliff failure. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Figure 1 This is a flowchart of the method in this application; An evaluation method for intelligent ship collision avoidance algorithms is proposed. This method addresses the testing, evaluation, and scope analysis needs of intelligent ship collision avoidance algorithms and the different decision-making processes of other vessels. It establishes a complete technical process from test scenario construction, batch simulation testing, result proxy evaluation to multi-dimensional failure identification, so as to improve the evaluation from discrete test results to overall operational scope. The method includes the following steps: S1: Based on the expected operating conditions, traffic constraints, rule constraints, and task requirements of the intelligent ship collision avoidance algorithm under test, a high-dimensional parameter space related to the collision avoidance task is constructed. The high-dimensional parameter space is used to uniformly describe the environmental conditions, relative positional relationship of the target ship, motion state, interactive behavior, and other key factors affecting collision avoidance decisions during the encounter process of ships, thereby transforming the traditional test object that relies on a small number of typical working conditions into a continuous, scalable, and organized parameterized test object. S2: Within the high-dimensional parameter space, candidate test scenarios are generated through spatial sampling methods, and the effectiveness of the candidate test scenarios is constrained by a screening method. Samples with unreasonable configurations, insufficient risks, low test value, or that do not conform to actual navigation conditions are eliminated to obtain a set of effective test scenarios. Through this process, the ability of test scenarios to represent complex encounter spaces can be improved, and the occupation of test resources by invalid and redundant samples can be reduced. S3: Connect the intelligent ship collision avoidance algorithm under test to a unified simulation test environment, conduct batch tests in the set of effective test scenarios, and obtain the test results of the intelligent ship collision avoidance algorithm under different scenario parameter combinations, different traffic interaction conditions, and different operating states.

[0049] The test results include at least trajectory information, collision risk information, passability safety margin, rule compliance, trajectory manipulation characteristics, and operational cost information during the collision avoidance process, and can be further extended to indicators such as energy consumption, environmental friendliness, and stability to form a multi-dimensional test result set.

[0050] S4: Establish a surrogate model to fit the batch test results and evaluate the continuous performance distribution in the high-dimensional parameter space. This enables the fitting, estimation, or extrapolation of the continuous performance distribution in the high-dimensional parameter space, thereby obtaining the overall performance of the tested algorithm under a wider range of operating conditions. Through this surrogate evaluation process, the performance change trend, risk clustering areas, local vulnerable areas, and performance degradation boundaries of the algorithm in the high-dimensional parameter space can be inferred from finite discrete test samples. This also elevates individual test results to system evaluation results oriented towards the overall operating range.

[0051] Steps S1 / S2 / S3 / S4 are executed sequentially.

[0052] Furthermore, based on the results of batch testing or proxy evaluation, a multi-dimensional comprehensive evaluation mechanism for intelligent ship collision avoidance algorithms is constructed to evaluate the tested intelligent ship collision avoidance algorithms. This comprehensive evaluation mechanism considers not only whether a collision occurs or whether collision avoidance is successful, but also compliance with international maritime collision avoidance rules, navigation safety, navigation economy, environmental friendliness, maneuverability, and robustness and stability under complex interaction conditions. At the rule level, the degree of rule compliance of the algorithm under different encounter relationships can be evaluated; at the safety level, changes in collision risk, nearest encounter safety margin, and whether passage space requirements are met can be evaluated; at the economic and environmental levels, the degree of detour, path redundancy, maneuver frequency, operating costs, and energy consumption levels can be evaluated; at the robustness level, the sensitivity of the algorithm to scene parameter disturbances, changes in traffic interaction, and changes in boundary conditions can be evaluated.

[0053] Finally, based on the aforementioned multidimensional evaluation results, this invention can identify the failure, degradation, high-risk, or boundary-sensitive states of the tested algorithm. The failure identification is not limited to the collision results themselves, but also includes rule violations, insufficient clearance margin, deterioration of navigation economics, decreased environmental friendliness, reduced maneuverability, abnormal concentration of local risks, and excessive sensitivity to minute parameter changes. By identifying and distinguishing these situations, information on the dangerous regions, performance degradation regions, and applicable boundaries of the tested algorithm in the high-dimensional parameter space can be obtained. Based on this, a back-analysis of the test results to the operational design domain level can be achieved, providing a basis for algorithm access review, scope declaration, certification testing organization, and subsequent optimization.

[0054] The surrogate model can also employ XGBoost algorithm, Gaussian fitting, random forest, support vector machine or neural network methods, etc.

[0055] Furthermore, the agent model is improved through three optimization methods: decoupling of scenario and strategy in group testing, enhancement of ship features, and small value sensitive weighting.

[0056] Furthermore, the process of group testing through scenario and strategy decoupling is as follows: The proxy model takes scene input parameters as input and outputs the minimum minimum distance between ships when they meet (DCPA) and the minimum encounter distance. In each fixed initial scenario, combined samples can be formed by traversing the intelligent ship collision avoidance algorithms carried by the two target ships. These samples share the same initial scenario parameters. The difference between the decision combinations of the current ship and the target ship is represented by the continuous sub-response as the sum of the basis term determined by the scenario and the offset term induced by the strategy combination, i.e.:

[0057] In the formula, g(s) represents the scene basis term determined by the initial scene parameters. This represents the policy offset term induced by the target ship algorithm combination in this scenario; The first Let the fixed initial scene be denoted as The corresponding intelligent ship collision avoidance algorithm combination sample set for the target ship is denoted as:

[0058] For a certain continuous sub-response Place it in the scene The following is a combination of intelligent ship collision avoidance algorithms for the target ship. Corresponding observations To decouple the scene basis from the policy offset, we first construct group mean labels within each scene group:

[0059] In the formula, Indicates a fixed initial scene Below, the average response level of the continuous sub-response; Further define the offset within the group:

[0060] In the formula, This describes the average level of the target ship's continuous response within the overall group under this fixed scenario; This describes a scenario where, due to changes in the combination of intelligent ship collision avoidance algorithms for both the ship and the target ship, the response shifts relative to the scenario baseline.

[0061] The method also includes constructing mean labels for group decomposition based on each scene group, and constructing two sub-models for each continuous sub-response of the ship, including: a scene basis model and an intra-group offset model; First, the scene-based model learns the mapping from scene parameters to group means by taking only fixed initial scene features as input:

[0062] In the formula, The scene basis model is defined as follows. The learning objective of the scene basis model is to estimate the basic level of continuous response in a scene by fixing only the scene parameters such as initial distance, relative bearing, heading, and speed. Secondly, the intra-group offset model uses both scene features and the features of the intelligent ship collision avoidance algorithms carried by the two target ships as input to learn the mapping from the complete sample to the intra-group offset, that is:

[0063] In the formula, This represents the intra-group offset model, which requires explicit use of information from the target ship's intelligent ship collision avoidance algorithm. For the output of each sub-response, the final predicted value of the sub-response output can be written as:

[0064] for Each output value is obtained through training. The regression sub-models make a joint prediction.

[0065] The method involves enhancing ship features as follows: by incorporating relevant geometric features related to the minimum DCPA, including the difference in heading between the two ships and the ship itself, speed difference, speed ratio, azimuth trigonometric components, relative position Cartesian components, and the interaction term between the default DCPA and relative motion, the prediction accuracy of the surrogate model is improved.

[0066] The method: The expression of the small-value sensitive weighting is as follows: For any sub-response objective First, calculate the 10th percentile of the sub-response target on the training set. Then define the sample weights as follows:

[0067] In the formula, This represents the small-value enhancement coefficient. It includes a sub-response model, which is also a surrogate model. This method also includes boundary sample enhancement. In testing, more attention is paid to changes in boundary samples, which often represent an extreme case, so boundary sample enhancement is necessary. When two ships correspond to samples with very similar sub-responses, for the two output quantities If its child response gap When the threshold is less than 1, it indicates that the contributions of the two ship pairs to the sub-response are similar, and the test scenario is near the boundary of the risk-dominant relationship. set up Let represent the 10th percentile of the corresponding gap in the training set. Then, the boundary sample weights are written as:

[0068] In the formula, The boundary enhancement coefficient, taking into account both small value enhancement and boundary enhancement, is used to determine the final sample weights:

[0069] Under weighted training, the objective function of the surrogate model can be written as:

[0070] In the formula, Let be the sample loss function. For the first Regularization terms for base learners.

[0071] Example 1 An evaluation method for intelligent ship collision avoidance algorithms, wherein the intelligent ship collision avoidance method is aimed at a ship collision avoidance decision-making method based on two-stage safety reinforcement learning; A ship collision avoidance decision-making method based on two-stage safety reinforcement learning includes the following steps: Design a ship motion model based on a set of reinforcement learning scenarios and parameters; Based on the ship motion model and combined with reinforcement learning algorithm, we designed the reward function for compliance with international maritime collision avoidance rules, the reward function for navigation economy, and the reward function for minimum encounter distance, and constructed the first-stage reinforcement learning ship collision avoidance decision model, which is used to make the ship take collision avoidance actions when there are other ships or obstacles in the surrounding area within the first threshold. The first-stage reinforcement learning ship collision avoidance decision model is trained to obtain a well-trained first-stage reinforcement learning ship collision avoidance decision model. Based on the ship motion model and the action probability distribution of the first-stage reinforcement learning ship collision avoidance decision model, the weight of each level 3 action is determined and the proportion of dangerous level 3 actions is used to design the reward function of the second stage and construct the second-stage safety reinforcement learning ship collision avoidance decision model. This model is used to limit the ship's action space to ensure the safety of ship navigation when there are other ships or obstacles in the surrounding area within the second threshold. The second-stage safety reinforcement learning ship collision avoidance decision model is trained to obtain a well-trained second-stage safety reinforcement learning ship collision avoidance decision model. When a ship encounters another ship or obstacle under the first threshold condition, the pre-trained first-stage reinforcement learning ship collision avoidance decision model is used to make collision avoidance decisions during the ship's navigation process. When a ship encounters other ships or obstacles under the second threshold condition, a pre-trained second-stage safety reinforcement learning ship collision avoidance decision-making model is used to make collision avoidance decisions during the ship's navigation process.

[0072] The first threshold condition refers to the maximum detection radius of the radar carried by the ship; The second threshold condition refers to the determination based on the collision risk value between the vessel and other vessels or obstacles.

[0073] A ship collision avoidance decision-making method based on two-stage safety reinforcement learning is a publicly available method; An evaluation method for intelligent ship collision avoidance algorithms includes the following steps: Step S1: Design a high-dimensional parameter space for the ship collision avoidance decision-making method ODD based on two-stage safety reinforcement learning to be tested: Based on the expected operating conditions, task constraints, traffic interaction characteristics, and rule requirements of the intelligent ship collision avoidance algorithm under test, key scenario elements affecting collision avoidance decisions and different decisions of other ships are extracted to establish a high-dimensional parameter space to describe the encounter relationship and operating conditions of ships. The high-dimensional parameter space is used to uniformly represent environmental conditions, the relative position of the target ship, motion state, interactive behavior, and other key factors affecting the algorithm output, so that the test object is expanded from a small number of typical scenarios to a continuous and organized parameterized scenario space.

[0074] For the algorithm to be tested (reinforcement learning algorithm), a policy-fixed reinforcement learning collision avoidance algorithm is defined. The ODD of the algorithm is: the environment is calm with no wind and waves and good visibility. All ship sensors are working normally. The ship is sailing in open water at a speed of 10 knots. There is a scene set of two target ships within a 6 NM range, where the speed of the target ships is limited to 8 to 15 knots.

[0075] For the test algorithms, we define three algorithms: the pressure-based VO collision avoidance algorithm, the speed-maintaining and heading-maintaining strategy collision avoidance algorithm, and the normal VO collision avoidance algorithm, to simulate different strategies of the target ship.

[0076] For coordinate setting and environment iteration, the initial position of the ship is defined as fixed at the origin, the heading is fixed as due north, and the speed is 10 knots.

[0077] From this point on, the geometric encounter situation of the target ship relative to the ship can be determined by four parameters: distance, bearing angle, heading, and speed. The algorithm used by the target ship is defined as a separate discrete parameter. There are a total of 10 dimensions for the two target ships, of which 8 dimensions represent the geometric encounter relationship with the ship and 2 dimensions indicate the strategy used.

[0078] Step S2: Discretize and sample the high-dimensional parameter space and generate the final test scenario library; For the high-dimensional parameter space described in S1, parameter intervals were defined, and the Sobol low-difference sequence spatial sampling and filling method was used to generate 32768 specific scenarios. The continuous parameter intervals are shown in Table 1.

[0079] Next, the 32,768 sampled specific scenarios will be subject to a three-layer screening mechanism. The first layer is to delete unreasonable scenarios, which will delete scenarios where the two ships are too close in the initial scenario and filter the remaining scenarios. The second layer is to filter low-risk scenarios, which will delete scenarios where there is no obvious collision risk in the initial scenario.

[0080] Based on the above methods, 4909 scenarios were finally selected as the scenario library for algorithm testing.

[0081] Step S3: Simulation test platform scenario test: Based on the parameter selection method in S1 and the test scenario library generated in S2, the algorithm under test, the collision avoidance algorithm, the ship motion model, and the information recording module are integrated into the simulation test platform to conduct large-scale testing and generate the performance of the algorithm under test in the test scenario library.

[0082] Step S4: Proxy Model Modeling: Based on the test results in S3, a proxy model for the XGBoost algorithm is further fitted. The input of this model is the scene input parameters proposed in S1, and the output is the minimum DCPA and minimum encounter distance of the scene.

[0083] (1) Group modeling idea of ​​decoupling scenario and strategy In each fixed initial scenario, combined samples can be formed by traversing the intelligent ship collision avoidance algorithms carried by the two target ships. These samples share the same initial scenario parameters; their differences mainly stem from the different decision combinations between the ship and the target ships, thus exhibiting a distinct group structure characteristic. The continuous sub-response can be represented as the sum of the basis terms determined by the scenario and the offset terms induced by the strategy combination, i.e.:

[0084] In the formula, g(s) represents the scenario basis term determined by the initial scenario parameters, and h(s,a_o,a_t) represents the policy offset term induced by the target ship algorithm combination in this scenario.

[0085] The first Let the fixed initial scene be denoted as The corresponding intelligent ship collision avoidance algorithm combination sample set for the target ship is denoted as:

[0086] For a certain continuous sub-response Place it in the scene The following algorithm combination for the target ship Corresponding observations To decouple the scene basis from the policy offset, we first construct group mean labels within each scene group:

[0087] In the formula, Indicates a fixed initial scene The average response level of this continuous sub-response. Since it averages all strategy combinations within the same scenario, it can be approximately understood as the shared base strength determined by the scenario itself, while relatively weakening the local fluctuations caused by the algorithm combination of specific target ships.

[0088] Based on this, the intra-group offset is further defined:

[0089] In the formula, This describes the overall level of the target ship's continuous response under this fixed scenario; This describes how much the response shifts relative to the scene baseline in this scenario due to changes in the algorithm combination between the local and target ships. Therefore, compared to directly... Compared to uniform fitting, group mean-within-group offset decomposition can more clearly distinguish the different sources of influence between scenario factors and policy factors.

[0090] Based on the above label decomposition, two sub-models are constructed for each continuous sub-response: the scene basis model and the intra-group offset model.

[0091] First, the scene-based model learns the mapping from scene parameters to group means by taking only fixed initial scene features as input:

[0092] In the formula, This represents the scenario-based model. Since this model only takes scenario features as input and not the target ship's algorithmic features as input, its learning objective is to estimate the baseline level of continuous response in the scenario based solely on fixed scenario parameters such as initial distance, relative bearing, heading, and speed, without considering the target ship's specific decision-making method.

[0093] Secondly, the intra-group offset model uses both scene features and features from the dual-ship intelligent vessel collision avoidance algorithm as input to learn the mapping from complete samples to intra-group offsets, i.e.:

[0094] In the formula, This represents the intra-group offset model. Unlike the scene-based model, the intra-group offset model requires explicit use of algorithmic combination information because its goal is to characterize how different dual-ship behavior patterns change continuous responses within the same scene.

[0095] For the output of each sub-response, the final predicted value can be written as:

[0096] This also means for Each output value, the method is trained The regression sub-models make a joint prediction.

[0097] (2) The idea of ​​enhancing nautical characteristics Since the output value includes the minimum DCPA of the scene, the accuracy of the model's DCPA prediction is improved by adding relevant geometric mechanism features related to DCPA, including the difference in heading between the two ships and the ship itself, the difference in speed, the speed ratio, the triangular component of the azimuth angle, the Cartesian component of the relative position, and the interaction term between the default DCPA and the relative motion.

[0098] (3) The idea of ​​small value sensitive weighting In the test data, the real focus should not be on the large number of ordinary samples within the safe range, but rather on the DCPA (Discretionary Distance Amount) and local samples where the minimum passing distance suddenly decreases. These samples are usually fewer in number, but often correspond to higher-risk scenarios. If uniform weighted training is still used, the model will prioritize optimizing the overall average error, thus weakening its ability to learn about dangerous low-risk regions.

[0099] For any sub-response objective First, calculate its 10th percentile on the training set. Then define the sample weights as follows:

[0100] In the formula, This is a small-value enhancement coefficient. This directly increases the influence of samples in the danger zone during training.

[0101] (3) Boundary sample enhancement Besides small-value samples, another type of sample that requires close attention is the one where the sub-responses of the two ships are very similar. For the two output quantities... If its child response gap When the value is relatively small, it indicates that the contributions of the two ships to the sub-response are similar, and the system is near the boundary of the risk-dominated relationship. Although such samples may not correspond to the absolute minimum, their vicinity is often accompanied by a shift in risk sources and abrupt changes in outcomes, making them equally important for the model to identify local hazardous structures.

[0102] set up Let represent the 10th percentile of the corresponding gap on the training set. Then the boundary sample weights can be written as:

[0103] In the formula, Here is the boundary enhancement coefficient. Considering both small-value enhancement and boundary enhancement, this paper takes the final sample weight as:

[0104] Under weighted training, the objective function of the surrogate model can be written as:

[0105] In the formula, Let be the sample loss function. For the first The regularization term of the base learner. Compared to standard training, this form differs only in that it assigns higher weights to dangerous samples, thus making the model more inclined to optimize the fitting error in high-risk regions.

[0106] Figure 2These are the results of the ablation experiment: (a) is the target ship DCPA Low-10% MAE, (b) is the target ship DCPAL Low-5% MAE, (c) is the target ship's minimum passing distance Low-10% MAE, and (d) is the target ship's minimum passing distance Low-5% MAE. Based on the above improvements, an ablation experiment was conducted, and the results were as follows: Figure 2 As shown in Tables 1 and 2: Table 1. XGBoost Improved Ablation Results

[0107] Table 2 Ablation results of intra-group offset decomposition at minimum through distance output

[0108] Overall, intra-group offset decomposition improves the model's ability to utilize scene group structures, DCPA feature enhancement improves the overall representation of continuous DCPA responses, and small-value sensitive weighting further enhances the model's ability to characterize high-risk regions. The fitting effect is particularly significantly improved in small-value regions.

[0109] Step S5: Define the failure types of the intelligent ship collision avoidance algorithm: In the embodiments, the failure of the algorithm under test is manifested as four typical modes: algorithm robustness failure, DCPA cliff failure, pass margin failure, and dangerous failure. These four types of failures reflect the potential defects of the algorithm in the logical scenario space from different perspectives, and their meanings can be summarized as shown in Table 3: Table 3. Description of Four Types of Failures

[0110] (1) Robust failure Figure 3 The results show the performance of the intelligent ship collision algorithm in the slice, where (a) both target ships are in normal V0, (b) both target ships are equipped with normal V0 and speed and term maintenance respectively, (c) both target ships are equipped with normal V0 and compression algorithm respectively, (d) both target ships are in normal V0 and speed and term maintenance, (e) both target ships are equipped with speed and term maintenance and compression algorithm respectively, and (d) both target ships are in compression algorithm. Robustness failure describes the sensitivity of an algorithm to small perturbations in scene parameters. If, under fixed algorithm combinations, a small perturbation to the scene parameters causes significant fluctuations in the output, it indicates that the algorithm lacks sufficient stability within that region. To avoid misjudgments due to the target ship algorithm, this failure mode should be studied while maintaining the target ship algorithm. This application uses the changes in minimum DCPA and minimum passing distance as criteria. When the initial parameters in the scene change slightly, a change in the output scene's minimum DCPA and minimum passing distance exceeding 0.2 nautical miles is considered a failure. Specifically, the criteria are: initial distance change within 0.05 nautical miles, initial azimuth and heading change within 3°, and speed change within 0.5 knots. (2) Failure through margin Figure 4 This is a margin failure, showing the performance of the wall-collision algorithm of the intelligent ship under test in the slice; Passage margin failure is used to characterize situations where there is insufficient actual space for passage during a ship encounter. The minimum passing distance is used as the criterion; when the minimum passing distance of any target ship is less than 0.5 nautical miles, the scenario is considered to have experienced passage margin failure.

[0111] (3) Hazardous failure Figure 5 The images show the performance of the smart ship collision algorithm under test in the slice, where (a) is the worst DCPA and (b) is the minimum passing distance. Hazardous failure is used to identify scenarios where both predicted and actual passage risks are at hazardous levels. This paper uses a joint threshold of minimum DCPA and minimum passage distance as the criterion. A failure is defined as when the minimum DCPA and minimum passage distance of any target vessel in the scenario are both less than 0.3 nautical miles.

[0112] (4) DCPA cliff failure Figure 6 This is a diagram showing the performance of the smart ship's collision algorithm in the slice during DCPA cliff failure. DCPA cliff failure is used to characterize the abrupt change in the minimum DCPA in the local parameter space. Let the gradient magnitude of DCPA with respect to the scene parameters be... A DCPA cliff failure is determined to have occurred in this scenario when the following formula is satisfied:

[0113] This criterion reflects the situation where, even with a low current risk level, minor parameter changes can still lead to further rapid compression of DCPA, thus identifying areas in the local response surface where there is a clear downward trend.

[0114] Step S6: Detecting the failure of the two-stage safety reinforcement learning-based ship collision avoidance decision-making method in ODD: Based on the surrogate model trained in S4 and the definition of failure conditions in S5, the algorithm under test is evaluated in ODD.

[0115] Considering that high-dimensional parameter space is difficult to display directly, the "target ship distance - target ship relative azimuth" slice is selected as the main display plane, and the other parameters are fixed as shown in Table 4.

[0116] Table 4 Fixed values ​​of target ship parameters shown in slice display

[0117] This slice can intuitively reflect the impact of changes in geometric relationships on the distribution of failure areas, and the morphological differences of different failure modes on this plane are quite obvious, which makes it easy to identify typical vulnerable areas of the algorithm in the ODD.

[0118] (1) Robust failure The performance of the algorithm under test in slices is shown in Table 5: Table 5 Performance of the algorithm under test in slices

[0119] Based on the above research methods, the performance of the algorithm under test in the overall ODD can be summarized as follows: the algorithm under test has a certain basic security when the oppressive algorithm is not considered. Its main shortcoming is the lack of robustness in the mid-range scenario. However, after considering the oppressive algorithm, the structural vulnerability of the algorithm is further transmitted to the pass safety and hazard control level, which significantly increases the pass margin failure and hazard failure.

[0120] The preliminary results demonstrate that the method proposed in this invention can effectively test and evaluate intelligent ship collision avoidance algorithms, and provide an explanation of the algorithm's applicable scope and feedback.

[0121] This method can also be used to test and evaluate other intelligent ship collision algorithms, including but not limited to analytical model-based methods, intelligent algorithms, and neural network-based methods.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating intelligent ship collision avoidance algorithms, characterized in that: Includes the following steps: Based on the expected operating conditions, traffic constraints, rule constraints, and task requirements of the tested intelligent ship collision avoidance algorithm, as well as the different decisions of other ships, a high-dimensional parameter space related to the collision avoidance task is constructed. Within the high-dimensional parameter space, candidate test scenarios are generated through spatial sampling, and the validity of the candidate test scenarios is constrained by a screening method to obtain a set of valid test scenarios. The intelligent ship collision avoidance algorithm under test is connected to a unified simulation test environment, and batch tests are carried out in the set of effective test scenarios to obtain the test results of the intelligent ship collision avoidance algorithm under different scenario parameter combinations, different traffic interaction conditions and different operating states. A proxy model is established to fit the batch test results and to evaluate the continuous performance distribution in a high-dimensional parameter space.

2. The evaluation method for intelligent ship collision avoidance algorithms according to claim 1, characterized in that: Based on the results of batch testing or proxy evaluation, a multi-dimensional comprehensive evaluation mechanism for intelligent ship collision avoidance algorithms is constructed to evaluate the tested intelligent ship collision avoidance algorithms. The evaluation dimensions of the multi-dimensional comprehensive evaluation mechanism for intelligent ship collision avoidance algorithms include at least: compliance with international maritime collision avoidance rules, navigation safety, navigation economy, environmental friendliness, maneuverability, and robustness and stability under complex interaction conditions. Based on the evaluation results of each evaluation dimension, a comprehensive evaluation index is output.

3. The evaluation method for intelligent ship collision avoidance algorithms according to claim 1, characterized in that: The proxy model is improved through three optimization methods: scenario and policy decoupling group testing, ship feature enhancement, and small value sensitive weighting.

4. The evaluation method for intelligent ship collision avoidance algorithms according to claim 1, characterized in that: The process of group testing through scenario and strategy decoupling is as follows: The proxy model takes scene input parameters as input and outputs the minimum minimum distance between ships when they meet (DCPA) and the minimum encounter distance. In each fixed initial scenario, combined samples can be formed by traversing the intelligent ship collision avoidance algorithms carried by the two target ships. These samples share the same initial scenario parameters. The difference between the decision combinations of the current ship and the target ship is represented by the continuous sub-response as the sum of the basis term determined by the scenario and the offset term induced by the strategy combination, i.e.: In the formula, g(s) represents the scene basis term determined by the initial scene parameters. This represents the policy offset term induced by the target ship algorithm combination in this scenario; The first Let the fixed initial scene be denoted as The corresponding intelligent ship collision avoidance algorithm combination sample set for the target ship is denoted as: For a certain continuous sub-response Place it in the scene The following is a combination of intelligent ship collision avoidance algorithms. Corresponding observations To decouple the scene basis from the policy offset, we first construct group mean labels within each scene group: In the formula, Indicates a fixed initial scene Below, the average response level of the continuous sub-response; Further define the offset within the group: In the formula, The description indicates that, under this fixed scenario, the target ship's continuous response is generally at an average level. This describes a scenario where, due to changes in the combination of intelligent ship collision avoidance algorithms for both the ship and the target ship, the response shifts relative to the scenario baseline.

5. The evaluation method for intelligent ship collision avoidance algorithms according to claim 1, characterized in that, It also includes mean labels based on the group decomposition constructed within each scene group, and two sub-models constructed for each continuous sub-response of the ship, including: scene basis model and intra-group offset model; First, the scene-based model learns the mapping from scene parameters to group means by taking only fixed initial scene features as input: In the formula, The scene basis model is defined as follows. The learning objective of the scene basis model is to estimate the basic level of continuous response in a scene by fixing only the scene parameters such as initial distance, relative bearing, heading, and speed. Secondly, the intra-group offset model uses both scene features and the features of the intelligent ship collision avoidance algorithms carried by the two target ships as input to learn the mapping from the complete sample to the intra-group offset, that is: In the formula, This represents the intra-group offset model, which requires explicit integration of information from other ship collision avoidance algorithms. For the output of each sub-response, the final predicted value of the sub-response output is written as: for Each output value is obtained through training. The regression sub-models make a joint prediction.

6. The evaluation method for intelligent ship collision avoidance algorithms according to claim 4, characterized in that: The process of enhancing ship features is as follows: By adding relevant geometric mechanism features related to the minimum DCPA, including the difference in heading between the two ships and the ship itself, speed difference, speed ratio, azimuth trigonometric components, relative position Cartesian components, and the interaction term between the default DCPA and relative motion, the prediction accuracy of the surrogate model is improved.

7. The evaluation method for intelligent ship collision avoidance algorithms according to claim 4, characterized in that: The expression for the small-value sensitive weighting is as follows: For any sub-response objective First, calculate the 10th percentile of the sub-response target on the training set. Then define the sample weights as follows: In the formula, This is the small-value enhancement coefficient.

8. The evaluation method for intelligent ship collision avoidance algorithms according to claim 5, characterized in that: It also includes boundary sample reinforcement, where, when the difference between the corresponding sub-responses of the two ships is less than a difference threshold, for the two output quantities... If its child response gap When the threshold is less than 1, it indicates that the contributions of the two ship pairs to the sub-response are similar, and the test scenario is near the boundary of the risk-dominant relationship. set up Let represent the 10th percentile of the corresponding gap in the training set. Then, the boundary sample weights are written as: In the formula, The boundary enhancement coefficient, taking into account both small value enhancement and boundary enhancement, is used to determine the final sample weights: Under weighted training, the objective function of the surrogate model can be written as: In the formula, Let be the sample loss function. This is the regularization term for the learner.

Images