System performance evaluation method based on scene complexity

Abstract

The invention relates to a system performance evaluation method based on scene complexity, belongs to the technical field of system performance evaluation, and solves the problem of overlarge error interval of performance evaluation in the prior art. The method comprises the following steps: acquiring different test samples, and calculating the scene complexity of the test samples to obtain a test sample library with the scene complexity; multiple types of system performance tests are carried out in sequence; for each system performance test, carrying out the system performance test at different scene complexity levels based on the test sample library with the scene complexity to obtain test results of the system performance test under the different scene complexity levels; and fusing test results of various types of system performance tests of the same scene complexity level to obtain system performance evaluation results under different scene complexity levels. And relatively accurate system performance evaluation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of system performance evaluation, and in particular to a system performance evaluation method based on scenario complexity. Background Art

[0002] In actual confrontation environments, the performance of the system is affected by many factors and becomes very complicated. The environmental sample space that needs to be evaluated is extremely large. If the environmental sample space is not differentiated, but the performance under the entire sample space is obtained, the sample workload will be huge. Since only limited test conditions can be selected for testing in actual evaluation, the evaluation results obtained under different test conditions may fluctuate within a large range, resulting in a particularly large error range for its performance evaluation indicators. Summary of the invention

[0003] In view of the above analysis, an embodiment of the present invention aims to provide a system performance evaluation method based on scenario complexity, so as to solve the problem of large error range of existing performance evaluation.

[0004] On the one hand, an embodiment of the present invention provides a system performance evaluation method based on scenario complexity, comprising the following steps:

[0005] Obtain different test samples, calculate the scene complexity of the test samples to obtain a test sample library with scene complexity;

[0006] Conduct various types of system performance tests in sequence; for each system performance test, based on a test sample library with scenario complexity, conduct system performance tests at different scenario complexity levels, and obtain test results of system performance tests at different scenario complexity levels;

[0007] The test results of various types of system performance tests at the same scenario complexity level are integrated to obtain the system performance evaluation results at different scenario complexity levels.

[0008] Based on the further improvement of the above method, various types of system performance tests are carried out in sequence; for each system performance test, based on the test sample library with scenario complexity, the system performance test is carried out at different scenario complexity levels, and the test results of the system performance test at different scenario complexity levels are obtained, including:

[0009] Carry out the first simulation test, the second simulation test and the actual test in sequence;

[0010] For each type of system performance test, for each scene complexity level, two groups of samples of the scene complexity level are selected from the test sample library with scene complexity to conduct tests and obtain test results;

[0011] The test results corresponding to the two groups of test samples of the scene complexity level are fused to obtain the test results of this type of system performance test at the scene complexity level.

[0012] Based on the further improvement of the above method, the test results of multiple types of system performance tests at the same scene complexity level are integrated to obtain the system performance evaluation results at different scene complexity levels, including:

[0013] Calculate the weight of each system performance test;

[0014] Based on the weight of each system performance test, the weighted Bayesian data fusion method is used to fuse the test results of multiple types of system performance tests.

[0015] Based on the further improvement of the above method, the weight of each system performance test is calculated in the following way:

[0016] Calculate the statistical confidence level for each system performance test based on the developer risk set for each system performance test;

[0017] The weight of each system performance test is obtained based on the credibility and statistical confidence of each system performance test.

[0018] Based on the further improvement of the above method, the statistical confidence of each system performance test is calculated using the following formula:

[0019]

[0020] Among them, α0, α1 and α2 represent the developer risk of the first, second and third system performance tests respectively, w r0 、w r1 and w r2 Represent the statistical confidence of the first, second and third system performance tests respectively.

[0021] Based on the further improvement of the above method, the weight of each system performance test is calculated using the following formula:

[0022]

[0023] Among them, w r0 、w r1 and w r2 Respectively represent the statistical confidence of the first, second and third system performance tests, w c0 、w c1 and w c2 Represent the credibility of the first, second and third system performance tests respectively, and Represent the weights of the first, second and third system performance tests respectively.

[0024] Based on the further improvement of the above method, the scene complexity of the test sample is calculated in the following way:

[0025] Acquire the sequence images of the test sample and the corresponding reference sequence images;

[0026] For each frame image in the sequence image of the test sample, the target similarity and target occlusion degree corresponding to the frame image are calculated based on the frame image and the reference image corresponding to the frame image; the basic complexity of the frame image is obtained based on the target similarity and the target occlusion degree;

[0027] Calculating the inter-frame information variation degree of the frame image based on the frame image and the frame image before the frame image;

[0028] The complexity of each frame image is obtained based on the basic complexity of each frame image and the information variation between frames in the sequence image of the test sample;

[0029] The scene complexity of the test sample is obtained by weighted summing up the complexity of each frame image.

[0030] Based on the further improvement of the above method, for each frame image in the sequence image of the test sample, the target similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image, including:

[0031] If the target distance corresponding to the frame image exceeds a first threshold, a global background similarity corresponding to the frame image is calculated based on the frame image and a reference image corresponding to the frame image as a target similarity corresponding to the frame image;

[0032] Otherwise, the local background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image as the target similarity corresponding to the frame image.

[0033] Based on the further improvement of the above method, the global background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image in the following manner:

[0034] Searching for an affected area in the frame image based on a target template in a reference image corresponding to the frame image;

[0035] Extracting features of the target template and each affected area respectively; calculating the similarity between the target template and each affected area based on the extracted features;

[0036] The weighted sum of the similarities between the target template and each affected area is used to obtain the global background similarity of the frame image.

[0037] Based on the further improvement of the above method, the target shielding degree corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image in the following manner:

[0038] Respectively calculating the grayscale features, gradient features and texture features of the target area of ​​the frame image and the target template in the reference image corresponding to the frame image;

[0039] Calculate the feature difference map of the grayscale feature, gradient feature and texture feature of the target area and the target template to obtain a comprehensive difference map;

[0040] The weight of each pixel in the target area is calculated based on the Gaussian kernel model to obtain a Gaussian weight map of the target area;

[0041] Based on the comprehensive difference map and Gaussian weight map, the target occlusion degree corresponding to the i-th frame image is calculated using the following formula:

[0042]

[0043] Among them, G(k) represents the value of the k-th pixel in the Gaussian weight map, dis(k) represents the value of the k-th pixel in the integrated difference map, and NL represents the number of pixels in the integrated difference map.

[0044] Compared with the prior art, the present invention obtains different test samples, calculates the scene complexity of the test samples to obtain test samples with scene complexity, thereby quantitatively grading the test samples according to the complexity, and sequentially performs various types of system performance tests. Each system performance test is based on the scene complexity level and is performed at different scene complexity levels to obtain performance results at different complexity levels. Then, the performance results of various types of system performance tests at the same scene complexity level are integrated to obtain performance evaluation results at different scene complexity levels. This allows the test scenarios covered by the test design to be as comprehensive as possible, and more accurate performance evaluation results at different scene complexity levels are obtained under limited test conditions. The evaluation results have a small error range and are more accurate.

[0045] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0047] Figure 1 The present invention is a flowchart of a method for evaluating system performance based on scene complexity according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0049] A specific embodiment of the present invention discloses a system performance evaluation method based on scene complexity, such as Figure 1 shown.

[0050] S1. Obtain different test samples, calculate the scene complexity of the test samples, and obtain a test sample library with scene complexity;

[0051] S2. Conduct various types of system performance tests in sequence; for each system performance test, based on a test sample library with scenario complexity, conduct system performance tests at different scenario complexity levels to obtain test results of system performance tests at different scenario complexity levels;

[0052] S3. The test results of various types of system performance tests at the same scene complexity level are integrated to obtain the system performance evaluation results at different scene complexity levels.

[0053] During implementation, various types of system performance tests include the first simulation test, the second simulation test and the physical test. The first simulation test and the second simulation test are two different simulation tests.

[0054] The system is, for example, a flight system. The test performance effect is not only related to the performance of the system, but also to the scene in which it is located, such as the sun angle, sea scale light, and clouds in the natural environment. In addition, it will be affected by various artificial interferences. Therefore, the scene influencing factors include specific environmental conditions such as the target, the background (for example, the sea background), the weather (for example, the amount of clouds and fog), the time of day, and artificial interference (for example, smoke interference). When the system is tested under the scene conditions of a certain sample, the sequence of images observed from the system's perspective is the scene of the sample.

[0055] The scene is affected by many factors, and the sample space that needs to be evaluated is extremely large. However, in actual evaluation, only limited test conditions can be selected for testing. The present invention reasonably simplifies the huge sample space corresponding to the original complex scene, and selects some typical samples from the huge sample space of the scene according to the principles of coverage and typicality to form a test condition set, which is convenient for subsequent test evaluation.

[0056] Compared with the prior art, the system performance evaluation method based on scenario complexity provided in this embodiment obtains different test samples, calculates the scenario complexity of the test samples to obtain test samples with scenario complexity, thereby quantitatively grading the test samples according to the complexity, and sequentially performs various types of system performance tests, each of which is based on the scenario complexity level and is performed at different scenario complexity levels to obtain performance results at different complexity levels, and then the performance results of various types of system performance tests at the same scenario complexity level are integrated to obtain performance evaluation results at different scenario complexity levels, so that the test scenarios covered by the test design are as comprehensive as possible, and more accurate performance evaluation results at different scenario complexity levels are obtained under limited test conditions, and the evaluation results have a small error range and are more accurate.

[0057] Calculating the scene complexity of the test sample is to calculate the complexity of the sequence image of the test sample. The scene complexity is mainly reflected in the impact of the scene on the system recognition target.

[0058] Therefore, the scene complexity of the test sample is calculated in the following way:

[0059] S11, obtaining a sequence image of a test sample and a corresponding reference sequence image;

[0060] S12, for each frame image in the sequence image of the test sample, calculate the target similarity and target shielding degree corresponding to the frame image based on the frame image and the reference image corresponding to the frame image; obtain the basic complexity of the frame image based on the target similarity and the target shielding degree;

[0061] S13, calculating the inter-frame information variation degree of the frame image based on the frame image and the frame image before the frame image;

[0062] S14, obtaining the complexity of each frame of the image based on the basic complexity of each frame of the image in the sequence of the test sample and the degree of information variation between frames;

[0063] S15. Perform weighted summation on the complexity of each frame of the image to obtain the scene complexity of the test sample.

[0064] It should be noted that the reference sequence images corresponding to the sequence images of the test sample refer to the sequence images taken by the system when there is no interference. The sequence images of the test sample are sequence images with interference added, and the interference can be, for example, weather interference such as clouds and fog, and artificial interference such as smoke. The sequence images of the test sample and the images in the reference sequence images correspond one to one; each frame of the sequence images of the test sample has a corresponding reference image.

[0065] Target similarity refers to the similarity between the target and the background.

[0066] Specifically, for each frame image in the sequence image of the test sample, the target similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image, including:

[0067] If the target distance corresponding to the frame image exceeds a first threshold, a global background similarity corresponding to the frame image is calculated based on the frame image and a reference image corresponding to the frame image as a target similarity corresponding to the frame image;

[0068] Otherwise, the local background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image as the target similarity corresponding to the frame image.

[0069] During implementation, the target similarity is calculated in different ways according to the distance from the target. When the target is far away, the similarity between the global background and the target affects the recognition accuracy from the similarity perspective, so the global background similarity is used as the target similarity; when the target is close, the similarity between the local background and the target mainly affects the recognition accuracy from the similarity perspective, so the local background similarity should be considered as the target similarity, and the corresponding similarity evaluation focus is determined according to the distance from the target, so that it is more suitable for the complexity evaluation of the scene and improves the accuracy of the evaluation. During implementation, the first threshold is determined according to the test distance of the system.

[0070] Specifically, the global background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image in the following manner:

[0071] S211, searching for an affected area in the frame image based on a target template in a reference image corresponding to the frame image;

[0072] S212, extracting features of the target template and each affected area respectively; calculating the similarity between the target template and each affected area based on the extracted features;

[0073] S213: Perform weighted summation on the similarities between the target template and each affected area to obtain the global background similarity of the frame image.

[0074] During implementation, for the i-th frame image in the sequence of images of the test sample, that is, the i-th image to be evaluated, it is necessary to first extract the minimum circumscribed rectangular area of ​​the target from the corresponding reference image to obtain the target template.

[0075] Then, searching the affected area in the frame image based on the target template in the reference image corresponding to the frame image specifically includes:

[0076] S2111, performing sliding scanning on the frame image with the target template size as the sliding window size to extract candidate impact regions;

[0077] S2112. Calculate the mutual correlation coefficient between the candidate influence region and the target template; if the mutual correlation coefficient is greater than a second threshold, the candidate influence region is the influence region.

[0078] In implementation, not all background areas will affect the recognition of the target, and some areas that are significantly different from the target should be excluded to improve computational efficiency. Therefore, the size of the target template is used as the sliding window size, and the sliding step is 1 pixel. A sliding scan is performed in the image to be evaluated to obtain the candidate impact area.

[0079] Specifically, the correlation coefficient between the candidate influence region and the target template is calculated using the following formula:

[0080]

[0081] Where (x,y)∈T i Represents the element (x, y) of the target template corresponding to the i-th frame image, T i (x,y) represents the pixel value of the element (x,y) of the target template, represents the average pixel value of the target template, B ij represents the jth candidate influence region of the i-th frame image, (u, v) represents the center coordinates of the jth candidate influence region, Represents the union pixel value of the jth candidate influence area.

[0082] If the correlation coefficient between the candidate influence region and the target template is greater than a second threshold, the candidate influence region is the influence region. The second threshold can be set according to calculation accuracy and efficiency requirements.

[0083] After the influence area is obtained, the features of the target template and each influence area are calculated respectively, and the similarity between the target template and each influence area is calculated based on the features.

[0084] During implementation, the features may include HOG (Histogram of Oriented Gradient) features and LBP (Local Binary Pattern) features. The similarity is calculated based on each feature, and then the similarity between the target template and the affected area is obtained by adding and averaging.

[0085] During implementation, the following formula is used to calculate the similarity between the target template and the affected area:

[0086]

[0087] Among them, S represents the number of features, L s represents the length of the sth feature, Represents the lth element value of the sth feature of the target template, represents the lth element value of the sth feature of the affected area, and ε represents the parameter. ε is a very small positive number to prevent the denominator from being zero.

[0088] Specifically, the similarity between the target template and each affected area is weighted and summed to obtain the global background similarity of the frame image, including:

[0089] Calculate the distance between the target template and each influence area

[0090] According to the formula Calculate the weight of the jth influence area

[0091] According to the formula Calculate the global background similarity CDGB of the i-th frame image i ;

[0092] Among them, δ xj =|x T -x Bj |,δ yj =|y T -y Bj |,(x T ,y T ) represents the center coordinate of the target template, (x Bj ,y Bj ) represents the center coordinates of the affected area, represents the width and height of the target template corresponding to the i-th frame image, θ represents the rate of change coefficient, CoS j represents the similarity between the target template and the jth impact area, N ia Indicates the number of affected areas.

[0093] The influence of the influence area is related to the distance from the target. The farther the distance, the smaller the influence. In order to reflect this difference, the distance between the target template and each influence area is calculated, and the weight of each influence area is determined based on the distance, so as to accurately obtain the global background similarity.

[0094] When implemented, the weight calculation function satisfies Where d is the distance between the impact area and the center of the target template.

[0095] θ is a positive number related to the rate of change. For the rate of change coefficient θ, when the distance between the affected area and the center of the target template reaches the maximum d max When , the influence degree is the minimum θ0≥3, then the rate of change coefficient is:

[0096]

[0097] The maximum spacing can be calculated based on the image size.

[0098] During implementation, the global background similarity is normalized to the range of [0,1) to facilitate fusion with other indicators.

[0099] When it is close to the target, the local background similarity is used as the target similarity.

[0100] Specifically, the local background similarity corresponding to the frame image is calculated in the following manner:

[0101] S221, extracting the target and its local background from the frame image, and calculating multiple feature maps of the target and its local background;

[0102] S222, calculating a saliency map of each feature map based on an image visual saliency algorithm, and obtaining a comprehensive saliency map of the target and its local background based on the saliency map of each feature map;

[0103] S223, extracting a comprehensive salient map of the target area from the comprehensive salient map of the target and its local background, according to Get the local background similarity corresponding to the frame image, where It represents the mean saliency of the comprehensive saliency map of the target area, and IDLB represents the similarity of the local background corresponding to the frame image.

[0104] The local background similarity reflects the impact of the local background on the target by measuring the prominence of the target area in the local background.

[0105] During implementation, for the i-th frame image in the sequence of images of the test sample, the minimum circumscribed rectangle surrounding the target in the frame image is taken as the target area, and the target and its local area are the area in the frame image that coincides with the center of the target area and has an area twice the size of the target area.

[0106] In implementation, multiple feature maps of the target and its local area are calculated, such as intensity feature map, gradient feature map, LBP feature map and phase consistency feature map. Intensity feature is the grayscale feature of the image. In implementation, the gradient feature of the image is calculated by Sobel operator to obtain the gradient feature map.

[0107] During implementation, the image visual saliency algorithm is, for example, the Graph-Based Visual Saliency (GBVS) algorithm. For each feature map, a saliency map of each feature map is calculated based on the image visual saliency algorithm, and the saliency maps of all feature maps are linearly averaged to obtain a comprehensive saliency map of the target and its local background.

[0108] Then, the image corresponding to the target area is extracted from the comprehensive saliency map to obtain the comprehensive saliency map of the target area. The saliency level of the target area is represented by the mean saliency of the comprehensive saliency map of the target area. The higher the saliency of the target in its local area, the greater the difference between the target and the surrounding area, and the easier it is to detect the target. Therefore, the local background similarity is: in represents the mean saliency of the comprehensive saliency map of the target area of ​​the i-th frame image, and IDLB represents the local background similarity corresponding to the i-th frame image.

[0109] Target obscuration refers to the degree to which the target area is obscured by clouds, fog, and artificial interference.

[0110] For the i-th frame image, the target occlusion degree corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image in the following manner:

[0111] S231, respectively calculating the grayscale features, gradient features, and texture features of the target area of ​​the frame image and the target template in the reference image corresponding to the frame image;

[0112] S232, calculating a feature difference map of grayscale features, gradient features, and texture features of the target area and the target template to obtain a comprehensive difference map;

[0113] S233, calculating the weight of each pixel in the target area based on the Gaussian kernel model to obtain a Gaussian weight map of the target area;

[0114] S234, based on the comprehensive difference map and the Gaussian weight map, the target shielding degree corresponding to the i-th frame image is calculated using the following formula:

[0115]

[0116] Among them, G(k) represents the value of the k-th pixel in the Gaussian weight map, dis(k) represents the value of the k-th pixel in the integrated difference map, and NL represents the number of pixels in the integrated difference map.

[0117] During implementation, the grayscale features of the target area and the target template, that is, the grayscale images of the target area and the target template. The gradient feature can be calculated using the Sobel operator to obtain a gradient feature map, and the texture feature can be obtained using an LBP feature map.

[0118] During implementation, the feature difference map of the grayscale features of the target area and the target template is calculated as an example for explanation.

[0119] For each pixel in the grayscale feature map, the value corresponding to the pixel position in the feature difference map is calculated using the following formula:

[0120]

[0121] Among them, dis(x,y) represents the value at position (x,y) in the feature difference map, T(x,y) represents the value at position (x,y) in the grayscale feature map of the target template, P(x,y) represents the value at position (x,y) in the grayscale feature map of the target area, and ε represents a parameter.

[0122] In the same way, we get the feature difference map of gradient feature and the feature difference map of texture feature. The feature difference maps of the three features are weighted fused to get the comprehensive difference map.

[0123] Usually, the shape of the target is not a standard rectangular frame. Therefore, there are a large number of background pixels in the target area intercepted by the rectangular frame, especially outside the target area. To this end, a Gaussian kernel model is established to assign smaller weights to pixels farther from the center.

[0124] Specifically, the constructed Gaussian kernel model is expressed as:

[0125] G(x,y)=g(x,σ r )·g(y,σ c )

[0126]

[0127] Among them, (r0,c0) is the center position of the Gaussian kernel model, r p and c p Respectively represent the number of pixel rows and columns in the target area. r and σ c Represent the scale factors of the Gaussian kernel function in the vertical and horizontal directions respectively. G(x,y) represents the weight of the (x,y) position, thus obtaining a Gaussian weight map.

[0128] Based on the comprehensive difference map and Gaussian weight map, the target occlusion degree corresponding to the i-th frame image is calculated using the following formula:

[0129]

[0130] Among them, G(k) represents the value of the k-th pixel in the Gaussian weight map, dis(k) represents the value of the k-th pixel in the integrated difference map, and NL represents the number of pixels in the integrated difference map.

[0131] The above calculation formula normalizes the Gaussian weights. Since the value range of dis is [0,1], the value range of LDTI is [0,1]. When the image is not occluded, LDTI = 0. As the degree of occlusion increases, the value of LDTI also increases.

[0132] The basic complexity of the i-th frame image is obtained by weighted summing of the target similarity and target occlusion of the i-th frame image. In implementation, the weights may be the same, all 0.5.

[0133] The change of target information between frames is an important factor in measuring image changes. Traditional indicators are too simple, and usually the first frame is used as a comparison object to calculate the amount of information change. The present invention proposes the target size change and target position change to measure the inter-frame information change of each frame, that is, for the i-th frame image, the target size change and target position change compared with the previous frame image are calculated to measure the inter-frame information change of the i-th frame image.

[0134] Specifically, for each frame image in the sequence of images of the test sample, the inter-frame information variation degree of the frame image is calculated based on the frame image and the previous frame image of the frame image, including:

[0135] Determine the target area in the frame image and the frame image before the frame image respectively;

[0136] Based on the formula Calculate the target size change of the frame image;

[0137] Among them, (w i-1 ,h i-1 ) represents the width and height of the target area in the i-1th frame image, (w i ,h i ) represents the width and height of the target area in the i-th frame image, and k1 represents the normalization factor.

[0138] In implementation, the normalization factor k1 is determined by the maximum change in the target size in the image sequence. The target size change in adjacent frames generally does not exceed the target size. Therefore, in implementation, k1 is set to 2. The target size change of the first frame image of the test sample scene is 0.

[0139] When the width and height of the target do not change, CDOS i = 0; when the target size change is the same as the size of the previous frame target, CDOS i = 1. For some CDOS i If it is greater than 1, it is set to 1, so the final target size change is: CDOS i =min{1,CDOS i}.

[0140] The target position change is a description of the target motion information between frames, including the target motion amplitude and direction change. Therefore, when constructing the target position change index, the target size should be taken into account.

[0141] Based on the above ideas, based on the formula The target position change degree of the frame image is calculated.

[0142] in, △p i (x)=(x i -x i-1 ) represents the displacement of the target in the x direction from the i-1th frame to the i-th frame, △p i (y)=(y i -y i-1 ) represents the displacement of the target in the y direction from the i-1th frame to the i-th frame, △p i-1 (x)=(x i-1 -x i-2 ) represents the displacement of the target in the x direction from the i-2th frame to the i-1th frame, △p i-1 (y)=(y i-1 -y i-2 ) represents the displacement of the target in the y direction from the i-2th frame to the i-1th frame, x i Indicates the x-axis coordinate of the center position of the target area in the i-th frame image, y i Indicates the y-axis coordinate of the center position of the target area in the i-th frame image, x i-1 Indicates the x-axis coordinate of the center position of the target area in the i-1th frame image, y i-1 Indicates the y-axis coordinate of the center position of the target area in the i-1th frame image, x i-2 Indicates the x-axis coordinate of the center position of the target area in the i-2th frame image, y i-2 Indicates the y-axis coordinate of the center position of the target area in the i-2th frame image.

[0143] The normalization factor k2 is set to 2, which means that when the target displacement change is equal to the target size, the target size change reaches the maximum value of 1, and if it is greater than 1, it is also set to 1. Therefore, the final target size change is: CDOP i =min{1,CDOP i When the target is stationary or moving at a constant speed, the target position change degree CDOP i =0.

[0144] The target position change degree of the first frame and the second needle image of the sequence image of the test sample is 0.

[0145] The inter-frame information variation of the i-th frame image is obtained by combining the target size variation and the target position variation. Inter-frame information change index CDI i The value range is [0,1]. When the target has not changed in size and displacement relative to the previous frame, CDI iThe minimum value is 0. As the size change and displacement amplitude increase, CDI i It is also constantly getting bigger.

[0146] After obtaining the basic complexity of each frame and the degree of information variation between frames, the complexity of each frame is obtained using the following formula:

[0147]

[0148] Among them, TRC i and CDI i They represent the basic complexity of the i-th frame image and the degree of information change between frames, and ξ represents the weight.

[0149] Considering that the geometric mean is less affected by extreme values, the complexity of each frame of the image is determined based on the concept of the geometric mean.

[0150] When implemented, the TRC indicator should be assigned more weight, i.e., ξ>0.5.

[0151] After obtaining the complexity of each frame of the image, the complexity of each frame of the image is weighted and summed to obtain the complexity of the sequence of images of the test sample, that is, the scene complexity of the test sample. Since a single frame image with high complexity has a greater impact on the complexity of the sequence of images of the test sample, the weight of the image with high complexity is increased. Specifically, the weight of each frame of the image is calculated in the following way:

[0152]

[0153] Among them, F i Represents the complexity of the i-th frame image in the sequence of test samples, ω i Represents the weight of the i-th frame image.

[0154] After calculating the scene complexity of each test sample, the test samples with scene complexity are obtained to form a test sample library with scene complexity.

[0155] During implementation, the scene complexity is graded, for example, into five levels, the scene complexity [0.0~0.2) belongs to the first level, [0.2~0.4) belongs to the second level, [0.4~0.6) belongs to the third level, [0.6~0.8) belongs to the fourth level, and [0.8~1.0) belongs to the fifth level.

[0156] Conduct various types of system performance tests in sequence. For each system performance test, conduct system performance tests at different levels of scene complexity based on the test sample library with scene complexity, and obtain the test results of system performance tests at different levels of scene complexity. Specifically include:

[0157] Carry out the first type of simulation test, the first type of simulation test and the real object test in sequence;

[0158] For each type of system performance test, for each scene complexity level, two groups of samples of the scene complexity level are selected from the test sample library with scene complexity to conduct tests and obtain test results;

[0159] The test results corresponding to the two groups of test samples of the scene complexity level are fused to obtain the test results of this type of system performance test at the scene complexity level.

[0160] During implementation, the first type of simulation test is carried out first, and two groups of test samples are designed for each scene complexity level.

[0161] During implementation, the following formula is used to determine the number of samples in each group of scene complexity levels:

[0162]

[0163] Among them, α is the developer risk, that is, the probability of making a true error, β is the user risk, that is, the probability of making a false error, δ represents the allowable deviation, σ0 represents the sample standard deviation, and Z α / 2 Indicates the quantile corresponding to α / 2 of the normal distribution table, Z β / 2 Indicates the quantile corresponding to β / 2 in the normal distribution table. During implementation, the allowable deviation and sample standard deviation are set values, and the quantile is obtained by looking up the table.

[0164] Assuming that the required sample size calculated according to the above formula is 1045, the actual sample size can be set to 1100.

[0165] After determining the sample size, two groups of test samples are selected from the samples with a complexity level of Level 3 in all scenarios of this type of system in the sample library according to the principles of randomness, coverage, and typical adversarial condition balance. Each group has a total of 1,100 samples. The first simulation test is carried out according to the design conditions of the samples and the test results, i.e., the system performance results, are obtained. When screening, the complexity should be evenly distributed. For example, the complexity of the test conditions of Level 3 (complexity range 0.4-0.6) should be between 0.4-0.6 to basically meet the uniform distribution.

[0166] After obtaining the test results of the two groups of test samples at each complexity level, the consistency check is first performed. The specific check process is:

[0167] Assuming that the test results of the two groups of test samples come from the same distribution, calculate the empirical distribution functions of the test results of the two groups of samples respectively;

[0168] Calculate the maximum difference between the empirical distribution functions of the test results of two groups of samples Where x represents the performance value, F m (x) and G n (x) represents the empirical distribution function of the test results of the two groups of samples;

[0169] like The test results of the two groups of samples are inconsistent, otherwise, the test samples of the two groups of samples are consistent, where represents the significance level, Indicates the significant level The corresponding critical values, m and n represent the number of samples in the two groups respectively.

[0170] If the consistency check fails, there is usually a problem with the simulation model, so the model needs to be adjusted and the test needs to be repeated.

[0171] If the consistency check is passed, the two groups of test results are fused. Specifically, the weighted Bayesian method is used for data fusion to obtain the test results of the first simulation test under different scene complexity levels. During implementation, the weights of the two groups of test results are the same.

[0172] After completing the first simulation test and passing the consistency test, the next step is to design the second simulation test. The second simulation test can be reasonably planned according to the test cycle and cost. If the cycle and cost permit, system performance tests of all scenario complexity levels can be carried out. If the cycle and cost do not permit, system performance tests of the main scenario complexity levels can be carried out.

[0173] During implementation, the required number of samples is calculated according to the above method for calculating the sample size, and the sample size of each group of tests at different complexity levels is confirmed during the second simulation test.

[0174] After obtaining the test results of the two groups of samples at each complexity level of the second simulation test, a consistency check is performed according to the above-mentioned consistency check method. If the check fails, the model is adjusted. If they are consistent, the two groups of test results are fused using the weighted Bayesian method to obtain the test results of each scene complexity level of the second simulation test.

[0175] When the second simulation test is completed, the samples of the physical test are determined in the same way to obtain the test results at each complexity level of the physical test scenario.

[0176] During implementation, physical tests can be reasonably planned based on the test cycle and cost. If the cycle and cost permit, system performance tests of all scenario complexity levels can be carried out. If the cycle and cost do not permit, system performance tests of major scenario complexity levels can be carried out.

[0177] Physical tests are usually more expensive, so when they are implemented, the sample size can be small. When implemented, the sample size for physical tests can be determined through existing methods, such as hypothesis testing.

[0178] After obtaining the test results of the system performance test for each posture test at different scene complexity levels, the test results of different types of system performance tests at the same scene complexity level are fused to obtain the system performance evaluation results at different scene complexity levels.

[0179] During implementation, the weighted Bayesian data fusion method is used to integrate the test results of multiple types of system performance tests with the same scene complexity level, including:

[0180] Calculate the weight of each system performance test;

[0181] Based on the weight of each system performance test, the weighted Bayesian data fusion method is used to fuse the test results of multiple types of system performance tests.

[0182] In implementation, when weighted Bayesian weights are used for fusion, different types of system performance tests have different weights.

[0183] Specifically, for each scenario complexity level, the weight of each system performance test is calculated in the following way:

[0184] Calculate the statistical confidence level for each system performance test based on the developer risk set for each system performance test;

[0185] The weight of each system performance test is obtained based on the credibility and statistical confidence of each system performance test.

[0186] Specifically, the statistical confidence level of each system performance test is calculated using the following formula:

[0187]

[0188] Among them, α0, α1 and α2 represent the developer risk of the first, second and third system performance tests respectively, w r0 、w r1 and w r2 Represent the statistical confidence of the first, second and third system performance tests respectively.

[0189] Specifically, the weight of each system performance test is calculated using the following formula:

[0190]

[0191] Among them, w r0 、w r1 and w r2Respectively represent the statistical confidence of the first, second and third system performance tests, w c0 、w c1 and w c2 Represent the credibility of the first, second and third system performance tests respectively, and Represent the weights of the first, second and third system performance tests respectively.

[0192] During implementation, the credibility of the system performance test can be determined through expert experience or existing methods.

[0193] After obtaining the weights, the distribution function of the performance test results is obtained based on the weighted Bayesian data fusion method, that is, the probability density function of the performance index to be evaluated at different complexity levels.

[0194] According to the probability density function The performance to be evaluated is at the confidence level The confidence interval for is:

[0195]

[0196] in, The probability density function Percentile, The probability density function Thus, the evaluation results of the performance to be evaluated at different complexity levels are obtained.

[0197] The above gives the general method of evaluation test design and data fusion assessment. Through the above test design and data fusion, performance indicators and their confidence intervals and confidence levels can be obtained. Moreover, the results are supported by strict statistical theories and methods. The evaluation results have a small error range and are more accurate, which can ensure the credibility of the evaluation process.

[0198] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0199] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A system performance evaluation method based on scenario complexity, characterized in that: The following steps are involved: Obtain different test samples, calculate the scene complexity of the test samples to obtain a test sample library with scene complexity; Conduct various types of system performance tests in sequence; for each system performance test, based on a test sample library with scenario complexity, conduct system performance tests at different scenario complexity levels, and obtain test results of system performance tests at different scenario complexity levels; The test results of various types of system performance tests at the same scenario complexity level are integrated to obtain the system performance evaluation results at different scenario complexity levels.

2. The system performance evaluation method based on scenario complexity according to claim 1 is characterized in that: Conduct various types of system performance tests in sequence; for each system performance test, based on the test sample library with scenario complexity, conduct system performance tests at different scenario complexity levels, and obtain test results of system performance tests at different scenario complexity levels, including: Carry out the first simulation test, the second simulation test and the actual test in sequence; For each type of system performance test, for each scene complexity level, two groups of samples of the scene complexity level are selected from the test sample library with scene complexity to conduct tests and obtain test results; The test results corresponding to the two groups of test samples of the scene complexity level are fused to obtain the test results of this type of system performance test at the scene complexity level.

3. The system performance evaluation method based on scenario complexity according to claim 1, characterized in that: The test results of various types of system performance tests at the same scene complexity level are integrated to obtain the system performance evaluation results at different scene complexity levels, including: Calculate the weight of each system performance test; Based on the weight of each system performance test, the weighted Bayesian data fusion method is used to fuse the test results of multiple types of system performance tests.

4. The system performance evaluation method based on scenario complexity according to claim 3 is characterized in that: The weight of each system performance test is calculated as follows: Calculate the statistical confidence level for each system performance test based on the developer risk set for each system performance test; The weight of each system performance test is obtained based on the credibility and statistical confidence of each system performance test.

5. The system performance evaluation method based on scenario complexity according to claim 4 is characterized in that: The statistical confidence level for each system performance test is calculated using the following formula: Among them, α0, α1 and α2 represent the developer risk of the first, second and third system performance tests respectively, w r0 、w r1 and w r2 Represent the statistical confidence of the first, second and third system performance tests respectively.

6. The system performance evaluation method based on scenario complexity according to claim 4 is characterized in that: The weight of each system performance test is calculated using the following formula: Among them, w r0 、w r1 and w r2 Respectively represent the statistical confidence of the first, second and third system performance tests, w c0 、w c1 and w c2 Represent the credibility of the first, second and third system performance tests respectively, and Represent the weights of the first, second and third system performance tests respectively.

7. The system performance evaluation method based on scenario complexity according to claim 1, characterized in that: The scene complexity of the test sample is calculated in the following way: Acquire the sequence images of the test sample and the corresponding reference sequence images; For each frame image in the sequence image of the test sample, the target similarity and target occlusion degree corresponding to the frame image are calculated based on the frame image and the reference image corresponding to the frame image; the basic complexity of the frame image is obtained based on the target similarity and the target occlusion degree; Calculating the inter-frame information variation degree of the frame image based on the frame image and the frame image before the frame image; The complexity of each frame image is obtained based on the basic complexity of each frame image and the information variation between frames in the sequence image of the test sample; The scene complexity of the test sample is obtained by weighted summing up the complexity of each frame image.

8. The system performance evaluation method based on scenario complexity according to claim 7 is characterized in that: For each frame image in the sequence image of the test sample, the target similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image, including: If the target distance corresponding to the frame image exceeds a first threshold, a global background similarity corresponding to the frame image is calculated based on the frame image and a reference image corresponding to the frame image as a target similarity corresponding to the frame image; Otherwise, the local background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image as the target similarity corresponding to the frame image.

9. The system performance evaluation method based on scenario complexity according to claim 7, characterized in that: The global background similarity corresponding to the frame image is calculated based on the frame image and the reference image corresponding to the frame image in the following manner: Searching for an affected area in the frame image based on a target template in a reference image corresponding to the frame image; Extracting features of the target template and each affected area respectively; calculating the similarity between the target template and each affected area based on the extracted features; The weighted sum of the similarities between the target template and each affected area is used to obtain the global background similarity of the frame image.

10. The system performance evaluation method based on scenario complexity according to claim 7, characterized in that: Based on the frame image and the reference image corresponding to the frame image, the target occlusion degree corresponding to the frame image is calculated in the following manner: Respectively calculating the grayscale features, gradient features and texture features of the target area of ​​the frame image and the target template in the reference image corresponding to the frame image; Calculate the feature difference map of the grayscale feature, gradient feature and texture feature of the target area and the target template to obtain a comprehensive difference map; The weight of each pixel in the target area is calculated based on the Gaussian kernel model to obtain a Gaussian weight map of the target area; Based on the comprehensive difference map and Gaussian weight map, the target occlusion degree corresponding to the i-th frame image is calculated using the following formula: Among them, G(k) represents the value of the k-th pixel in the Gaussian weight map, dis(k) represents the value of the k-th pixel in the integrated difference map, and NL represents the number of pixels in the integrated difference map.

Images