A method for evaluating autonomous identification and analysis algorithm of reconnaissance payload targets

By decomposing the target recognition and analysis functions of the reconnaissance payload target autonomous recognition and analysis algorithm, the comprehensive evaluation results are calculated, and the problem of lack of objective algorithm evaluation in the existing technology is solved, which meets the requirements of reconnaissance payload equipment in complex battlefield environments and provides detailed evaluation As a result, it is convenient for users to choose the preferred algorithm.

CN118094173BActive Publication Date: 2024-11-08BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311537798.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-17
Publication Date
2024-11-08
Estimated Expiration
2043-11-17

AI Technical Summary

Technical Problem

The existing technology lacks the quality of objective evaluation of the autonomous identification and analysis algorithm for reconnaissance payload targets, and cannot effectively help users choose appropriate algorithms, and lacks decomposition and evaluation of functional modules.

Method used

A method of evaluation of the algorithm for independent identification and analysis of reconnaissance payload targets is proposed. By obtaining the algorithm's target recognition results and attack sequence analysis results, the algorithm's comprehensive evaluation results are calculated, including identification number scores, strike sequence scores, search scores, identification scores, and stability. Tracking scores, etc. are decomposed into evaluations of target recognition functions and target analysis functions, to meet the special needs of reconnaissance payload equipment in complex battlefield environments.

Benefits of technology

The objective evaluation of the autonomous identification and analysis algorithm of reconnaissance payload targets is realized, which can meet the task needs in complex environments such as high dynamics, high uncertainty and strong confrontation, and provides detailed evaluation results to facilitate users to choose the preferred algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118094173B_ABST
    Figure CN118094173B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for evaluating an autonomous identification and analysis algorithm of a reconnaissance payload target, which belongs to the field of autonomous target identification and analysis technology, and solves the problem of the lack of an objective method for evaluating an autonomous identification and analysis algorithm of a reconnaissance payload target in the prior art. The method comprises obtaining a target identification result returned by the autonomous identification and analysis algorithm of the reconnaissance payload target to be evaluated; calculating a target identification evaluation result of the algorithm to be evaluated based on the target identification result; sending target distance information to the algorithm to be evaluated based on the target identification result returned by the algorithm to be evaluated; obtaining a strike sequence analysis result returned by the algorithm to be evaluated; calculating a target analysis evaluation result of the algorithm to be evaluated based on the strike sequence analysis result returned by the algorithm to be evaluated; and obtaining a comprehensive evaluation result of the algorithm to be evaluated based on the target identification scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated. Objective and rapid evaluation of the autonomous identification and analysis algorithm of the target is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous target identification and analysis, and in particular to a method for evaluating an algorithm for autonomous identification and analysis of a reconnaissance payload target. Background Art

[0002] The autonomous recognition and analysis technology of reconnaissance payload targets is an important component of the perception module of unmanned systems. This technology includes the autonomous recognition of typical combat targets in a battlefield environment, and the judgment and ranking of the threat level of targets based on the recognition results.

[0003] With the development of technologies such as deep learning, there are currently many algorithms that can realize autonomous identification and analysis of reconnaissance payload targets. The identification and analysis capabilities of different algorithms vary. There is currently a lack of corresponding methods for objectively evaluating different algorithms to help users choose more suitable algorithms, and there is also a lack of decomposition and evaluation of the various functional modules required for reconnaissance payloads to complete established tasks. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a method for evaluating an autonomous recognition and analysis algorithm of a reconnaissance payload target, so as to solve the problem that there is a lack of objective methods for evaluating the quality of an autonomous recognition and analysis algorithm of a reconnaissance payload target.

[0005] On the one hand, an embodiment of the present invention provides a method for evaluating an autonomous identification and analysis algorithm of a reconnaissance payload target, comprising the following steps:

[0006] Obtaining a target recognition result transmitted back by the algorithm to be evaluated, wherein the algorithm is an autonomous recognition and analysis algorithm for reconnaissance payload targets; calculating a target recognition evaluation result of the algorithm to be evaluated based on the target recognition result transmitted back by the algorithm to be evaluated;

[0007] Obtain the strike sequence analysis result returned by the algorithm to be evaluated; calculate the target analysis and evaluation result of the algorithm to be evaluated based on the strike sequence analysis result returned by the algorithm to be evaluated;

[0008] A comprehensive evaluation result of the algorithm to be evaluated is obtained based on the target recognition scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated.

[0009] Based on the further improvement of the above method, the target analysis and evaluation results of the algorithm to be evaluated are calculated according to the attack sequence analysis results returned by the algorithm to be evaluated, including:

[0010] For each target type, the recognition number score of the target type by the algorithm to be evaluated is calculated according to the number of real targets of the target type and the number of targets of the target type recognized by the algorithm to be evaluated;

[0011] For each target type, the strike order score of the algorithm to be tested for the target type is calculated based on the difference between the strike order of the real target of the target type and the strike order returned by the algorithm to be tested;

[0012] The target analysis and evaluation results of the algorithm to be tested are obtained based on the recognition number score and strike order score of each target type of the algorithm to be tested.

[0013] Based on the further improvement of the above method, the target analysis and evaluation results of the algorithm to be evaluated are calculated using the following formula:

[0014]

[0015] Among them, β j represents the weight of the jth target type, m represents the number of target types, and D 1j represents the recognition score of the jth target type, D 2j represents the strike order score of the j-th target type, and D represents the target analysis and evaluation result of the algorithm to be evaluated.

[0016] Based on the further improvement of the above method, the target recognition evaluation result of the algorithm to be evaluated is calculated according to the target recognition result returned by the algorithm to be evaluated, including:

[0017] Calculate the target search score of the algorithm to be evaluated based on the target position and recognition time in the target recognition result returned by the algorithm to be evaluated;

[0018] Obtaining the target recognition score of the algorithm to be evaluated according to the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated;

[0019] Calculate the target stability tracking score of the algorithm to be evaluated according to the target identifier in the target recognition result returned by the algorithm to be evaluated;

[0020] The target recognition evaluation result of the algorithm to be evaluated is obtained based on the target search score, target recognition score and target stable tracking score.

[0021] Based on the further improvement of the above method, the target search score of the algorithm to be evaluated is calculated according to the target position and recognition time in the target recognition result returned by the algorithm to be evaluated, including:

[0022] Determine whether there is a false alarm in each frame of the image based on the target position identified in each frame of the image sent back by the algorithm to be evaluated; calculate the false alarm rate score of the algorithm to be evaluated based on the number of images with false alarms;

[0023] For each real target, the correct search score of the real target is calculated according to the number of times the real target is correctly identified in the target recognition results returned by the algorithm to be evaluated;

[0024] For each real target, the search time score of the real target is calculated according to the time when the real target is correctly identified for the first time in the target recognition result sent back by the algorithm to be evaluated;

[0025] For each real target, the search efficiency score of the real target is calculated based on the start time of the evaluation and the time when the real target is first correctly identified in the target recognition result returned by the algorithm to be evaluated;

[0026] The target search score of the algorithm to be evaluated is calculated based on the false alarm rate score of the algorithm to be evaluated, the correct search score of each real target, the search time score of each real target, and the search efficiency score of each real target.

[0027] Based on the further improvement of the above method, the target search score of the algorithm to be evaluated is calculated using the following formula:

[0028]

[0029] Among them, α i represents the weight of the i-th real target, n represents the number of real targets, A1 represents the false alarm rate score of the algorithm to be evaluated, and A 2i represents the correct search score of the i-th true target, A 3i represents the search time score of the i-th true target, A 4i represents the search efficiency score of the i-th real target, and A represents the target search score of the algorithm to be evaluated.

[0030] Based on the further improvement of the above method, the following method is used to determine whether there is a false alarm in the image sent back by the algorithm to be evaluated:

[0031] For each bounding box in the image, calculate the intersection and union ratio of the bounding box and the bounding box of each real target in the frame image. If the intersection and union ratio of the bounding box and the bounding box of the real target is greater than the threshold, the bounding box is valid, otherwise, the bounding box is invalid;

[0032] If the number of invalid bounding boxes in the image exceeds a preset threshold, there is a false alarm in the image frame.

[0033] Based on the further improvement of the above method, the target recognition score of the algorithm to be evaluated is obtained according to the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated, including:

[0034] For each real target, the target type recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized type is the type of the real target in the target recognition results returned by the evaluation algorithm;

[0035] For each real target, the target state recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized state is the state of the real target in the target recognition results returned by the evaluation algorithm;

[0036] The target recognition score of the algorithm to be evaluated is obtained based on the target type recognition score and target state recognition score of each real target.

[0037] Based on the further improvement of the above method, the target recognition score of the algorithm to be evaluated is calculated using the following formula:

[0038]

[0039] Among them, α i represents the weight of the i-th true target, n represents the number of true targets, B 1i represents the target type recognition score of the i-th real target, B 2i represents the target state recognition score of the i-th real target, and B represents the target recognition score of the algorithm to be evaluated.

[0040] Based on the further improvement of the above method, the target stable tracking score of the algorithm to be evaluated is calculated according to the target identifier in the target recognition result returned by the algorithm to be evaluated, including:

[0041] For each real target, obtain the image in which the real target is correctly identified in the target recognition result returned by the algorithm to be evaluated, and extract the target identifier of the identified target corresponding to the real target; obtain the target stability tracking score of the real target according to the number of target identifiers;

[0042] The target stability tracking score of the algorithm to be evaluated is obtained based on the target stability tracking score of each real target.

[0043] Compared with the prior art, the present invention decomposes the evaluation of the autonomous recognition and analysis algorithm of the reconnaissance payload target into the evaluation of the target recognition function and the evaluation of the target analysis function, thereby refining the autonomous recognition task of the reconnaissance payload target, so that the evaluation result can meet the special working conditions of the reconnaissance payload equipment (the reconnaissance payload equipment has complex battlefield environment conditions such as high dynamics, high uncertainty, strong denial and strong confrontation), and calculates the target recognition score result of the algorithm to be evaluated and the target analysis score result of the algorithm to be evaluated based on the target recognition result returned by the algorithm to be evaluated, so as to objectively obtain the comprehensive evaluation result of the algorithm to be evaluated. The method is simple and easy to implement.

[0044] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0046] Figure 1 It is a flow chart of a method for autonomous identification and analysis algorithm evaluation of reconnaissance payload targets according to an embodiment of the present invention;

[0047] Figure 2 This is a flow chart of target recognition and evaluation of a single-frame image implemented in the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0049] Before evaluating the autonomous recognition and analysis algorithm of reconnaissance payload targets, we first arrange the targets to be reconnaissanceed in a real environment according to their difficulty levels, and simulate the common working conditions of the reconnaissance payload equipment during reconnaissance (such as the two-degree-of-freedom rotation of the servo turntable, the field of view type of the camera, and other basic adjustment conditions) to collect image information (including visible light images, infrared images), ranging information (distance of the target), and other information.

[0050] Among them, the targets to be reconnaissance include four types: armored equipment, vehicles, fortifications, and personnel. The status of armored equipment and vehicles includes moving and stationary. The status of personnel includes shooting, throwing, moving, and crawling.

[0051] Before the test, the multiple frames of images used for the test are first determined, and each frame of the image is identified by a unique timestamp. Then the real target in each frame of the image is determined, and a unique ID number is assigned to each real target. The real target information includes the real target ID, the real target's bounding box, the real target type, the real target's state, and the real target's distance. For each type of real target, each real target is sorted according to the threat level based on the distance of the real target in the multiple frames of the image and the state of the real target. The target distance belongs to a high-level threat level, and the closer the distance, the higher the threat; the target state belongs to a low-level threat level. For armored equipment and vehicles, the threat level of the mobile state is greater than that of the stationary state. For personnel, the threat level is from high to low: shooting, throwing, moving, crawling and hiding, and stationary threat level. According to this sorting principle, the attack order of the real target of each target type is obtained.

[0052] Afterwards, the multiple frames of images used for testing and the status information of the reconnaissance payload equipment corresponding to each frame of the image (servo turntable pitch angle, servo turntable azimuth angle visible light field of view mode information, infrared field of view mode information), and the target information to be identified are sent to the algorithm to be evaluated. The algorithm to be evaluated performs target identification and analysis and transmits the results of the identification and analysis back. The algorithm to be evaluated can be evaluated based on the identification results returned by the algorithm. The target information includes the image information where the target is located, the camera mode information, and the target type information. The image information where the target is located is used to indicate whether the target is located in the infrared image or the visible light image, and guide the test subject to select the visible light or infrared image as the main recognition image according to the information; the camera mode information indicates whether the client under test has the conditions for the fusion processing of the visible light image and the infrared image; the target type information guides the client under test to identify the main test target class at the end.

[0053] A specific embodiment of the present invention discloses a method for evaluating an autonomous identification and analysis algorithm of a reconnaissance payload target, such as Figure 1 As shown, the following steps are included:

[0054] S1. Obtain the target recognition result transmitted back by the algorithm to be evaluated, where the algorithm is an autonomous recognition and analysis algorithm for reconnaissance payload targets; calculate the target recognition evaluation result of the algorithm to be evaluated based on the target recognition result transmitted back by the algorithm to be evaluated;

[0055] S2. Obtain the strike sequence analysis result returned by the algorithm to be evaluated; calculate the target analysis and evaluation result of the algorithm to be evaluated based on the strike sequence analysis result returned by the algorithm to be evaluated;

[0056] S3. Obtain a comprehensive evaluation result of the algorithm to be evaluated based on the target recognition scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated.

[0057] The present invention decomposes the evaluation of the autonomous recognition and analysis algorithm of the reconnaissance payload target into the evaluation of the target recognition function and the evaluation of the target analysis function, thereby refining the autonomous recognition task of the reconnaissance payload target, so that the evaluation result can meet the special working conditions of the reconnaissance payload equipment (the reconnaissance payload equipment has complex battlefield environment conditions such as high dynamics, high uncertainty, strong denial and strong confrontation), and calculates the target recognition scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated based on the target recognition result returned by the algorithm to be evaluated, so as to objectively obtain the comprehensive evaluation result of the algorithm to be evaluated. The method is simple and easy to implement.

[0058] The target recognition results returned by the algorithm to be evaluated include the target information recognized in each frame of the image, including the recognized target ID, the bounding box of the recognized target, the type of recognized target, and the status of the recognized target. By comparing with the real target in each frame of the image, the target recognition evaluation results of the algorithm to be evaluated can be obtained.

[0059] Specifically, in step S1, the target recognition evaluation result of the algorithm to be evaluated is calculated according to the target recognition result returned by the algorithm to be evaluated, including:

[0060] S11, calculating the target search score of the algorithm to be evaluated according to the target position and recognition time in the target recognition result returned by the algorithm to be evaluated;

[0061] S12, obtaining a target recognition score of the algorithm to be evaluated according to the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated;

[0062] S13, calculating the target stable tracking score of the algorithm to be evaluated according to the target identifier in the target recognition result returned by the algorithm to be evaluated;

[0063] S14. Obtain the target recognition evaluation result of the algorithm to be evaluated according to the target search score, the target recognition score and the target stable tracking score.

[0064] The correctness of target search is to assess whether the algorithm to be evaluated can correctly search for the target. The more targets are correctly identified, the higher the target search score is. The recognition time is to assess the efficiency of the algorithm to be evaluated in searching for the target. The earlier the target is correctly identified, the higher the target search score is.

[0065] Specifically, step S11 calculates the target search score of the algorithm to be evaluated according to the target position and recognition time in the target recognition result returned by the algorithm to be evaluated, and includes the following steps S111-S115.

[0066] S111, judging whether there is a false alarm in each frame of the image according to the target position identified in each frame of the image transmitted back by the algorithm to be evaluated; and calculating the false alarm rate score of the algorithm to be evaluated according to the number of images with false alarms;

[0067] In order to avoid the situation where the algorithm to be measured deliberately sets many bounding boxes in order to cover the real target, a false alarm rate indicator is set. If there are too many invalid bounding boxes in a frame image returned by the algorithm to be evaluated, it is considered a false alarm image.

[0068] During implementation, the following method is used to determine whether there is a false alarm in the image:

[0069] For each bounding box in the image, calculate the intersection and union ratio of the bounding box and the bounding box of each real target in the frame image. If the intersection and union ratio of the bounding box and the bounding box of the real target is greater than the threshold, the bounding box is valid, otherwise, the bounding box is invalid;

[0070] If the number of invalid bounding boxes in the image exceeds a preset threshold, there is a false alarm in the image frame.

[0071] During implementation, the threshold can be set according to the specific false alarm rate requirements. For the algorithm to be evaluated, each frame is judged frame by frame whether it is a false alarm image. If the number of invalid bounding boxes in a frame exceeds the preset threshold, such as 10, the frame is considered a false alarm image, and the false alarm rate count is increased by one. After traversing each image frame by frame, if the false alarm rate count is greater than the false alarm rate tolerance number X e , then according to the formula A=X e -X calculates the false alarm rate score, where X e Indicates that the algorithm is allowed to have a certain false recognition rate, X represents the false alarm rate count, and A represents the false alarm rate score. If the false alarm rate count is less than or equal to the false alarm rate tolerance number Xe, then A=0.

[0072] S112. For each real target, calculate the correct search score of the real target according to the number of times the real target is correctly identified in the target recognition results returned by the algorithm to be evaluated;

[0073] During implementation, for a real target, each frame of the image returned by the algorithm to be evaluated is judged frame by frame. If the real target exists in the frame, it is judged whether there is a bounding box whose intersection and union ratio with the bounding box of the real target is greater than the threshold in the bounding box identified by the algorithm to be evaluated. If so, the correct search count P is increased by 1. After traversing frame by frame, the correct search count P is calculated by formula A. 2i= P+X p / N p Among them, N p Indicates the total number of frames where the real target appears, that is, the real target appears in N p Appeared in images, X p A represents the correct search error tolerance number, which means that the algorithm is allowed to have a certain recognition error rate. 2i represents the correct search score of the i-th true target.

[0074] S113, for each real target, calculating the search time score of the real target according to the time when the real target is correctly identified for the first time in the target recognition result returned by the algorithm to be evaluated;

[0075] The evaluation of the search efficiency of the algorithm to be evaluated includes the time difference between the first time the algorithm searches for the real target and the first time the real target appears. In implementation, for a real target, the time of its first appearance (the timestamp of the image in which it is located) is counted as ts, the time when the real target is first correctly identified in the result returned by the algorithm to be evaluated is counted as t, and the preset search time threshold is counted as tse. If t-ts≥tse, then A 3i =0, otherwise Among them A 3i Indicates the search time score of the i-th real target. For example, a real target appears for the first time in the i-th frame image, and the algorithm to be evaluated recognizes the real target for the first time in the j-th frame image. If the time difference between the j-th frame image and the i-th frame image is greater than or equal to the preset time threshold, then this item is not scored. Otherwise, the search time score of the real target is calculated according to the above formula.

[0076] S114. For each real target, the search efficiency score of the real target is calculated according to the start time of the evaluation and the time when the real target is first correctly identified in the target recognition result returned by the algorithm to be evaluated;

[0077] The evaluation of the search efficiency of the algorithm to be evaluated also includes the time it takes for the algorithm to search for the target for the first time. During implementation, the time to start the evaluation is counted as tc. For a real target, the time when the real target is correctly identified for the first time in the results returned by the algorithm to be evaluated is counted as ta. The preset search efficiency threshold is tee. If ta-tc≥tee, then A 4i =0, otherwise Among them, A 4i represents the search efficiency score of the i-th true target.

[0078] It should be noted that, during implementation, in order to correctly evaluate the search efficiency of the algorithm, before sending the multiple frames of images used for testing to the algorithm to be evaluated, the image names are first encrypted and sorted in random order, and then the image set is sent to the algorithm to be evaluated. Only after receiving the data decryption token can the algorithm to be evaluated decrypt the image names according to the decryption logic to obtain the correct test image order, so that before starting the evaluation, the algorithm to be evaluated cannot perform target recognition in advance by means of sequential indexing of image names.

[0079] During implementation, the evaluator sends decryption tokens frame by frame, that is, the decryption tokens corresponding to each frame of the image to be identified are different. The algorithm to be evaluated finds the current image to be identified and the corresponding working condition information of the reconnaissance payload based on the received decryption tokens, runs its own target autonomous recognition algorithm to identify the target, and sends the recognition results back to the evaluator, who then sends the decryption token of the next frame to the algorithm to be evaluated.

[0080] Only when the evaluation direction sends a decryption token to the algorithm to be evaluated can it start target recognition. Assuming that the algorithm to be evaluated correctly recognizes the real target for the first time when recognizing the i-th frame image, the time tc to start the evaluation is the time when the evaluation direction sends the decryption token of the i-th frame image to the evaluation algorithm.

[0081] S115. Calculate the target search score of the algorithm to be evaluated based on the false alarm rate score of the algorithm to be evaluated, the correct search score of each real target, the search time score of each real target, and the search efficiency score of each real target.

[0082] After obtaining the false alarm rate score of the algorithm to be evaluated, the correct search score of each real target, the correct search score of each real target, and the search efficiency score of each real target, the target search score of the algorithm to be evaluated is calculated using the following formula:

[0083]

[0084] Among them, α i represents the weight of the i-th real target, n represents the number of real targets, A1 represents the false alarm rate score of the algorithm to be evaluated, and A 2i represents the correct search score of the i-th true target, A 3i represents the search time score of the i-th true target, A 4i represents the search efficiency score of the i-th real target, and A represents the target search score of the algorithm to be evaluated.

[0085] The algorithm to be evaluated must not only correctly and quickly identify the target, but also correctly identify the type and state of the target. Therefore, it is necessary to obtain the target recognition score of the algorithm to be evaluated based on the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated. Step S12 specifically includes:

[0086] For each real target, the target type recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized type is the type of the real target in the target recognition results returned by the evaluation algorithm;

[0087] For each real target, the target state recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized state is the state of the real target in the target recognition results returned by the evaluation algorithm;

[0088] The target recognition score of the algorithm to be evaluated is obtained based on the target type recognition score and target state recognition score of each real target.

[0089] The target recognition results returned by the algorithm to be evaluated include not only the location of each target, but also the type and state information of the target. Therefore, the target type recognition and target state recognition scores are calculated by comparing the recognized type and state with the actual type and state.

[0090] For example, for a real target, it has appeared in Np frames. For each frame in which it appears, the target recognition result returned by the algorithm to be evaluated is used to determine whether the algorithm to be evaluated has correctly identified the real target. If the real target is correctly identified, it is determined whether the type of the marked target is consistent with the type of the real target. If so, the type recognition count is increased by 1. After the traversal is completed, according to B 1i = Type recognition count / total number of frames Np where the target appears Calculate the target type recognition score B of the i-th real target 1i Following the same process, the target state recognition scores of i real targets can be calculated.

[0091] Based on the target type recognition score and target state recognition score of each real target, the target recognition score of the algorithm to be evaluated is calculated using the following formula:

[0092]

[0093] Among them, α i represents the weight of the i-th true target, n represents the number of true targets, B 1i represents the target type recognition score of the i-th real target, B 2i represents the target state recognition score of the i-th real target, and B represents the target recognition score of the algorithm to be evaluated.

[0094] The higher the tracking performance of the algorithm, the higher the probability that the same real target appearing in multiple images will be recognized as the same target. In order to evaluate the tracking performance of the algorithm, the target stable tracking score of the algorithm to be evaluated is calculated according to the target identifier in the target recognition result returned by the algorithm to be evaluated. The target stable tracking score is calculated in the following way:

[0095] For each real target, obtain the image in which the real target is correctly identified in the target recognition result returned by the algorithm to be evaluated, and extract the target identifier of the identified target corresponding to the real target; obtain the target stability tracking score of the real target according to the number of target identifiers;

[0096] The target stability tracking score of the algorithm to be evaluated is obtained based on the target stability tracking score of each real target.

[0097] For example, for the i-th real target, the target recognition result sent back by the evaluation algorithm correctly recognizes the target multiple times, but the recognized targets are identified with different identifiers, that is, different IDs are used to regard them as different objects. If the number of different target identifiers is greater than or equal to the preset tracking calculation threshold, the target recognition score is zero. Otherwise, according to formula C 1i =1-F / Fe to calculate the target stable tracking score, where C 1i The target stability tracking score of the i-th true target, F represents the number of target identifications, and Fe represents the tracking calculation threshold.

[0098] Based on the target stability tracking score of each real target, according to the formula Calculate the target stability tracking score C of the algorithm to be evaluated, where α i represents the weight of the i-th true target, and n represents the number of true targets.

[0099] According to the target search score, target recognition score and target stable tracking score, the target recognition evaluation result Scorce1 of the algorithm to be evaluated is calculated according to the formula Scorce1 = ω1A + ω2B + ω3C. Among them, ω1, ω2, ω3 represent the weights of the target search score, target recognition score and target stable tracking score respectively.

[0100] During implementation, the target recognition evaluation process for a single frame image is as follows: Figure 2As shown, first receive the single-frame target autonomous recognition result containing the absolute timestamp of the test frame returned by the algorithm to be evaluated, and then extract the real target information corresponding to the timestamp, including the bounding box of the real target, the type and state of the real target. According to the intersection-and-union matching principle, the bounding box of the returned target recognition result is matched with the bounding box of the real target. If the number of unsuccessful matches exceeds the false alarm rate threshold, the false alarm rate counter X counts, and the frame is considered to be an invalid frame and does not participate in the calculation of other items. If the false alarm rate threshold is not exceeded, for the bounding box that successfully matches the bounding box of the real target, the correct search counter P of the corresponding real target is counted, and the target type recognition counter M and the target state recognition counter O are respectively updated according to the type and state of the corresponding recognition target of the bounding box and the type and state of the real target. Then determine whether it is the first time that the algorithm to be evaluated recognizes a real target. If so, calculate the search time and search efficiency of the real target. If it is not the first time that a real target is recognized, determine whether the target ID of the recognized target in the target recognition result returned by the algorithm to be evaluated is the same as the ID of any recognized target returned in history. If they are different, the target tracking technology F of the real target is counted. After all the image frames to be tested are completed, the target recognition evaluation result of the algorithm to be evaluated is obtained according to the results of each counter and the above calculation formula.

[0101] The present invention not only evaluates the target recognition ability of the algorithm, but also evaluates the analysis ability of the algorithm. Specifically, according to the target recognition result returned by the algorithm to be evaluated, the target distance information is sent to the algorithm to be evaluated, and then the target analysis evaluation result of the algorithm to be evaluated is calculated according to the strike sequence analysis result returned by the algorithm to be evaluated.

[0102] During implementation, for each real target, record the moment (timestamp of the image) when the real target is first correctly searched in the target recognition results sent back by the algorithm to be evaluated, the moment when the real target is last correctly searched, and the moment when the target identifier (ID) of the target changes in the middle. The timestamp, target identifier, and the distance of the real target at the timestamp are sent to the algorithm to be evaluated as a target distance information. During implementation, the target distance information sent by each real target includes up to 4 target distance information corresponding to the distance information corresponding to the first correct search of the real target and the distance information corresponding to the last correct recognition.

[0103] The algorithm to be evaluated calculates the threat level of each type of target based on the target distance information, the identified target type, and the target state, sorts the attack order based on the prestige, regresses the sorting results, and calculates the target analysis and evaluation results of the algorithm to be evaluated based on the attack order analysis results returned by the algorithm to be evaluated, including:

[0104] For each target type, the recognition number score of the target type by the algorithm to be evaluated is calculated according to the number of real targets of the target type and the number of targets of the target type recognized by the algorithm to be evaluated;

[0105] For each target type, the strike order score of the algorithm to be tested for the target type is calculated based on the difference between the actual strike order of the target type and the strike order returned by the algorithm to be tested;

[0106] The target analysis and evaluation results of the algorithm to be tested are obtained based on the recognition number score and strike order score of each target type of the algorithm to be tested.

[0107] During implementation, the strike sequence analysis results sent back by the algorithm to be evaluated can be shown in Table 1. This includes the number of targets of each target type identified by the algorithm, and the targets with high threat levels in each target type, for example, the top five target IDs of the strike sequence sent back by targets of the types of armored equipment and fortifications, and the top ten target IDs of the strike sequence sent back by targets of the types of vehicles and personnel.

[0108] Table 1

[0109]

[0110]

[0111] For a certain target type, such as armored equipment, the number of real targets belonging to this type is counted as J, and the number of targets belonging to this type identified by the algorithm is K. According to the formula Calculate the recognition data score of this type of target.

[0112] For a certain target type, such as armored equipment, according to the difference between the actual attack sequence of this type and the attack sequence returned by the algorithm to be evaluated, according to the formula Calculate the attack order score of the algorithm to be tested for the target type, where F represents the number of inconsistencies between the attack order returned by the algorithm to be tested and the actual attack order of the target type, and Ge represents the sorting error tolerance number.

[0113] Based on the recognition number score and strike order score of each target type for the algorithm to be tested, the target analysis evaluation result of the algorithm to be tested is calculated using the following formula:

[0114]

[0115] Among them, β j represents the weight of the jth target type, m represents the number of target types, and D 1j represents the recognition score of the j-th target type, D 2jrepresents the strike order score of the j-th target type, and D represents the target analysis and evaluation result of the algorithm to be evaluated.

[0116] A comprehensive evaluation result of the algorithm to be evaluated is obtained based on the target recognition scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated.

[0117] By conducting sub-item evaluations on the algorithm's target recognition and target analysis capabilities, we can objectively obtain the comprehensive performance evaluation results of the reconnaissance payload target autonomous recognition and analysis algorithm, making it easier for users to choose a more optimal algorithm.

[0118] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0119] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for evaluating an autonomous identification and analysis algorithm for a reconnaissance payload target, characterized in that: The following steps are involved: Obtaining a target recognition result transmitted back by the algorithm to be evaluated, wherein the algorithm is an autonomous recognition and analysis algorithm for reconnaissance payload targets; calculating a target recognition evaluation result of the algorithm to be evaluated based on the target recognition result transmitted back by the algorithm to be evaluated; Obtain the strike sequence analysis result returned by the algorithm to be evaluated; calculate the target analysis and evaluation result of the algorithm to be evaluated based on the strike sequence analysis result returned by the algorithm to be evaluated; The strike sequence analysis results include the number of targets of each target type identified by the algorithm to be evaluated and the targets with high threat level in each target type; Obtaining a comprehensive evaluation result of the algorithm to be evaluated based on the target recognition scoring result of the algorithm to be evaluated and the target analysis scoring result of the algorithm to be evaluated; The target analysis and evaluation results of the algorithm to be evaluated are calculated based on the attack sequence analysis results returned by the algorithm to be evaluated, including: For each target type, the recognition number score of the target type by the algorithm to be evaluated is calculated according to the number of real targets of the target type and the number of targets of the target type recognized by the algorithm to be evaluated; For each target type, the strike order score of the algorithm to be tested for the target type is calculated based on the difference between the strike order of the real target of the target type and the strike order returned by the algorithm to be tested; The target analysis and evaluation results of the algorithm to be evaluated are obtained based on the recognition number score and the strike order score of each target type for the algorithm to be evaluated; The target analysis and evaluation results of the algorithm to be evaluated are calculated using the following formula: Among them, β j represents the weight of the jth target type, m represents the number of target types, and D 1j represents the recognition score of the jth target type, D 2j represents the strike order score of the j-th target type, and D represents the target analysis and evaluation result of the algorithm to be evaluated; The target recognition evaluation result of the algorithm to be evaluated is calculated based on the target recognition result returned by the algorithm to be evaluated, including: Calculate the target search score of the algorithm to be evaluated based on the target position and recognition time in the target recognition result returned by the algorithm to be evaluated; Obtaining the target recognition score of the algorithm to be evaluated according to the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated; Calculate the target stability tracking score of the algorithm to be evaluated according to the target identifier in the target recognition result returned by the algorithm to be evaluated; The target recognition evaluation result of the algorithm to be evaluated is obtained based on the target search score, target recognition score and target stable tracking score.

2. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 1 is characterized in that: The target search score of the algorithm to be evaluated is calculated based on the target position and recognition time in the target recognition result returned by the algorithm to be evaluated, including: Determine whether there is a false alarm in each frame of the image based on the target position identified in each frame of the image sent back by the algorithm to be evaluated; calculate the false alarm rate score of the algorithm to be evaluated based on the number of images with false alarms; For each real target, the correct search score of the real target is calculated according to the number of times the real target is correctly identified in the target recognition results returned by the algorithm to be evaluated; For each real target, the search time score of the real target is calculated according to the time when the real target is correctly identified for the first time in the target recognition result sent back by the algorithm to be evaluated; For each real target, the search efficiency score of the real target is calculated based on the start time of the evaluation and the time when the real target is first correctly identified in the target recognition result returned by the algorithm to be evaluated; The target search score of the algorithm to be evaluated is calculated based on the false alarm rate score of the algorithm to be evaluated, the correct search score of each real target, the search time score of each real target, and the search efficiency score of each real target.

3. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 2 is characterized in that: The target search score of the algorithm to be evaluated is calculated using the following formula: Among them, α i represents the weight of the i-th real target, n represents the number of real targets, A1 represents the false alarm rate score of the algorithm to be evaluated, and A 2i represents the correct search score of the i-th true target, A 3i represents the search time score of the i-th true target, A 4i represents the search efficiency score of the i-th real target, and A represents the target search score of the algorithm to be evaluated.

4. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 2 is characterized in that: The following method is used to determine whether there is a false alarm in the image sent back by the algorithm to be evaluated: For each bounding box in the image, calculate the intersection and union ratio of the bounding box with the bounding box of each real target in the image. If the intersection and union ratio of the bounding box with the bounding box of the real target is greater than the threshold, the bounding box is valid, otherwise, the bounding box is invalid; If the number of invalid bounding boxes in an image exceeds a preset threshold, there is a false alarm in the image.

5. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 2 is characterized in that: The target recognition score of the algorithm to be evaluated is obtained based on the target type and target state recognition results in the target recognition results returned by the algorithm to be evaluated, including: For each real target, the target type recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized type is the type of the real target in the target recognition results returned by the evaluation algorithm; For each real target, the target state recognition score of the real target is calculated according to the number of times the real target is correctly recognized and the recognized state is the state of the real target in the target recognition results returned by the evaluation algorithm; The target recognition score of the algorithm to be evaluated is obtained based on the target type recognition score and target state recognition score of each real target.

6. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 5 is characterized in that: The target recognition score of the algorithm to be evaluated is calculated using the following formula: Among them, α i represents the weight of the i-th true target, n represents the number of true targets, B 1i represents the target type recognition score of the i-th real target, B 2i represents the target state recognition score of the i-th real target, and B represents the target recognition score of the algorithm to be evaluated.

7. The method for evaluating the autonomous identification and analysis algorithm of reconnaissance payload targets according to claim 2 is characterized in that: The target stable tracking score of the algorithm to be evaluated is calculated based on the target identifier in the target recognition result returned by the algorithm to be evaluated, including: For each real target, obtain the image in which the real target is correctly identified in the target recognition result returned by the algorithm to be evaluated, and extract the target identifier of the identified target corresponding to the real target; obtain the target stability tracking score of the real target according to the number of target identifiers; The target stability tracking score of the algorithm to be evaluated is obtained based on the target stability tracking score of each real target.