Method and apparatus for evaluating inference capability of visual deep learning model

Abstract

The application relates to a visual deep learning model inference ability evaluation method and device, wherein the method comprises the following steps: presetting impossible database node in a preset normal database set, and changing a query result list of a pedestrian re-identification model; judging whether the changed current query result list meets preset effective conditions; if the preset effective conditions are met, the effectiveness of the current query result list within a preset time is based on statistical data, and the inference ability of the pedestrian re-identification model to the current scene is evaluated according to the statistical data, so as to evaluate and prompt the application system. Therefore, when the pedestrian re-identification model is deployed in a changed scene, the model inference ability is reduced, the upper application system cannot update the model according to the change of the inference ability of the pedestrian re-identification model, and the precision and reliability of the pedestrian identification function are greatly affected.

Description

Technical Field

[0001] This application relates to the field of pedestrian re-identification technology, and in particular to a method and apparatus for evaluating the reasoning ability of a visual deep learning model. Background Technology

[0002] With the widespread adoption of intelligent security hardware, a large number of surveillance devices generate massive amounts of video footage. It is anticipated that computer vision technology can be used to intelligently analyze this video footage and extract valuable content. Since pedestrians are the most critical target in surveillance videos, many intelligent video analysis systems consider pedestrian recognition a crucial task, significantly aiding in early warning of key personnel and tracking of missing persons. Pedestrian recognition methods are expected to identify individuals appearing in a given image for subsequent business logic analysis.

[0003] Typically, pedestrian detection is the first step in video analysis and processing. The application system uses a pedestrian object detection model to extract pedestrian bounding boxes, crops a portion of the image based on the location information of the bounding boxes, and uses this as the query image. This query image is then input into a pedestrian re-identification model. Through the forward propagation process of the pedestrian re-identification model, features are extracted from the query image, and the highest similarity match is found in a known image database to determine the identity of the pedestrian in the current query image. Current mainstream pedestrian recognition methods treat this process as an image retrieval task, that is, outputting a list of database images sorted by similarity for the query image, and determining the pedestrian's identity based on the sorting results. The working mechanism of the pedestrian re-identification model is as follows: Figure 1 As shown, its basic process is as follows:

[0004] 1. Assuming the current pedestrian re-identification model is M, the pedestrian identification task in the current process, given the query sample q, i In this case, it is possible to find the matching result with the smallest distance in a normal base set G by calculating the feature distance extracted by the pedestrian re-identification model;

[0005] 2. After the pedestrian re-identification model is deployed to the application system, feature extraction is performed on all base database images in batches to obtain g for each base database image. i Feature vectors:

[0006]

[0007] The aforementioned feature vectors are saved to a database or memory, awaiting feature distance calculation during subsequent queries. When the query image qi is received, the person re-identification model can extract the features of the query sample to obtain the feature vectors.

[0008]

[0009] Where, q i This indicates that the size of the input query image will be uniformly scaled to the standard size accepted by the model during the pedestrian re-identification process;

[0010] 3. Once the feature vectors are calculated, sample features can be queried using cosine similarity calculation. Features of various base database images Distance:

[0011]

[0012] In practice, matrix operations are typically used to calculate the distances between the query sample and all base database images in parallel. Then, the base database images are sorted based on these distances to obtain an ordered list.

[0013] L(M(q i ), M(G))={(g i1 g i2 , ..., g ik )|g ij ∈G, j=1,2,...,k}

[0014] However, the current L(M(q) i M(G)) contains only all the base database content relative to the query image q. i It provides partial order information, but cannot determine from it whether the current base database content's sorting result correctly reflects the queried image q. i The degree of matching.

[0015] Therefore, current re-identification models typically only focus on the accuracy of query results without considering the reasoning ability of the re-identification model. In actual deployment scenarios, the reasoning ability of the re-identification model may drop sharply due to changes in the scenario, which may lead to feature extraction failure due to the performance degradation of the pedestrian re-identification model. This makes it impossible for the retrieval results of the pedestrian re-identification model to objectively reflect the similarity between the base list and the query image, resulting in bizarre identity matching results.

[0016] Currently, related technologies can determine the validity of a query by judging the distance between the first matching result and the query sample and a fixed threshold. For example, if the distance between the query sample and the first matching result is less than 0.5, the query is considered valid; otherwise, the query is considered invalid. In addition, related technologies can also use the first few bits of information in the matching result to perform feature fusion and then perform the query again through query expansion, obtaining the final result through multiple queries.

[0017] However, traditional methods for determining the threshold of the first matching result are quite sensitive to the selection of the threshold. Furthermore, when the model's estimation ability declines, it cannot truly and objectively reflect the model's reasoning ability for the current scenario, and may also affect the correct query results in normal scenarios. In addition, the relevant technology can only have a gain effect when the first few pieces of information in the query results are relatively reasonable. Otherwise, it may further reduce the reliability of the query results under the influence of incorrect matching results, affecting the user experience. These issues urgently need to be addressed. Summary of the Invention

[0018] This application provides a method and apparatus for evaluating the reasoning ability of a visual deep learning model, in order to solve the problems that the reasoning ability of a pedestrian re-identification model decreases when the deployment scene changes, and the upper-layer application system cannot update the model according to the changes in the reasoning ability of the pedestrian re-identification model, which greatly affects the accuracy and reliability of the pedestrian recognition function.

[0019] The first aspect of this application provides a method for evaluating the reasoning ability of a visual deep learning model, comprising the following steps: presetting impossible base database nodes in a preset normal base database set, modifying the query result list of a pedestrian re-identification model; determining whether the modified current query result list meets preset validity conditions, and if it meets the preset validity conditions, then based on statistical data of the validity of the current result list within a preset time period, evaluating the reasoning ability of the pedestrian re-identification model for the current scene based on the statistical data, so as to provide evaluation prompts for the application system.

[0020] Optionally, in one embodiment of this application, the step of presetting impossible database nodes in a preset normal database set and modifying the query result list of the pedestrian re-identification model includes: constructing an impossible database set that meets preset requirements; setting a maximum person identity tag number that meets a first preset condition based on the number of person identities in the impossible database set and the preset normal database set; assigning person identity tag numbers to pedestrian images in the impossible database set based on the maximum person identity tag number; determining whether the pedestrian image belongs to the impossible database node based on the relationship between the person identity tag number corresponding to the pedestrian image and the maximum person identity tag number; if the pedestrian image belongs to the impossible database node, then mixing the normal database set and the impossible database set to obtain an expanded database set; and in the process of pedestrian identity recognition query, replacing the normal database set with the expanded database set to perform feature extraction and distance calculation operations to obtain a query result list.

[0021] Optionally, in one embodiment of this application, determining whether the modified current query result list meets the preset valid conditions includes: extracting the first p base database nodes in the query result list, and determining whether there is a matching anomaly in the query process based on the number of impossible base database nodes in the first p base database nodes, where p is a positive integer; extracting the first k base database nodes in the query result list, and determining whether there is a distance anomaly in the query process based on the average distance between the impossible base database nodes in the first k base database nodes and the query sample, where k is a positive integer; and determining that the current query is invalid if the current query has a matching anomaly or a distance anomaly based on the determination results of the matching anomaly and the distance anomaly.

[0022] Optionally, in one embodiment of this application, the step of evaluating the pedestrian re-identification model's reasoning ability for the current scene based on statistical data of the validity of the current result list within a preset time period if the preset validity condition is met includes: defining a set of all query images within the preset time period, and obtaining an invalid subset of the set of all query images, wherein the invalid subset contains all query images that are invalid within the preset time period; calculating the query inefficiency rate of the application system based on the set of all query images within the preset time period and the invalid subset, so as to evaluate the pedestrian re-identification model's reasoning ability for the current scene.

[0023] Optionally, in one embodiment of this application, the formula for calculating query inefficiency is:

[0024]

[0025] Where t1 is the query start time and t2 is the query end time. This represents the number of elements in the set of all queried images within the time period t1-t2. The number of elements in the invalid subset.

[0026] A second aspect of this application provides a device for evaluating the reasoning ability of a visual deep learning model, comprising: a modification module for modifying the query result list of a pedestrian re-identification model by pre-setting impossible base database nodes in a preset normal base database set; a judgment module for judging whether the modified current query result list meets preset validity conditions; and an evaluation module for evaluating the pedestrian re-identification model's reasoning ability for the current scene based on statistical data of the validity of the current result list within a preset time period if the preset validity conditions are met, so as to provide evaluation prompts for the application system.

[0027] Optionally, in one embodiment of this application, the modification module includes: a construction unit, configured to construct an impossible database set that meets preset requirements; a setting unit, configured to set a maximum person identity tag number that meets a first preset condition based on the number of person identities in the impossible database set and a preset normal database set; an assignment unit, configured to assign a person identity tag number to pedestrian images in the impossible database set based on the maximum person identity tag number; a determination unit, configured to determine whether a pedestrian image belongs to the impossible database node based on the relationship between the person identity tag number corresponding to the pedestrian image and the maximum person identity tag number; and an acquisition unit, configured to, if the pedestrian image belongs to the impossible database node, mix the normal database set with the impossible database set to obtain an expanded database set, and in the process of pedestrian identity recognition query, replace the normal database set with the expanded database set to perform feature extraction and distance calculation operations to obtain a query result list.

[0028] Optionally, in one embodiment of this application, the judgment module includes: a first extraction unit, configured to extract the first p base database nodes in the query result list, and determine whether there is a matching anomaly in the query process based on the number of impossible base database nodes in the first p base database nodes, wherein p is a positive integer; a second extraction unit, configured to extract the first k base database nodes in the query result list, and determine whether there is a distance anomaly in the query process based on the average distance between the impossible base database nodes in the first k base database nodes and the query sample, wherein k is a positive integer; and an anomaly unit, configured to determine that the query is invalid if the current query has a matching anomaly or a distance anomaly based on the judgment results of the matching anomaly and the distance anomaly.

[0029] Optionally, in one embodiment of this application, the evaluation module includes: a definition unit, configured to define a set of all query images within the preset time period and obtain an invalid subset of the set of all query images, wherein the invalid subset includes all query images that are invalid within the preset time period; and a calculation unit, configured to calculate the query inefficiency of the application system based on the set of all query images within the preset time period and the invalid subset, so as to evaluate the reasoning ability of the pedestrian re-identification model for the current scene.

[0030] Optionally, in one embodiment of this application, the formula for calculating query inefficiency is:

[0031]

[0032] Where t1 is the query start time and t2 is the query end time. This represents the number of elements in the set of all queried images within the time period t1-t2. The number of elements in the invalid subset.

[0033] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the visual deep learning model inference capability evaluation method as described in the above embodiments.

[0034] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for evaluating the inference capabilities of a visual deep learning model.

[0035] Therefore, the embodiments of this application have the following beneficial effects:

[0036] This application embodiment can modify the query result list of the pedestrian re-identification model by pre-setting impossible base database nodes in a preset normal base database set; it then determines whether the modified query result list meets preset validity conditions. If it does, it uses statistical data on the validity of the query result list within a preset time period to evaluate the pedestrian re-identification model's inference ability in the current scenario, providing an evaluation prompt to the application system. This application evaluates the pedestrian re-identification model's inference ability in the current scenario to determine the reliability of the model's current inference ability and the timing for iterative upgrades in the upper-layer application system. This does not require modification of model parameters, exhibits strong scenario adaptability, and significantly improves the stability and reliability of the pedestrian re-identification model during cross-domain deployment. Therefore, it solves the problems of decreased inference ability of the pedestrian re-identification model when the deployment scenario changes, and the inability of the upper-layer application system to update the model according to changes in the inference ability, which greatly affects the accuracy and reliability of pedestrian recognition functions.

[0037] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0038] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0039] Figure 1 A schematic diagram of the pedestrian re-identification model workflow;

[0040] Figure 2 This is a flowchart of a method for evaluating the reasoning ability of a visual deep learning model according to an embodiment of this application;

[0041] Figure 3 A schematic diagram illustrating the mixing and identification of impossible and normal warehouses, provided as an embodiment of this application;

[0042] Figure 4 An execution logic architecture diagram of a method for evaluating the reasoning ability of a visual deep learning model is provided as an embodiment of this application;

[0043] Figure 5 A schematic diagram illustrating the judgment of matching anomalies during a query process, as provided in one embodiment of this application;

[0044] Figure 6 A schematic diagram illustrating the timing of model updates based on query inefficiency and prior knowledge of the actual scenario, provided as an embodiment of this application;

[0045] Figure 7 This is an example diagram of a visual deep learning model reasoning ability evaluation device according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0047] Among them, 10-Visual deep learning model reasoning ability evaluation device, 100-Modification module, 200-Judgment module, 300-Evaluation module, 801-Memory, 802-Processor, 803-Communication interface. Detailed Implementation

[0048] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0049] The following describes a method and apparatus for evaluating the inference ability of a visual deep learning model according to embodiments of this application, with reference to the accompanying drawings. As mentioned in the background art, in practical applications, cross-domain deployed pedestrian re-identification models may experience a decline in feature extraction capabilities, resulting in inaccurate similarity calculations and difficulty in ensuring the reliability of ranking results. However, the pedestrian re-identification model is unlikely to perceive this phenomenon, and the upper-layer application system cannot constantly monitor the inference ability of the pedestrian re-identification model. Consequently, after a period of operation, the business system may easily provide incorrect matching results based on the model with ineffective inference capabilities, greatly affecting the user experience. This application provides a method for evaluating the inference ability of a visual deep learning model. In this method, the result list during the pedestrian re-identification model query process can be modified by pre-setting impossible base database nodes in the base database set. Furthermore, by analyzing the position and distance information of the pre-set virtual base database nodes in the query result list, the validity of the current query is determined. Based on statistical data of the validity of query results over a period of time, the model's inference ability for the current scene is evaluated, and relevant prompts are provided to the application system. This application evaluates the reasoning ability of a pedestrian re-identification model in the current scenario to determine the reliability of the model's current reasoning capability and the appropriate timing for iterative upgrades in the upper-layer application system. It does not require modification of model parameters, exhibits strong scenario adaptability, and avoids online model updates, effectively preventing attacks on the model. When the pre-set database size is comparable to the original database size, it effectively controls the model's computational complexity and inference time, significantly improving the stability and reliability of the pedestrian re-identification model during cross-domain deployment. Therefore, it solves the problems of declining inference ability of pedestrian re-identification models when deployment scenarios change, and the inability of upper-layer application systems to update the model according to changes in inference ability, which greatly affects the accuracy and reliability of pedestrian recognition functions.

[0050] Specifically, Figure 2 This is a flowchart illustrating a method for evaluating the reasoning ability of a visual deep learning model, as provided in an embodiment of this application.

[0051] like Figure 2 As shown, the method for evaluating the reasoning ability of this visual deep learning model includes the following steps:

[0052] In step S201, impossible base database nodes are preset in the preset normal base database set, and the query result list of the pedestrian re-identification model is modified.

[0053] Those skilled in the art will understand that, generally speaking, when the result with the highest similarity to the query image is retrieved from the base database list by the pedestrian recognition method, the first matching base database with the smallest distance can usually be selected as the matching result, and the identity information of the person in the matching base database image can be determined as the pedestrian identity information of the query image. However, the embodiments of this application can mine the implicit properties of the query results by pre-setting impossible base database nodes during the above retrieval process, thereby providing a basis for judging the reliability of subsequent queries.

[0054] Optionally, in one embodiment of this application, impossible database nodes are pre-set in a preset normal database set, and the query result list of the pedestrian re-identification model is modified, including: constructing an impossible database set that meets preset requirements; setting the maximum person identity tag number that meets the first preset condition based on the number of person identities in the impossible database set and the preset normal database set; assigning person identity tag numbers to pedestrian images in the impossible database set based on the maximum person identity tag number; determining whether a pedestrian image belongs to an impossible database node based on the relationship between the person identity tag number corresponding to the pedestrian image and the maximum person identity tag number; if the pedestrian image belongs to an impossible database node, then the normal database set and the impossible database set are mixed to obtain an expanded database set, and during the pedestrian identity recognition query process, the expanded database set replaces the normal database set to perform feature extraction and distance calculation operations to obtain a query result list.

[0055] It should be noted that the embodiments of this application can find a base database G that has no overlap with the person identities of the current pedestrian re-identification model in the current working scenario. im In other words, for any input query sample image, its identity information will not match G. im The personnel information in the two pictures is the same, among which G im G satisfies the following equation:

[0056]

[0057] Since most pedestrian re-identification models use clothing as an important basis for matching features, in the embodiments of this application, G im It is possible to construct the database using data from outside the domain. For example, in domestic pedestrian re-identification scenarios, images of pedestrians wearing ancient costumes from period dramas can be used as the base image database. Furthermore, due to G... im It is impossible to match the identity of any real person, G im Pedestrian images generated by computer graphics rendering can also be used as the base library.

[0058] When G imAfter construction, embodiments of this application can set a maximum person identity tag number N based on the number of person identities N in the original base database set G. out , making N out >N; and numbered N based on the identity of the most prominent person. out As a boundary, give G im The pedestrian images in the image are assigned a value greater than N. out The value is the identity tag number. Therefore, in subsequent processing, the identity tag number of the person corresponding to the image can be used as a reference to N. out Based on the relationship, determine whether it belongs to the pre-set impossible bottom-base node according to the following formula:

[0059]

[0060] Among them, Virtual(g i ) is the bottom library g i Is it a pre-defined impossible base node, ID(g) i ) is the base image library g i Personnel identification number.

[0061] It should be noted that during actual implementation, technicians can use the base image database (g) i The personnel identification information is directly identified in the file name or recorded through a hash table.

[0062] Furthermore, embodiments of this application can combine the original base set G with the impossible base set G. im Mix them to obtain an expanded base library set G. combine The process of merging the pre-set impossible base bank set with the normal base bank set is as follows: Figure 3 As shown, G is used in the pedestrian identification retrieval process. combine It replaces G in performing the original feature extraction and distance calculation operations and obtains the retrieval results.

[0063] Therefore, the embodiments of this application expand the normal database by pre-setting impossible database nodes, thereby generating a hybrid database, providing reliable data basis for judging the validity of subsequent queries.

[0064] In step S202, it is determined whether the modified current query result list meets the preset valid conditions.

[0065] By pre-setting impossible base database nodes in the base database set, and after modifying the result list during the pedestrian re-identification model query process, the embodiments of this application can further determine the validity of the current query by analyzing the location and distance information of the pre-set virtual base database nodes in the query result list. Figure 4 As shown, the reliability of the first matching result of the query is described and analyzed.

[0066] Optionally, in one embodiment of this application, determining whether the modified current query result list meets the preset valid conditions includes: extracting the first p base database nodes from the query result list, and determining whether there is a matching anomaly in the query process based on the number of impossible base database nodes in the first p base database nodes, where p is a positive integer; extracting the first k base database nodes from the query result list, and determining whether there is a distance anomaly in the query process based on the average distance between the impossible base database nodes in the first k base database nodes and the query sample, where k is a positive integer; based on the determination results of matching anomaly and distance anomaly, if the current query has a matching anomaly or a distance anomaly, then the current query is determined to be invalid.

[0067] It should be noted that in each query process, the pedestrian re-identification model will perform feature extraction in the original manner, calculate the distance information between the query sample features and the features stored in the hybrid database, and return the result list according to the rule of sorting the distance from smallest to largest. By observing the position and distance information of the preset impossible database nodes in the query results, the validity of this query can be comprehensively judged.

[0068] Specifically, based on the basic idea of ​​metric learning, it is assumed that the model has a strong ability to distinguish features extracted from pedestrian images, which can ensure that the distance between images of people with the same identity is as small as possible and the distance between images of people with different identities is as large as possible.

[0069] Therefore, embodiments of this application can utilize the triplet loss function to constrain the relationships between sampled training data during the training process of the pedestrian re-identification model:

[0070] triplelet_loss=max(dist(anchor,p)-dist(anchor,n)+margin,0)

[0071] In the formula, anchor is a randomly selected anchor image, p represents another image with the same person identity as the anchor image, n represents another image with a different person identity than the anchor image, and margin represents the distance interval threshold. The meaning of the above triplet loss function is that the distance between any image and other images with the same identity is closer than the distance between any image and other images with different identities, and the distance difference is at least margin; otherwise, a loss value will be generated.

[0072] Understandably, for an ideal pedestrian re-identification model, since the identity information of any person in any image cannot be matched with it, the distance must be very large. After inputting real pedestrian images in the scene, impossible base database nodes should not appear at the top of the search results. If a pre-set impossible base database node appears at the top of the search results, then the model's reasoning ability may have problems.

[0073] For the expanded base warehouse G combine Each query will yield a list of search results, and the first p base nodes from the search results will be analyzed. This application's embodiment defines a Match... Error To determine if a matching error exists during the query process, i.e., if at least q impossible base database nodes appear in the first p base database nodes, as shown in the following formula, the matching error determination logic is as follows: Figure 5 As shown:

[0074]

[0075] To avoid the influence of some abnormal samples on the results, embodiments of this application can also analyze the first k results of the search results, and determine whether there is a distance from the average distance between the preset impossible base database nodes and the query samples.

[0076] Specifically, embodiments of this application may set Dist R Let Dist be the average distance between the normal base database nodes and the query sample in the first k results of the search. I To determine the Dist value, we first calculate the average distance between the first k search results and the query sample, considering the pre-defined impossible base node. R With Dist I Does it meet the following conditions:

[0077]

[0078] Since the distance between the query sample and the impossible base database node sample should be larger than the distance between the query sample and the normal base database sample, this distance can be measured using a margin. In practice, the margin parameter value here is usually based on the hyperparameter of the triplet loss function selected during model training. Therefore, if Dist R With Dist I If the above formula is satisfied, it can be determined that there is a distance anomaly during the query process.

[0079] The embodiments of this application combine the above-mentioned matching anomaly and distance anomaly cases. If a matching anomaly occurs in the current query, the solution is to... Error =True, or a distance anomaly (Dist) occurs. ErrorIf the query is True, then the query is considered invalid, i.e., Valid(q) i If ) = False, then Valid(q) i The calculation method for ) is as follows:

[0080] Valid(q i ) = ! Match error (q i And! Dist error (q i )

[0081] It is understood that, in the embodiments of this application, there are two main possibilities that could lead to an invalid query:

[0082] 1. If the person whose image belongs to the input query is not in the base database sample, a matching anomaly may be triggered.

[0083] Second, the model's reasoning ability in the current scenario decreases, resulting in a weakened feature extraction capability and a reduced reliability in calculating the distance between the query image and the base database images, thereby triggering matching anomalies.

[0084] In the specific implementation process, the embodiments of this application need to add the calculation of the distance between the query image and the virtual base database in the above analysis process. In addition, after the distance calculation is completed, the length of the sorted data is longer than the original length, which increases the time complexity. However, the embodiments of this application will use a preset impossible base database size equal to the original base database size to ensure that the additional computing overhead will not increase exponentially.

[0085] Therefore, embodiments of this application determine whether the first matching result in the base database should be selected as the identity recognition result. Only when the query is valid is the identity information of the first matching result used as the pedestrian re-identification result. Based on this metric for determining the validity of the current query, the pedestrian re-identification model can not only retrieve the most similar matching result to the query image from the base database images, but also determine whether the matching result satisfies identity matching. When the matching result is insufficient to prove identity matching, the query is deemed invalid, effectively ensuring the subsequent evaluation of the pedestrian re-identification model's inference capabilities.

[0086] In step S203, if the preset valid conditions are met, the statistical data of the validity of the current result list is queried based on the preset time, and the pedestrian re-identification model's reasoning ability for the current scene is evaluated based on the statistical data, so as to provide evaluation prompts for the application system.

[0087] Furthermore, embodiments of this application can also evaluate the model's perception capability of the current scenario based on the validity statistics of multiple queries within a specific time range, thereby providing important feedback information to the upper-layer application system.

[0088] Optionally, in one embodiment of this application, if a preset validity condition is met, the pedestrian re-identification model is evaluated for its reasoning ability in the current scene based on statistical data of the validity of the current result list within a preset time period. This includes: defining a set of all queried images within a preset time period, and obtaining an invalid subset of the set of all queried images, wherein the invalid subset contains all queried images that are invalid within the preset time period; and calculating the query inefficiency of the application system based on the set of all queried images within the preset time period and the invalid subset, in order to evaluate the pedestrian re-identification model's reasoning ability in the current scene.

[0089] It should be noted that the embodiments of this application can determine the validity of a query by utilizing the information mined from the query results of the expanded base database, and the application system can record data such as the images, returned results, and query validity of each query.

[0090] Therefore, in the embodiments of this application, Q(t1, t2) can be defined as the set of all queried images within the time range from t1 to t2:

[0091] Q(t1, t2)={∪q i |time(q i )~(t1,t2]}

[0092] The total number of queries from time t1 to t2 is:

[0093] Count Q(t1，t2) =||Q(t1, t2)||

[0094] Based on information such as query validity, embodiments of this application can find a subset Q of Q(t1, t2). invalid (t1, t2), which contains elements that are invalid query images, i.e.:

[0095] Q invalid (t1, t2) = {q i |q i ∈Q(t1, t2)and Valid(q) i )=False}

[0096] Similarly, in the embodiments of this application, the total number of invalid queries from time t1 to t2 can be defined as:

[0097]

[0098] Therefore, the embodiments of this application can obtain the proportion of invalid queries to all queries within a certain period of time, i.e., query inefficiency.

[0099] Therefore, embodiments of this application can determine the model's reasoning ability in the current scenario based on the validity statistics of multiple queries over a period of time, and provide feedback information to the upper-layer application system, thereby enabling the application system to perform appropriate model update operations in a timely manner when the deployment scenario changes.

[0100] Optionally, in one embodiment of this application, the formula for calculating query inefficiency is:

[0101]

[0102] Where t1 is the query start time and t2 is the query end time. This represents the number of elements in the set of all queried images within the time period t1-t2. The number of elements in the invalid subset.

[0103] It should be noted that, in the embodiments of this application, the proportion of queries with invalid results within the aforementioned period of time to all queries, i.e., the formula for calculating query inefficiency, is as follows:

[0104]

[0105] Where t1 is the query start time and t2 is the query end time. This represents the number of elements in the set of all queried images within the time period t1-t2. The number of elements in the invalid subset.

[0106] When query inefficiency is too high, the first matching content of most query results in the current scenario cannot be used as a basis for determining the identity of the person in the query image. There are two main reasons that may lead to high query inefficiency:

[0107] 1. The pedestrians appearing in the query image are indeed not within the normal population database. This causes images in both the normal population database and the pre-set impossible population database to have too large a feature distance from the query image, resulting in matching failures in the query result ranking list. Error =True;

[0108] Second, changes in the environmental factors of the reasoning scenario cause problems with the model's feature extraction capabilities, rendering the features extracted from the query images ineffective. The distance between the image features and the images of different individuals in the base database no longer satisfies the identity constraint.

[0109] As one possible approach, embodiments of this application can exclude the first scenario that leads to excessively high query inefficiency by:

[0110] Since pedestrian re-identification models are typically used to identify people and track their movements, they have some prior knowledge about when key personnel will appear. Based on the temporal and spatial relationships, it can be determined whether personnel from the base station should be present. For example, in cross-camera tracking applications, if it is known that a person X from the base station appears under camera A at time t0, it is very likely that they will appear under a camera geographically adjacent to camera A near time t0. If the query efficiency under the relevant cameras within the time range (t0-T, t0+T) is still very high, it indicates that the result is not due to the presence of non-base station personnel.

[0111] Furthermore, based on the periodicity of events, if there are people in the database appearing in a fixed time period t1 to t2 every day, but suddenly a surge in query inefficiency is found on a certain day, this contradicts the prior knowledge expectation of the actual application situation, and is likely due to a decrease in the model's reasoning ability caused by changes in the scenario.

[0112] Therefore, embodiments of this application, by combining query inefficiency with prior knowledge of specific business scenarios, can determine whether the model's reasoning ability has indeed declined, i.e., whether the scenario has changed, and thus update the model accordingly. Figure 6 As shown.

[0113] In summary, the embodiments of this application can utilize the relevant properties of the query results of pre-set impossible base database nodes during the operation of the pedestrian re-identification model to determine the validity of the current query, and perform inefficiency statistics on queries over a period of time. Combined with the actual business situation, the causes of excessively high query inefficiency can be determined. If the situation of excessively high query inefficiency caused by the presence of non-base database personnel can be ruled out based on prior knowledge, it can be determined that the current scenario has changed, resulting in a decrease in the model's reasoning ability. In this way, the model can be updated in a timely manner or other emergency mechanisms can be used. Thus, the embodiments of this application enable the pedestrian re-identification model to have the ability to perceive the scene and to have a certain prediction of its own reasoning ability. When the model deployment scenario changes and the model's reasoning ability decreases, the upper-layer application system can update the model in a timely manner to ensure the effectiveness of the pedestrian identification function in the business scenario.

[0114] The following will illustrate the performance of the proposed visual deep learning model reasoning ability evaluation method through experimental analysis.

[0115] In the specific experiment, this application selected three models and two different datasets, named D1 and D2 respectively, and the three pre-trained models named M1, M2 and M3 respectively. It is known that the three pre-trained models have different performance on D1 and D2. M1 performs best on the D1 dataset, but has only a small inference ability on the D2 dataset. M2 performs best on the D2 dataset, but its inference ability on the D1 dataset is weaker than that of M1. M3 performs poorly on both datasets.

[0116] In the experiment, this application used images from the virtual synthetic pedestrian dataset TAGPerson as the source of impossible base database nodes. After expanding the base database, the query results were analyzed to determine the overall invalid query rate. The relevant experimental results are shown in Table 1.

[0117] Table 1. Experimental Results on the Number of Invalid Queries

[0118]

[0119] As shown in Table 1, for the same dataset, the model with better actual performance (higher first-order matching accuracy) also has a lower invalid query rate. That is, the invalid query rate is negatively correlated with the model's reasoning ability. Therefore, in the same dataset, the invalid query rate can be used to evaluate the model's reasoning ability to a certain extent, and the model with the strongest reasoning ability can be selected for application.

[0120] Furthermore, for the same model, the performance on different datasets is negatively correlated with the invalid query rate. For example, model M1 performs much better than D2 on dataset D1, and its query invalidity rate on dataset D1 is also lower than that on dataset D2. Therefore, as the reasoning scenario changes, the change in the model's reasoning ability can be roughly determined based on the query invalidity rate.

[0121] According to the visual deep learning model inference capability evaluation method proposed in this application, this application can modify the result list during the pedestrian re-identification model query process by pre-setting impossible base database nodes in the base database set. Furthermore, by analyzing the position and distance information of the pre-set virtual base database nodes in the query result list, it determines whether the current query is valid. Based on the statistical data of query result validity over a period of time, it evaluates the model's inference capability for the current scene and provides relevant prompts to the application system. This application evaluates the inference capability of the pedestrian re-identification model in the current scene to determine the reliability of the model's current inference capability and the timing for the upper-layer application system to iterate and upgrade the model. It does not require modification of model parameters, has strong scene adaptability, and avoids online model update processes, effectively preventing attacks on the model. When the size of the pre-set impossible base database is comparable to the original base database size, it effectively controls the model's computational complexity and inference time, greatly improving the stability and reliability of the pedestrian re-identification model during cross-domain deployment.

[0122] Next, with reference to the accompanying drawings, a visual deep learning model reasoning ability evaluation device proposed according to an embodiment of this application is described.

[0123] Figure 7 This is a block diagram of a visual deep learning model reasoning ability evaluation device according to an embodiment of this application.

[0124] like Figure 7 As shown, the visual deep learning model reasoning ability evaluation device 10 includes: a modification module 100, a judgment module 200, and an evaluation module 300.

[0125] The modification module 100 is used to pre-set impossible base database nodes in the preset normal base database set and modify the query result list of the pedestrian re-identification model.

[0126] The judgment module 200 is used to determine whether the modified current query result list meets the preset valid conditions.

[0127] The evaluation module 300 is used to query the statistical data of the validity of the current result list within a preset time if the preset valid conditions are met, and evaluate the reasoning ability of the pedestrian re-identification model for the current scene based on the statistical data, so as to provide evaluation prompts for the application system.

[0128] Optionally, in one embodiment of this application, the modification module 100 includes: a construction unit, a setting unit, an assigning unit, a determination unit, and an acquisition unit.

[0129] Among them, the construction unit is used to construct an impossible base library set that meets preset requirements.

[0130] The setting unit is used to set the maximum person identity tag number that meets the first preset condition, based on the number of person identities in the impossible bottom set and the preset normal bottom set.

[0131] The unit is used to assign a person identification tag number to pedestrian images in the impossible base set based on the maximum person identification tag number.

[0132] The determination unit is used to determine whether a pedestrian image belongs to an impossible bottom-database node based on the relationship between the person identity tag number corresponding to the pedestrian image and the maximum person identity tag number.

[0133] The acquisition unit is used to mix the normal base set with the impossible base set if the pedestrian image belongs to the impossible base set, so as to obtain an expanded base set. During the pedestrian identification query process, the expanded base set replaces the normal base set to perform feature extraction and distance calculation operations to obtain a query result list.

[0134] Optionally, in one embodiment of this application, the determination module 200 includes: a first extraction unit, a second extraction unit, and an anomaly unit.

[0135] The first extraction unit is used to extract the first p base database nodes in the query result list, and to determine whether there is a matching anomaly in the query process based on the number of impossible base database nodes in the first p base database nodes, where p is a positive integer.

[0136] The second extraction unit is used to extract the first k base database nodes in the query result list. Based on the average distance between the impossible base database nodes in the first k base database nodes and the query sample, it is used to determine whether there is a distance anomaly in the query process, where k is a positive integer.

[0137] The exception unit is used to determine the result based on the matching exception and the distance exception. If the current query has a matching exception or a distance exception, the current query is determined to be invalid.

[0138] Optionally, in one embodiment of this application, the evaluation module includes a definition unit and a calculation unit.

[0139] The definition unit is used to define a set of all query images within a preset time period and to obtain an invalid subset of the set of all query images, wherein the invalid subset contains all query images that are invalid within the preset time period.

[0140] The calculation unit is used to calculate the query inefficiency of the application system based on the set of all query images and invalid subsets within a preset time period, so as to evaluate the reasoning ability of the pedestrian re-identification model for the current scene.

[0141] Optionally, in one embodiment of this application, the formula for calculating query inefficiency is:

[0142]

[0143] Where t1 is the query start time and t2 is the query end time. This represents the number of elements in the set of all queried images within the time period t1-t2. The number of elements in the invalid subset.

[0144] It should be noted that the foregoing explanation of the embodiment of the visual deep learning model reasoning ability evaluation method also applies to the visual deep learning model reasoning ability evaluation device of this embodiment, and will not be repeated here.

[0145] According to the visual deep learning model inference capability evaluation device proposed in this application, this application can modify the result list during the person re-identification model query process by pre-setting impossible base database nodes in the base database set. Furthermore, by analyzing the position and distance information of the pre-set virtual base database nodes in the query result list, it determines whether the current query is valid. Based on the statistical data of the validity of the query results over a period of time, it evaluates the model's inference capability for the current scene and provides relevant prompts to the application system. This application evaluates the inference capability of the person re-identification model in the current scene to determine the reliability of the model's current inference capability and the timing for the upper-layer application system to iterate and upgrade the model. It does not require modification of model parameters, has strong scene adaptability, and avoids online model update processes, effectively preventing attacks on the model. When the size of the pre-set impossible base database is comparable to the original base database size, it effectively controls the model's computational complexity and inference time, greatly improving the stability and reliability of the person re-identification model during cross-domain deployment.

[0146] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0147] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0148] When the processor 802 executes the program, it implements the visual deep learning model inference capability evaluation method provided in the above embodiments.

[0149] Furthermore, electronic devices also include:

[0150] Communication interface 803 is used for communication between memory 801 and processor 802.

[0151] The memory 801 is used to store computer programs that can run on the processor 802.

[0152] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0153] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0154] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0155] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0156] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for evaluating the inference capability of a visual deep learning model.

[0157] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0158] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0159] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0161] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0162] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0164] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for evaluating the reasoning ability of a visual deep learning model, characterized in that, Includes the following steps: Pre-set impossible base database nodes in the preset normal base database set, and modify the query result list of the pedestrian re-identification model; Determine whether the modified query result list meets the preset valid conditions, and If the preset valid conditions are met, the statistical data of the validity of the current query result list is queried within a preset time, and the reasoning ability of the pedestrian re-identification model for the current scene is evaluated based on the statistical data, so as to provide evaluation prompts for the application system. The step of presetting impossible base database nodes in the preset normal base database set and modifying the query result list of the pedestrian re-identification model includes: Construct an impossible set of base libraries that meets the preset requirements; Based on the number of personnel identities in the impossible bottom set and the preset normal bottom set, set the maximum personnel identity tag number that satisfies the first preset condition. Based on the maximum person identification tag number, assign a person identification tag number to the pedestrian images in the impossible base set; Based on the relationship between the person identity tag number corresponding to the pedestrian image and the largest person identity tag number, determine whether the pedestrian image belongs to the impossible bottom database node; If the pedestrian image belongs to the impossible base database node, the normal base database set is mixed with the impossible base database set to obtain an expanded base database set. During the pedestrian identification query process, the expanded base database set replaces the normal base database set to perform feature extraction and distance calculation operations to obtain a query result list. The determination of whether the modified current query result list meets the preset valid conditions includes: Extract the first few items from the query results list Each base library node, and based on the aforementioned... The number of impossible base database nodes in each base database node is used to determine whether there are matching anomalies in the query process, wherein the... It is a positive integer; Extract the first few items from the query results list k Each base database node, based on the previous k The average distance between the impossible base database nodes and the query sample in each base database node is used to determine whether there are distance anomalies in the query process, wherein the... k It is a positive integer; Based on the judgment results of the matching anomaly and distance anomaly, if the matching anomaly or distance anomaly occurs in the current query, the current query is determined to be invalid.

2. The method according to claim 1, characterized in that, If the preset validity condition is met, then based on the statistical data of the validity of the current query result list within a preset time period, and based on the statistical data, the reasoning ability of the pedestrian re-identification model for the current scene is evaluated, including: Define a set of all query images within the preset time period, and obtain an invalid subset of the set of all query images, wherein the invalid subset contains all query images that are invalid within the preset time period; Based on the set of all queried images within the preset time period and the invalid subset, the query inefficiency of the application system is calculated to evaluate the reasoning ability of the pedestrian re-identification model for the current scene.

3. The method according to claim 2, characterized in that, The formula for calculating the query inefficiency is: in, For the start time of the query, The query end time. for The number of elements in the set of all queried images within the specified time period. The number of elements in the invalid subset.

4. A device for evaluating the reasoning ability of a visual deep learning model, characterized in that, include: The modification module is used to pre-set impossible base database nodes in the preset normal base database set and modify the query result list of the pedestrian re-identification model; The judgment module is used to determine whether the modified list of results for the current query meets the preset valid conditions, and The evaluation module is used to query the statistical data of the validity of the current query result list within a preset time period if the preset valid conditions are met, and evaluate the reasoning ability of the pedestrian re-identification model for the current scene based on the statistical data, so as to provide evaluation prompts for the application system. The modification module includes: Construction unit, used to construct an impossible base library set that meets preset requirements; The setting unit is used to set the maximum person identity tag number that satisfies the first preset condition based on the number of person identities in the impossible bottom database set and the preset normal bottom database set. The assignment unit is used to assign a person identity tag number to the pedestrian images in the impossible base set based on the maximum person identity tag number; The determination unit is used to determine whether the pedestrian image belongs to the impossible bottom database node based on the relationship between the person identity tag number corresponding to the pedestrian image and the maximum person identity tag number; The acquisition unit is configured to, if the pedestrian image belongs to the impossible base database node, mix the normal base database set with the impossible base database set to obtain an expanded base database set, and in the process of pedestrian identification query, replace the normal base database set with the expanded base database set to perform feature extraction and distance calculation operations to obtain a query result list; The judgment module includes: The first extraction unit is used to extract the first few items from the query result list. Each base library node, and based on the aforementioned... The number of impossible base database nodes in each base database node is used to determine whether there are matching anomalies in the query process, wherein the... It is a positive integer; The second extraction unit is used to extract the first few items from the query result list. k Each base database node, based on the previous k The average distance between the impossible base database nodes and the query sample in each base database node is used to determine whether there are distance anomalies in the query process, wherein the... k It is a positive integer; An exception unit is used to determine that a query is invalid if either a matching exception or a distance exception occurs during the current query, based on the judgment results of the matching exception and the distance exception.

5. The apparatus according to claim 4, characterized in that, The evaluation module includes: A definition unit is used to define a set of all query images within the preset time period and obtain an invalid subset of the set of all query images, wherein the invalid subset contains all query images that are invalid within the preset time period; The calculation unit is used to calculate the query inefficiency of the application system based on the set of all query images within the preset time period and the invalid subset, so as to evaluate the reasoning ability of the pedestrian re-identification model for the current scene.

6. The apparatus according to claim 5, characterized in that, The formula for calculating the query inefficiency is: in, For the start time of the query, The query end time. for The number of elements in the set of all queried images within the specified time period. The number of elements in the invalid subset.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the visual deep learning model inference capability evaluation method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the visual deep learning model inference capability evaluation method as described in any one of claims 1-3.

Images