Image recognition model evaluation methods, devices, and storage media

Abstract

This disclosure proposes an image recognition model evaluation method, apparatus, and storage medium. The method includes: determining a first confidence level of the image recognition model using multiple test images; determining a quality assessment value of the multiple test images when the first confidence level is less than a first threshold; calculating a first similarity between the multiple test images and multiple training images of the image recognition model when the quality assessment value is not less than a second threshold; and retraining the image recognition model when the first similarity is less than a third threshold. This method can sequentially evaluate the confidence level, image quality, and working environment of the image recognition model. When it is determined that the model performance is reduced due to environmental changes, the model is updated in a timely manner to ensure that the image recognition model can be applied to various scenarios, thereby improving the image recognition effect.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and more particularly to an image recognition model evaluation method, apparatus, and storage medium. Background Technology

[0002] With the continuous improvement of the level of intelligence in coal mines, imaging sensing devices are widely used in all aspects of coal production, and deep learning technology is applied to process the acquired image data. Compared with other vision application scenarios, the image acquisition environment of imaging devices in open-pit coal mines changes constantly as mining progresses. The input image data received by the deployed deep learning model may differ significantly from the training data. To ensure the recognition effect of the model, frequent updates and optimizations of the model are required. Currently, there is a lack of an automatic monitoring and evaluation method for the performance of video analysis models in open-pit coal mine scenarios. Summary of the Invention

[0003] This disclosure presents an image recognition model evaluation method, apparatus, and storage medium, aiming to at least partially solve one of the technical problems in the related art.

[0004] The first aspect of this disclosure proposes an image recognition model evaluation method, comprising: determining a first confidence level of the image recognition model using multiple test images; determining a quality assessment value of the multiple test images if the first confidence level is less than a first threshold; calculating a first similarity between the multiple test images and multiple training images of the image recognition model if the quality assessment value is not less than a second threshold; and retraining the image recognition model if the first similarity is less than a third threshold.

[0005] A second aspect of this disclosure provides an image recognition model evaluation apparatus, comprising: a first calculation module for determining a first confidence level of the image recognition model using multiple test images; a second calculation module for determining a quality assessment value of the multiple test images when the first confidence level is less than a first threshold; a third calculation module for calculating a first similarity between the multiple test images and multiple training images of the image recognition model when the quality assessment value is not less than a second threshold; and a training module for retraining the image recognition model when the first similarity is less than a third threshold.

[0006] A third aspect of this disclosure provides a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image recognition model evaluation method of the embodiments of this disclosure.

[0007] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the image recognition model evaluation method disclosed in the embodiments of this disclosure.

[0008] In this embodiment, a first confidence level of the image recognition model is determined using multiple test images. If the first confidence level is less than a first threshold, a quality assessment value of the multiple test images is determined. If the quality assessment value is not less than a second threshold, a first similarity between the multiple test images and multiple training images of the image recognition model is calculated. If the first similarity is less than a third threshold, the image recognition model is retrained. This allows for the sequential evaluation of the image recognition model's confidence level, image quality, and working environment. When it is determined that the model's performance has decreased due to environmental changes, the model is updated in a timely manner, ensuring that the image recognition model can be applied to various scenarios, thereby improving the image recognition effect.

[0009] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0010] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 This is a schematic flowchart of an image recognition model evaluation method provided according to an embodiment of the present disclosure;

[0012] Figure 2 This is a flowchart illustrating an image recognition model evaluation method according to another embodiment of the present disclosure;

[0013] Figure 3 This is a schematic flowchart of calculating the first similarity according to an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram of an image recognition model evaluation apparatus according to another embodiment of the present disclosure;

[0015] Figure 5 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0017] It should be noted that the execution subject of the image recognition model evaluation method in this embodiment can be an image recognition model evaluation device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0018] Figure 1 This is a flowchart illustrating an image recognition model evaluation method according to an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes:

[0019] S101: Determine the first confidence level of the image recognition model using multiple test images.

[0020] In this embodiment, the image recognition model is a trained deep learning model that can be applied to any scenario for image recognition.

[0021] In a specific application scenario, with the continuous improvement of the level of intelligence in coal mines, imaging sensing devices are widely used in various stages of coal production. For example, the image recognition model in this embodiment is applied to the open-pit coal mine scenario. It can be configured at the image acquisition equipment in the open-pit coal mine to collect and recognize images of various stages of the coal mine. That is to say, the image recognition model evaluation method in this embodiment can evaluate the image recognition model deployed in the open-pit coal mine.

[0022] The images used to evaluate the image recognition model can be referred to as test images. This embodiment can provide a test dataset, which may include multiple test images, for example, m test images constituting the test dataset. In some embodiments, the test images may be images captured in real time by an image acquisition device, and this is not limited.

[0023] Confidence score, as an important criterion for evaluating the performance of deep learning models, is correlated with the model's accuracy. When the prediction confidence score of a model generally decreases, it can usually be considered that the model's accuracy has declined. In this embodiment, the confidence score of the image recognition model can be referred to as the first confidence score, which can be determined using multiple test images.

[0024] To implement the image recognition model evaluation method, this embodiment also provides an open-pit coal mine video analysis system. This system may include, for example, a computing center (e.g., a server) and edge computing devices, with the edge computing devices connected to the image acquisition equipment. In some embodiments, the image recognition model evaluation method can be deployed and executed at the computing center, or transmitted to the edge computing device via transmission protocols such as RTSP or national standards. That is, it is deployed on the edge computing device, where analysis and calculation are completed, and the evaluation results are output. Only the evaluation results, key frames, or key video data need to be sent back; the original video data can be stored locally or discarded, reducing the data transmission and processing pressure on the network and servers.

[0025] S102: If the first confidence level is less than the first threshold, determine the quality assessment value of multiple test images.

[0026] After determining the first confidence level of the image recognition model as described above, this embodiment can further compare the first confidence level with a first threshold. If the first confidence level is less than the first threshold, it indicates that the model's confidence level has decreased, and the model's performance can be considered to have changed. In this case, this embodiment further determines the cause of the change in model performance. The first threshold can be flexibly set according to the actual application scenario and is not limited thereto.

[0027] In practical applications, there are various reasons that can cause changes in the performance of image recognition models. For example, the unique nature of open-pit coal mines means that dust generated during mining may temporarily affect the quality of acquired images, or the lens of the image acquisition equipment may be obscured by mud or dust, leading to abnormal image quality and thus affecting the performance of the image recognition model. Another example is that as open-pit coal mining progresses, the environmental information of the acquired images changes significantly, such as changes in lighting, shadows, and background compared to the model's training images—that is, changes in the acquisition scene—which can affect the performance of the image recognition model. Alternatively, any other possible reason can cause changes in the performance of the image recognition model, and there are no restrictions on this.

[0028] In this embodiment, after determining that the first confidence level is less than the first threshold, it can first determine whether the image imaging abnormality is caused by dust or mud covering the lens of the image acquisition device, resulting in a change in model performance. Specifically, this embodiment can calculate the quality assessment values ​​of multiple test images to evaluate the image quality of the multiple test images, thereby determining whether the image recognition model performance is affected by dust or mud covering. Any algorithm can be used to evaluate the image quality, and there are no limitations on this.

[0029] Understandably, if the first confidence level is not less than the first threshold, it means that the model performance has not degraded. In this case, the subsequent operations of this method will not be performed.

[0030] S103: If the quality assessment value is not less than the second threshold, calculate the first similarity between multiple test images and multiple training images of the image recognition model.

[0031] After determining the quality assessment value, this embodiment can further compare the quality assessment value with a second threshold. If the quality assessment value is not less than the second threshold, it indicates that the quality of the test image acquired by the image acquisition device is normal, and the influence of dust or mud on the performance of the image recognition model can be ruled out. The second threshold can be flexibly set according to the actual application scenario and is not limited thereto.

[0032] In this context, this embodiment can further determine whether the performance of the image recognition model changes due to a change in the image acquisition scene (environment). Specifically, this embodiment can calculate the similarity between multiple test images and multiple training images, which can be referred to as the first similarity. The first similarity can be calculated in any manner, without limitation.

[0033] In this embodiment, the images used during the training of the image recognition model are referred to as training images. This embodiment can provide a training dataset, which may include multiple training images. Each training image may correspond to a specific acquisition scene. If the acquisition scene changes, the scene of the test image will differ from that of the training images, thus altering the model's performance. Therefore, this embodiment can calculate the similarity (first similarity) between the test image and the training images. Based on the first similarity, it can be determined whether the image acquisition scene has changed.

[0034] Understandably, if the quality assessment value is less than the second threshold, it indicates that the model performance may have changed due to dust or mud covering. In this case, manual investigation is required, and the subsequent operations of this method should not be performed.

[0035] S104: If the first similarity is less than the third threshold, retrain the image recognition model.

[0036] After determining the first similarity, this embodiment can further compare the first similarity with a third threshold. If the first similarity is less than the third threshold, it can be considered that a change in the image acquisition scene has caused a change in the performance of the image recognition model. In this case, this embodiment retrains the image recognition model, for example, by retraining the model using test images, so that the model can adapt to the new acquisition scene.

[0037] In this embodiment, a first confidence level of the image recognition model is determined using multiple test images. If the first confidence level is less than a first threshold, a quality assessment value of the multiple test images is determined. If the quality assessment value is not less than a second threshold, a first similarity between the multiple test images and multiple training images of the image recognition model is calculated. If the first similarity is less than a third threshold, the image recognition model is retrained. This allows for the sequential evaluation of the image recognition model's confidence level, image quality, and working environment. When it is determined that the model's performance has decreased due to environmental changes, the model is updated in a timely manner, ensuring that the image recognition model can be applied to various scenarios, thereby improving the image recognition effect.

[0038] Figure 2 This is a flowchart illustrating an image recognition model evaluation method according to another embodiment of the present disclosure, such as... Figure 2 As shown, the method includes:

[0039] S201: Predict the second confidence level of the image recognition model using each test image.

[0040] In determining the first confidence level of an image recognition model, this embodiment first predicts the second confidence level of the image recognition model using each test image, which can be used... This indicates that multiple test images can yield multiple second confidence levels.

[0041] In some embodiments, the second confidence level can be calculated according to the following formula:

[0042]

[0043]

[0044] in, σ represents the second confidence level. SM () represents the softmax operator of the image recognition model, z i This represents the logical vector for each test image, i.e., the input to the softmax operator, where K is the number of categories recognized by the image recognition model.

[0045] S202: Calculate the average of multiple second confidence levels corresponding to multiple test images, and use it as the first confidence level.

[0046] In other words, this embodiment can calculate multiple second confidence levels. The average value is used as the first confidence level, which can be used as... The formula for calculating the first confidence level is as follows:

[0047]

[0048] in, represents the average value of the second confidence level, and m represents the number of test images.

[0049] Therefore, this embodiment can calculate the average confidence level. Compared with the first threshold (lower confidence limit p) l ) compare, if Below the lower confidence limit p l This indicates a decrease in model performance. Therefore, this embodiment can calculate the average confidence level as the basis for judging changes in model performance, thereby improving the accuracy of the judgment.

[0050] S203: If the first confidence level is less than the first threshold, calculate the peak signal-to-noise ratio of each test image to each training image.

[0051] In determining the quality assessment values ​​of multiple test images, this embodiment of the present disclosure can employ the method of calculating the Peak-Signal to Noise Ratio (PSNR) to calculate the degree of difference in image quality between the current test dataset and the dataset used for model training, thereby determining the impact of dust or mud occlusion on model performance.

[0052] Specifically, in this embodiment, the peak signal-to-noise ratio (PSNR) of each test image to each training image can be calculated first; that is, the PSNR of all test images to training images is calculated one by one. It should be noted that the training images and test images in this embodiment have the same resolution.

[0053] In some embodiments, the peak signal-to-noise ratio (PSNR) for each test and training image can be calculated using the following formula:

[0054]

[0055] Where PSNR represents Peak Signal-to-Noise Ratio, and x(i,j) represents the pixel value of the pixel at coordinates (i,j) in each test image. Let x represent the pixel value of the pixel at coordinate (i, j) in each training image, M and N be the number of rows and columns of pixels in the test and training images, respectively, where 0 ≤ i ≤ M-1, 0 ≤ j ≤ N-1, and x max x represents the maximum pixel value. max =2 K -1, K represents the number of binary bits per pixel. In practical applications, K can be 8, then x max =255.

[0056] The peak signal-to-noise ratio (PSNR) is calculated for each test image and each training image, thus multiple PSNRs can be obtained from multiple training images and multiple test images.

[0057] S204: Calculate the average of multiple peak signal-to-noise ratios (PSNRs) of multiple test images and multiple training images as a quality assessment value.

[0058] In other words, this embodiment calculates the average of multiple peak signal-to-noise ratios (PSNRs) as the quality assessment value. Therefore, this embodiment uses PSNR to determine image quality and calculates the average of multiple PSNRs as the quality assessment value, thus improving the accuracy of image quality assessment.

[0059] S205: If the quality assessment value is not less than the second threshold, hash the multiple test images to obtain multiple first hash sequence labels, and hash the multiple training images to obtain multiple second hash sequence labels.

[0060] Specifically, Figure 3 This is a flowchart illustrating the calculation of the first similarity according to an embodiment of this disclosure, as shown below. Figure 3 As shown, in the operation of calculating the first similarity between multiple test images and multiple training images of the image recognition model in this embodiment, the test dataset images (i.e., multiple test images) can be subjected to perceptual feature extraction, namely: perceptual hashing (PHash), to obtain the perceptual hash sequence of each test image, which can be called the first hash sequence label, and a perceptual hash sequence database is established to store the first hash sequence label; similarly, this embodiment can obtain the perceptual hash sequence of each training image, which can be called the second hash sequence label.

[0061] S206: Calculate multiple Hamming distances between each first hash sequence label and multiple second hash sequence labels, and calculate the average of the partial Hamming distances among the multiple Hamming distances as the second similarity between each test image and multiple training images.

[0062] Furthermore, in this embodiment, the Hamming distance between the first hash sequence label of each test image and the second hash sequence labels of multiple training images can be calculated one by one. This Hamming distance can represent the similarity between the test image and the training images, and multiple Hamming distances can be obtained by comparing with the second hash sequence labels of multiple training images. Further, in this embodiment, the average of the partial Hamming distances of each test image can be calculated as the second similarity between the test image and the multiple training images. In some embodiments, for example, the multiple Hamming distances can be sorted in descending order, and the average of the top 5% of the sorted Hamming distances can be calculated.

[0063] S207: Calculate the average of multiple second similarities corresponding to multiple test images, and use it as the first similarity.

[0064] After calculating multiple second similarities for multiple test images as described above, this embodiment can further calculate the average of the multiple second similarities as a first similarity, which can reflect the degree of similarity between the test dataset and the training dataset.

[0065] Therefore, this embodiment can use a perceptual hash algorithm to calculate the similarity between the test image and the training image, which can accurately determine whether the acquisition scene has changed.

[0066] S208: If the first similarity is less than the third threshold, retrain the image recognition model.

[0067] For a detailed description of S208, please refer to the above embodiments, which will not be repeated here.

[0068] In this embodiment, a first confidence level of the image recognition model is determined using multiple test images. If the first confidence level is less than a first threshold, a quality assessment value is determined for each test image. If the quality assessment value is not less than a second threshold, a first similarity is calculated between the test images and multiple training images of the image recognition model. If the first similarity is less than a third threshold, the image recognition model is retrained. This allows for the sequential evaluation of the image recognition model's confidence level, image quality, and working environment. When environmental changes cause a decrease in model performance, the model is updated promptly, ensuring its applicability to various scenarios and improving image recognition performance. Furthermore, this embodiment can calculate the average confidence level as a basis for assessing model performance changes, improving the accuracy of the judgment. Additionally, this embodiment uses peak signal-to-noise ratio (PSNR) to determine image quality and calculates the average of multiple PSNR values ​​as the quality assessment value, thus improving the accuracy of image quality assessment. Moreover, this embodiment can use a perceptual hashing algorithm to calculate the similarity between test images and training images, accurately determining whether the acquisition scene has changed.

[0069] Figure 4 This is a schematic diagram of an image recognition model evaluation apparatus provided according to another embodiment of the present disclosure. Figure 4 As shown, the image recognition model evaluation device 40 includes:

[0070] The first calculation module 401 is used to determine the first confidence level of the image recognition model using multiple test images;

[0071] The second calculation module 402 is used to determine the quality assessment values ​​of multiple test images when the first confidence level is less than a first threshold; and

[0072] The third calculation module 403 is used to calculate the first similarity between multiple test images and multiple training images of the image recognition model, provided that the quality assessment value is not less than a second threshold; and

[0073] Training module 404 is used to retrain the image recognition model if the first similarity is less than the third threshold.

[0074] In some embodiments, the first calculation module 401 is specifically used to: predict the second confidence level of the image recognition model using each test image; and calculate the average of the multiple second confidence levels corresponding to multiple test images as the first confidence level.

[0075] In some embodiments, the second confidence level is calculated according to the following formula:

[0076]

[0077]

[0078] in, σ represents the second confidence level. SM () represents the softmax operator of the image recognition model, z i This represents the logical vector for each test image, where K is the number of categories recognized by the image recognition model.

[0079] Calculate the average of multiple second confidence levels using the following formula:

[0080]

[0081] in, represents the average value of the second confidence level, and m represents the number of test images.

[0082] In some embodiments, the second calculation module 402 is specifically used to: calculate the peak signal-to-noise ratio (PSNR) of each test image and each training image; and calculate the average of the PNR of multiple test images and multiple training images as a quality evaluation value.

[0083] In some embodiments, the peak signal-to-noise ratio is calculated according to the following formula:

[0084]

[0085] Where PSNR represents Peak Signal-to-Noise Ratio, and x(i,j) represents the pixel value of the pixel at coordinates (i,j) in each test image. Let x represent the pixel value of the pixel at coordinate (i, j) in each training image, M and N be the number of rows and columns of pixels in the test and training images, respectively, where 0 ≤ i ≤ M-1, 0 ≤ j ≤ N-1, and x max x represents the maximum pixel value. max =2 K -1, K represents the number of binary bits per pixel.

[0086] In some embodiments, the third calculation module 403 is specifically used for: hashing multiple test images to obtain multiple first hash sequence labels, hashing multiple training images to obtain multiple second hash sequence labels; calculating multiple Hamming distances between each first hash sequence label and multiple second hash sequence labels respectively, and calculating the average of some Hamming distances among the multiple Hamming distances as the second similarity between each test image and multiple training images; and calculating the average of the multiple second similarities corresponding to multiple test images as the first similarity.

[0087] In this embodiment, a first confidence level of the image recognition model is determined using multiple test images. If the first confidence level is less than a first threshold, a quality assessment value of the multiple test images is determined. If the quality assessment value is not less than a second threshold, a first similarity between the multiple test images and multiple training images of the image recognition model is calculated. If the first similarity is less than a third threshold, the image recognition model is retrained. This allows for the sequential evaluation of the image recognition model's confidence level, image quality, and working environment. When it is determined that the model's performance has decreased due to environmental changes, the model is updated in a timely manner, ensuring that the image recognition model can be applied to various scenarios, thereby improving the image recognition effect.

[0088] According to embodiments of this disclosure, this disclosure also provides a computer device, a readable storage medium, and a computer program product.

[0089] To implement the above embodiments, this disclosure also proposes a computer program product that, when executed by an instruction processor, performs the image recognition model evaluation method as proposed in the foregoing embodiments of this disclosure.

[0090] Figure 5 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 5 The computer device 12 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0091] like Figure 5 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0092] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0093] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0094] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive".

[0095] although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0096] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0097] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0098] The processing unit 16 executes various functional applications by running programs stored in the system memory 28, such as implementing the image recognition model evaluation method mentioned in the foregoing embodiments.

[0099] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0100] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0101] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0102] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0103] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0105] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0106] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] Although embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for evaluating an image recognition model, characterized in that, include: The first confidence level of the image recognition model is determined using multiple test images; If the first confidence level is less than the first threshold, determine the quality assessment value of the plurality of test images; If the quality assessment value is not less than the second threshold, calculate the first similarity between the plurality of test images and the plurality of training images of the image recognition model; as well as If the first similarity is less than the third threshold, the image recognition model is retrained.

2. The method as described in claim 1, characterized in that, The step of determining the first confidence level of the image recognition model using multiple test images includes: Predict the second confidence level of the image recognition model using each test image; and The average of the multiple second confidence levels corresponding to the multiple test images is calculated as the first confidence level.

3. The method as described in claim 2, characterized in that, The second confidence level is calculated using the following formula: in, σ represents the second confidence level. SM ( ) represents the softmax operator of the image recognition model, z i K represents the logical vector of each test image, where K is the number of categories recognized by the image recognition model. The average of the multiple second confidence levels is calculated using the following formula: in, denoted as the average value of the second confidence level, and m represents the number of test images.

4. The method as described in claim 1, characterized in that, Determining the quality assessment value of the plurality of test images includes: Calculate the peak signal-to-noise ratio (PSNR) of each test image to each training image; and The average of the peak signal-to-noise ratios of the multiple test images and the multiple training images is calculated as the quality evaluation value.

5. The method as described in claim 4, characterized in that, in, The peak signal-to-noise ratio is calculated using the following formula: Wherein, PSNR represents the peak signal-to-noise ratio, and x(i,j) represents the pixel value of the pixel at coordinates (i,j) in each test image. Let x represent the pixel value of the pixel at coordinates (i, j) in each training image, M and N be the number of rows and columns of pixels in the test image and the training image, respectively, 0 ≤ i ≤ M-1, 0 ≤ j ≤ N-1, and x. max x represents the maximum pixel value. max =2 K -1, K represents the number of binary bits per pixel.

6. The method as described in claim 1, characterized in that, The calculation of the first similarity between the plurality of test images and the plurality of training images of the image recognition model includes: The multiple test images are hashed to obtain multiple first hash sequence labels, and the multiple training images are hashed to obtain multiple second hash sequence labels; Calculate multiple Hamming distances between each first hash sequence label and the multiple second hash sequence labels, and calculate the average of the partial Hamming distances among the multiple Hamming distances as the second similarity between each test image and the multiple training images; The average of the multiple second similarities corresponding to the multiple test images is calculated as the first similarity.

7. An image recognition model evaluation device, characterized in that, include: The first calculation module is used to determine the first confidence level of the image recognition model using multiple test images; The second calculation module is used to determine the quality assessment value of the plurality of test images when the first confidence level is less than the first threshold. The third calculation module is used to calculate the first similarity between the plurality of test images and the plurality of training images of the image recognition model when the quality assessment value is not less than the second threshold. as well as The training module is used to retrain the image recognition model if the first similarity is less than a third threshold.

8. The apparatus as claimed in claim 7, characterized in that, The first calculation module is specifically used for: Predict the second confidence level of the image recognition model using each test image; and The average of the multiple second confidence levels corresponding to the multiple test images is calculated as the first confidence level.

9. The apparatus as claimed in claim 7, characterized in that, The third calculation module is specifically used for: The multiple test images are hashed to obtain multiple first hash sequence labels, and the multiple training images are hashed to obtain multiple second hash sequence labels; Calculate multiple Hamming distances between each first hash sequence label and the multiple second hash sequence labels, and calculate the average of the partial Hamming distances among the multiple Hamming distances as the second similarity between each test image and the multiple training images; The average of the multiple second similarities corresponding to the multiple test images is calculated as the first similarity.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images