Model optimization method, model optimization device and electronic equipment

Abstract

The present disclosure provides a model optimization method, a model optimization device and an electronic device, which can be used in the field of artificial intelligence, finance or other fields. The method includes: sending a test request including a set identifier to a server side, so that the server side obtains the values ​​of static indicators of test images in a test image set corresponding to the set identifier; in response to a threshold input operation, sending indicator thresholds of at least part of the static indicators to the server side, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds; and in response to the test accuracy from the server side, displaying the test accuracy to determine the model optimization information.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology and finance, and more specifically, to a model optimization method, a model optimization device and an electronic device. Background Art

[0002] Artificial intelligence technology is increasingly being used in areas such as image recognition. Before a model is officially released online, it is necessary to test whether the model meets the online standards and iterate and optimize the model based on the test results.

[0003] In the process of implementing the concept of the present disclosure, the applicant found that there are at least the following problems in the related art: the model testing situation is relatively complex, involving many technical test indicators, and the technical test indicators are discrete and not strongly correlated. This makes it inconvenient for modelers to determine the problems of the model based on a single test indicator. Summary of the invention

[0004] In view of this, the present disclosure provides a model optimization method, a model optimization device and an electronic device, which dynamically combine static indicators to facilitate users to determine the impact of a single static indicator on the model recognition result by adjusting the value of the static indicator, so as to quickly determine the problems existing in the model.

[0005] One aspect of the present disclosure provides a model optimization method performed by a terminal, comprising: sending a test request including a set identifier to a server side, so that the server side obtains the values ​​of static indicators of test images in a test image set corresponding to the set identifier; in response to a threshold input operation, sending indicator thresholds of at least part of the static indicators to the server side, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds; and in response to the test accuracy from the server side, displaying the test accuracy, so as to determine the model optimization information.

[0006] One aspect of the present disclosure provides a model optimization method performed by a server, comprising: receiving a test request from a terminal, the test request including a set identifier of a test image set; using a trained model to process the test image set corresponding to the set identifier to obtain values ​​of static indicators for the test images in the test image set; in response to indicator thresholds of at least part of the static indicators from the terminal, determining a test accuracy for the test image set based on the values ​​of the static indicators and the indicator thresholds; and sending the test accuracy to the terminal to determine model optimization information.

[0007] One aspect of the present disclosure provides a model optimization device, which is arranged in a terminal and includes: a request sending module, an indicator threshold sending module and an accuracy display module. The request sending module is used to send a test request including a set identifier to a server side, so that the server side obtains the value of the static indicator of the test image in the test image set corresponding to the set identifier; the indicator threshold sending module is used to send the indicator threshold of at least part of the static indicator to the server side in response to the threshold input operation, so that the server side determines the test accuracy for the test image set based on the value of the static indicator of the test image and the indicator threshold; and the accuracy display module is used to display the test accuracy in response to the test accuracy from the server side, so as to determine the model optimization information.

[0008] One aspect of the present disclosure provides a model optimization device, which is arranged at a server end and includes: a request receiving module, an image processing module, an accuracy determination module and an accuracy sending module. The request receiving module is used to receive a test request from a terminal, and the test request includes a set identifier of a test image set; the image processing module is used to process the test image set corresponding to the set identifier using a trained model to obtain the value of the static index for the test image in the test image set; the accuracy determination module is used to respond to the index threshold of at least part of the static index from the terminal, and determine the test accuracy for the test image set based on the value of the static index and the index threshold; and the accuracy sending module is used to send the test accuracy to the terminal so as to determine the model optimization information.

[0009] Another aspect of the present disclosure provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the above method is implemented.

[0010] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above-mentioned training model optimization model training method and / or model optimization method.

[0011] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the above training model optimization model training method and / or model optimization method when executed.

[0012] The model optimization method, model optimization device and electronic device provided by the embodiments of the present disclosure enable users to adjust the indicator thresholds of static indicators online when conducting model testing, so that multiple static indicators can be dynamically linked, helping model builders or testers to quickly determine problems with the model based on the test accuracy linked to the indicator threshold.

[0013] The model optimization method, model optimization device and electronic device provided by the embodiments of the present disclosure, by setting some static indicators that are closely related to business indicators to be dynamically adjustable, facilitate business-related staff to adjust these static indicators, effectively lower the technical threshold for personnel to participate in testing, and facilitate business-related staff to give model optimization suggestions based on their own business experience, thereby improving the problem that models that rely solely on technical testing cannot meet the needs of specific scenarios.

[0014] The model optimization method, model optimization device, and electronic device provided by the embodiments of the present disclosure facilitate modelers or testers to determine the category to which the error image belongs by observing the error test image, which helps to assist in optimizing the training image set and effectively speeds up the model iteration speed. In addition, it is convenient to vertically compare the test results of multiple iteration versions and select a model suitable for the current scene.

[0015] The model optimization method, model optimization device and electronic device provided by the embodiments of the present disclosure can avoid invalid optimization of some rare categories by combining the experience of business-related staff. The selection of training images can also be more in line with the optimization direction of the model, closer to business indicators, accelerate the iteration of the model, optimize the iteration process, and save development time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenarios of the model optimization method, the model optimization device and the electronic device according to the embodiments of the present disclosure are schematically shown;

[0018] Figure 2 Schematically illustrates an exemplary system architecture to which a model optimization method and a model optimization device according to an embodiment of the present disclosure can be applied;

[0019] Figure 3 A flow chart of a model optimization method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 4 A schematic diagram schematically shows a test image according to an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of a linkage analysis interface of confidence and intersection-over-union ratio according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 6 A schematic diagram schematically shows the test accuracy for a test image set according to an embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of identifying an erroneous image according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 A flow chart of a model optimization method according to another embodiment of the present disclosure is schematically shown;

[0025] Fig. 9 The following schematically shows a logic diagram of a model optimization method according to an embodiment of the present disclosure;

[0026] Fig.10 A block diagram of a model optimization device according to an embodiment of the present disclosure is schematically shown;

[0027] Fig.11 A block diagram schematically shows a model optimization device according to another embodiment of the present disclosure; and

[0028] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0033] In enterprise applications, the model needs to be tested before it is officially released. The model testing is relatively complex, and the test indicators involve multi-category recognition effects and the position of the target positioning box. There are more than ten common test indicators. The applicant found the following problems.

[0034] On the one hand, the various technical test indicators are discrete and not strongly correlated. For example, indicators such as loss value, confusion matrix, intersection-over-union ratio, etc. make it difficult for modelers to directly locate problems in the model based on a single indicator.

[0035] On the one hand, modelers may lack relevant scenario experience and cannot propose model improvement plans in a timely manner. In enterprise applications, methods based entirely on technical testing are not intuitive enough to fit business indicators, and cannot fully integrate the expertise of personnel in various professional fields, which is not conducive to model analysis and optimization.

[0036] On the one hand, purely static model test results are not convenient for obtaining comprehensive test results for the model. There is a lack of methods for dynamic real-time linkage adjustment after parameter adjustment, which makes it difficult to implement iterative testing of technical indicators and business indicators and to present the model effects in a three-dimensional and comprehensive manner.

[0037] The embodiments of the present disclosure provide a model optimization method, a model optimization device, and an electronic device. The model optimization method executed by the terminal includes an instruction sending process and an information display process. First, a test request including a set identifier is sent to the server side so that the server side obtains the value of the static indicator of the test image in the test image set corresponding to the set identifier. Then, in response to the threshold input operation, the indicator threshold of at least part of the static indicator is sent to the server side so that the server side determines the test accuracy for the test image set based on the value of the static indicator of the test image and the indicator threshold. After completing the instruction sending process, the information display process is entered, and the test accuracy is displayed in response to the test accuracy from the server side to determine the model optimization information.

[0038] Figure 1The application scenarios of the model optimization method, model optimization device and electronic device according to the embodiments of the present disclosure are schematically illustrated.

[0039] like Figure 1 As shown, the user can set at least some of the static indicators on the terminal device, such as modifying the recognition result confidence, intersection-over-union ratio and / or confusion matrix values, so that the obtained model output results are changed, such as the test accuracy and the recognition error image subset. This makes it easier for the user and / or the terminal to determine the model optimization information based on the change of the model output result, and for the model developers to iteratively optimize the model.

[0040] Figure 2 The exemplary system architecture to which the model optimization method and the model optimization device according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 2 What is shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. It should be noted that the model optimization method, model optimization device and electronic device provided in the embodiments of the present disclosure can be used in the field of artificial intelligence in aspects related to model optimization, and can also be used in various fields other than the field of artificial intelligence, such as the financial field, etc. The application fields of the model optimization method, model optimization device and electronic device provided in the embodiments of the present disclosure are not limited.

[0041] like Figure 2 As shown, the system architecture 200 according to this embodiment may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 may include multiple gateways, routers, hubs, network cables, etc., which are used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0042] The user can use the terminal devices 201, 202, 203 to interact with other terminal devices and the server 205 through the network 204 to receive or send information, such as uploading a collection identifier, sending a request, uploading a test image, sending a model optimization instruction, receiving a recognition result, etc. The terminal devices 201, 202, 203 can be installed with various communication client applications, such as web browser applications, banking applications, e-commerce applications, search applications, office applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0043] The terminal devices 201, 202, and 203 include but are not limited to smart phones, desktop computers, augmented reality devices, tablet computers, laptop computers, and other electronic devices that can support functions such as web browsing. The terminal devices can download trained models to recognize images.

[0044] The server 205 may receive model training requests, model test requests, model optimization requests, image processing requests, model download requests, etc., and process the requests. For example, the server 205 may be a background management server, a server cluster, etc. The background management server may analyze and process the received model training requests, model optimization requests, image processing requests, etc., and feed back the processing results (such as recognition results, test results, model training results, etc.) to the terminal device.

[0045] It should be noted that the model optimization method provided in the embodiment of the present disclosure can be performed by the terminal device 201, 202, 203 or the server 205. Accordingly, the model optimization device provided in the embodiment of the present disclosure can be set in the terminal device 201, 202, 203 or the server 205. It should be understood that the number of terminal devices, networks and servers is only illustrative. According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0046] Figure 3 The flowchart of the model optimization method according to the embodiment of the present disclosure is schematically shown. Figure 3 As shown, the model optimization method may include operations S302 to S306.

[0047] In operation S302, a test request including a set identifier is sent to a server, so that the server obtains the value of a static indicator of a test image in a test image set corresponding to the set identifier.

[0048] In this embodiment, the test request may be generated based on text type information input by the user, or may be generated based on voice type information input by the user.

[0049] Specifically, the user can use the browser of the terminal to log in to the server side, so as to select the test image set to be used in the user interface displayed by the browser. The test image set can be determined based on the set identifier selected by the user.

[0050] For example, after the user sends a test request to the server through the terminal, the server can determine the corresponding test image set based on the set identifier included in the test request, and then input the test image in the test image set into the trained model (such as the image recognition model trained by the model developer) to obtain the image recognition result. Among them, the image recognition result may include intermediate results and final results, such as the intermediate results output by each layer of the neural network, the results output by each sub-network, and the results of the overall output of the neural network. Among them, the value of the static indicator can be the intermediate result of the model. The test accuracy can be the final result output by the model or the statistical result of the final result output by the overall model.

[0051] In one embodiment, the test image has annotation positioning frame information and object annotation information for the image within the annotation positioning frame.

[0052] Figure 4 A schematic diagram of a test image according to an embodiment of the present disclosure is schematically shown.

[0053] like Figure 4 As shown, the test image is an image of a contract, which includes multiple clauses, the contract signing date, and the identity information and certification information of both parties to the contract, such as the company seal, representative signature, etc. The test image has annotation information, which may include image identification, annotation positioning frame, object annotation information for the image in the annotation positioning frame, etc. The annotation information may be manually annotated, and the annotation information may be used as supervision information for the training model. Figure 4 It includes 4 positioning frames, and the object annotation information of the image in each positioning frame is Party A's seal, Party B's signature, legal representative and date of conclusion.

[0054] Accordingly, in the process of recognizing the test image by the trained image, it is necessary to determine the object annotation information of the image in the target area. The target area can be represented by a positioning frame, which is automatically determined by the trained model, and determines the text information or information type included in the image in the positioning frame. The positioning frame can be represented by coordinates, such as using the point coordinates of two diagonal points.

[0055] In a specific embodiment, the user selects parameters such as test data sets and containers according to the model to be tested, and the parameters such as containers can be dynamically bound through the front-end controls. The test data set can be a test image set, and the test images in the test image set have annotation information, such as image identification (ID), positioning frame information, and identification content. This information is stored in the database and input into the model together with the transmission of the test image set.

[0056] In operation S304, in response to the threshold input operation, the indicator thresholds of at least part of the static indicators are sent to the server side, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds.

[0057] In this embodiment, the user can set the indicator threshold of the static indicator on the interactive interface of the browser of the terminal and transmit it to the server side, so that the server side can determine the test result for the test image based on the indicator threshold. For example, the test result can be affected by modifying the confidence threshold of the recognition result. If the recognition result of test image 1 includes the seal of Party A, the confidence of the recognition result is 90%. If the confidence threshold is changed from 89% to 91%, the recognition result output by the model may change from the seal of Party A to the seal of non-Party A, which will affect the accuracy of the test.

[0058] In one embodiment, for a scenario in which a test image has an annotated positioning frame information and object annotation information for the image within the annotated positioning frame, static indicators may include: a recognition result confidence for the test image and an intersection-and-union ratio for the annotated positioning frame, wherein the intersection-and-union ratio of the annotated positioning frame represents the overlap ratio of the recognition positioning frame relative to the annotated positioning frame.

[0059] Figure 5 A schematic diagram of a linkage analysis interface of confidence and intersection-over-union ratio according to an embodiment of the present disclosure is schematically shown.

[0060] like Figure 5 As shown, the user can set the confidence level and / or the intersection-over-union ratio threshold in the interactive interface so that the confidence level and / or the intersection-over-union ratio threshold is sent to the server.

[0061] In operation S306 , in response to the test accuracy from the server, the test accuracy is displayed to determine model optimization information.

[0062] In this embodiment, the test accuracy can be calculated by the server side or by the terminal. The model optimization information can be determined by the user based on the confidence and / or intersection-over-union thresholds set by the user, and the test accuracy from the server side, such as based on expert experience. The model optimization information can be determined based on a preset rule or a preset mapping relationship, as well as the confidence and / or intersection-over-union thresholds set above, and the test accuracy from the server side.

[0063] For example, the test accuracy is the ratio between the number of test images in the subset of identified errors and the total number of test images in the test image set. Figure 5The interactive interface shown may also display a recognition error image. To facilitate the user to view the cause of the recognition error, a recognition positioning box, an annotation positioning box, object annotation information and object recognition information corresponding to the recognition error image may also be displayed.

[0064] Figure 6 A schematic diagram schematically shows the test accuracy for a test image set according to an embodiment of the present disclosure.

[0065] like Figure 6 As shown, the user used different test image sets (test1, test2 and test4) to detect the trained model, and the accuracy rates were 75%, 82% and 97% respectively.

[0066] In one embodiment, the model optimization information includes at least one of the type of training image set, the number of training images in the training image set, the number of network layers, model parameters, and the size of the training images. By feeding back the model optimization information to the server, it is convenient to quickly feed back the model optimization information to the model builder or model trainer so as to optimize the model.

[0067] In one embodiment, after displaying the test accuracy, the above method may further include the following operation: in response to the optimization information input operation, sending the model optimization information to the server. The user may send model optimization suggestions to the server through the terminal based on personal experience or business experience. For example, when the business personnel are viewing the subset of recognition error images, they find that the recognition error image belongs to a rare business scenario, and they may make the following suggestions: the error case belongs to a rare business scenario, and there is no need to spend a lot of time and computing resources to retrain the model, and prompt for manual entry for the rare business scenario. For another example, if the test images of multiple error cases belong to the same category, it indicates that the number of training images for this category is insufficient, and the number of training images needs to be increased. For example, displaying at least part of the subset of recognition error images includes: for each of the at least part of the recognition error images displayed, displaying the recognition error image and at least one of the following: the recognition positioning box, the annotation positioning box, the object annotation information and the object recognition information corresponding to the recognition error image.

[0068] In one embodiment, after sending the indicator thresholds of at least part of the static indicators to the server, the method may further include the following operation: receiving a subset of recognition error images for the test image set from the server.

[0069] Accordingly, in response to the test accuracy from the server side, displaying the test accuracy so as to determine the model optimization information may include the following operations: in response to the recognition error image subset and the test accuracy from the server side, displaying the indicator thresholds and the test accuracy of at least part of the static indicators in the recognition error image subset, so that the user can determine the model optimization information based on the recognition error image subset, the indicator thresholds and the test accuracy of the static indicators.

[0070] The disclosed embodiments make it easy to combine the experience of professional business personnel, avoid invalid optimization of some rare categories, and make the selection of training images more in line with the optimization direction of the model, closer to business indicators, accelerate the iteration of the model, optimize the iteration process, and save development time.

[0071] In order to improve the quality of the model optimization information provided by the user, the client can also display prompt information. For example, a preset mapping relationship can be displayed to guide the user.

[0072] In one embodiment, the above method may further include the following operations.

[0073] For example, a preset mapping relationship is displayed, where the preset mapping relationship includes a corresponding relationship between an indicator threshold of a static indicator and / or a test accuracy rate and model optimization information.

[0074] For another example, the model optimization information corresponding to the subset of recognized error images is determined based at least on a preset mapping relationship, an indicator threshold of a static indicator, and a test accuracy rate.

[0075] In a specific embodiment, if the confidence of the recognition result is less than a first threshold and the intersection-and-union ratio is less than a second preset threshold, the model optimization information includes at least one of the following: abnormal annotation information of the training image set, the number of training image sets is less than a preset image number threshold, the model is underfitting, and at least one of increasing the input image size.

[0076] If the confidence of the recognition result is less than the first threshold and the intersection-and-union ratio is greater than the second preset threshold, the model optimization information includes at least one of the following: the number of training image sets is less than the preset image number threshold, the number of model training rounds is increased, the number of pictures input for each round of training is reduced, the training images are sliced, and there are at least one of the images whose categories of training images overlap with other categories.

[0077] If the confidence of the recognition result is greater than the first threshold and the intersection-and-union ratio is less than the second preset threshold, the model optimization information includes at least one of the following: abnormal annotation information of the training image set, improving the quality of the training images of the training image set, and increasing the size of the input image.

[0078] If the types of at least some images in the subset of images with recognition errors are rare types in business applications, the model optimization information includes at least one of reducing or canceling the training image set belonging to the rare type and prompting manual processing. For example, when the business personnel are testing, they find that the recognition accuracy of certain types of test images is low. According to the previous business input experience, the proportion of this category in the entire scene is extremely low, so the data enhancement and iterative optimization of this category can be ignored. Every time this category is encountered, manual input is used. This situation greatly reduces the time for model optimization and improves the efficiency of modeling. In the selection of training sets, the amount of data for each category can be selected according to the previous input of business personnel. For some frequently appearing categories, provide more training image sets to improve recognition accuracy. For categories that appear less frequently, provide fewer training image sets.

[0079] In a specific embodiment, the static indicator also includes a confusion matrix. Accordingly, if the first proportion of the false negative class relative to the negative class is greater than the third threshold and / or the second proportion of the false positive class relative to the positive class is greater than the fourth threshold, the model optimization information includes at least one of the following: the number of training image sets is less than a preset image number threshold, the number of model training rounds is increased, the number of images input per round of training is reduced, the images are sliced, and there is at least one of images in which the category of the training image overlaps with other categories.

[0080] Due to the situation where the model training effect is not good, the training image set needs to be modified. For example, when the intersection-over-union ratio of a certain category is significantly low or not detected at all, it is necessary to add more training images of this category to the training image set to improve the model's feature extraction of this category and improve the recognition accuracy.

[0081] refer to Figure 5 The schematic diagram of the confidence and intersection analysis interface shown in Figure 1 shows that if the test result for a certain test image set is successful (refer to Figure 6 As shown, the status is successful), the user can click the action button to view the error case information. Figure 5 The two static indicators given in the interface select qualified images from the server-side database and transmit them to the terminal. The terminal plug-in is used to mark the positioning box, identify the positioning box, identify the content, and draw the labeled content. At the same time, the corresponding abnormal analysis and model optimization suggestions are given according to the current intersection-over-union (IOU), confidence parameter, category, and training image ratio.

[0082] For example, if both IOU and confidence are lower than the expected threshold, it indicates: 1) there is a problem with the annotation, 2) the quality and quantity of the data itself are not up to standard, 3) the model is underfitting, and the model parameters need to be adjusted, such as increasing the number of training epochs, adjusting the number of training images per training step, and increasing the image size of the input model. If the IOU meets expectations but the confidence is too low, it indicates: 1) the number of training images of this type is insufficient, and the model has not fully trained this type of training images, 2) it is because the model is underfitting this type of training images, you can appropriately increase the number of training epochs, reduce the number of images input per training step, and slice the images (ThreePartsCut, FourPatrsCut, etc.), 3) there are training images where the recognition category overlaps with other categories. If the confidence exceeds the threshold but the IOU is low, it indicates: 1) there is an obvious deviation in the data annotation, and the data needs to be re-annotated, 2) improve the quality of the training images and increase the image size of the input model. If the detection algorithm is a binary classification, the information of the confusion matrix can also be added. If the proportion of false negatives and false positives is too high, the optimization suggestions provided are higher than the intersection-over-union ratio, but the confidence level is the same.

[0083] Figure 7 The figure schematically shows a schematic diagram of identifying an erroneous image according to an embodiment of the present disclosure.

[0084] like Figure 7 As shown in the figure, the solid-line frame is the annotation positioning frame, and the dotted-line frame is the recognition positioning frame. It can be seen that the overlap of the two positioning frames is relatively high, but the recognition result is wrong. If the proportion of such error cases is relatively high relative to all error cases, optimization suggestions can be given based on the above mapping relationship: such as insufficient number of training images, increasing the number of training rounds, and the presence of training images with overlapping ID cards and seals. It is necessary to add more similar training images or use image technology to create more similar training images.

[0085] The disclosed embodiments change from previous static indicators to dynamic linkage of multiple indicators to help modelers more quickly understand the problems with the model. No matter it is a location problem or an identification problem, by observing error cases, it is possible to quickly determine which categories require incremental optimization, which greatly speeds up model iteration. At the same time, by vertically comparing the results of multiple iterative versions, it is possible to better select the model that best suits the current scenario.

[0086] Another aspect of the present disclosure provides a model optimization method performed by a server.

[0087] Figure 8 A flowchart of a model optimization method according to another embodiment of the present disclosure is schematically shown.

[0088] like Figure 8As shown, the model optimization method may include operations S802 to S808.

[0089] In operation S802, a test request is received from a terminal, where the test request includes a set identifier of a test image set.

[0090] In operation S804, the trained model is used to process the test image set corresponding to the set identifier to obtain the value of the static index for the test image in the test image set.

[0091] In addition, after the model processes the input test image set, it can also return information such as the positioning box coordinates, the number of positioning boxes, the recognized content, and the confidence level, and store the information returned by the model in the database corresponding to the test image.

[0092] In a specific embodiment, the recognition and positioning accuracy and intersection-over-union ratio information of the entire test image set are calculated, and the above information is respectively filled into the database to prepare data for error case display and model linkage analysis.

[0093] The intersection-and-union ratio is calculated based on the positioning information returned by the model, using the coordinates of the upper left vertex and the lower right vertex of the two positioning boxes. The calculation formula is shown in formula (1).

[0094]

[0095] Harea is the combined area of ​​the two positioning boxes, such as (x1, y1), (x2, y2) are the coordinates of the upper left vertices of the annotation positioning box and the recognition positioning box, and (x3, y3), (x4, y4) are the coordinates of the lower right vertices of the annotation positioning box and the recognition positioning box. The calculation formula of harea is shown in formula (2).

[0096] harea=(min(x2,x4)-Max(x1,x3))*(min(y2,y4)-Max(y1,y3)) Formula (2)

[0097] The intersection-and-union ratio can be calculated in the above manner. It should be noted that in this embodiment, the process of determining the intersection-and-union ratio is completed on the server side for illustration. However, the process can also be implemented on the terminal, for example, the server side sends the coordinates of the vertices of the identification positioning box and the coordinates of the vertices of the marking positioning box to the terminal so that the terminal calculates the intersection-and-union ratio.

[0098] The confidence of the recognition result is output by the model. The higher the confidence, the higher the probability that the recognition result is correct.

[0099] In operation S806, in response to the indicator threshold of at least part of the static indicator from the terminal, a test accuracy rate for the test image set is determined based on the value of the static indicator and the indicator threshold.

[0100] In this embodiment, the calculation formula of the test accuracy is as follows: for pictures with inconsistent numbers of positioning boxes and recognition positioning boxes or inconsistent recognition contents, an incorrect picture is identified and an incorrect tag is added. The calculation formula of the test accuracy is shown in formula (3).

[0101]

[0102] It should be noted that the operation of testing accuracy can also be performed by the terminal.

[0103] In operation S808, the test accuracy is sent to the terminal to determine model optimization information.

[0104] In one embodiment, after determining the test accuracy for the test image set, the above method may further include the following operations.

[0105] First, a subset of misrecognized images corresponding to the test accuracy is determined.

[0106] Then, the subset of misrecognition images is sent to the terminal so that the terminal displays the subset of misrecognition images.

[0107] For example, the test image has annotation positioning frame information and object annotation information for the image within the annotation positioning frame.

[0108] Correspondingly, the static indicators include: the confidence of the recognition result for the test image and the intersection-and-union ratio for the annotated positioning box, and the intersection-and-union ratio of the annotated positioning box represents the overlap ratio of the recognition positioning box relative to the annotated positioning box.

[0109] Fig. 9 A logic diagram of a model optimization method according to an embodiment of the present disclosure is schematically shown.

[0110] like Fig. 9 As shown, after the user selects a test image set at the terminal, the server processes the selected test image set and feeds back the processing results to the terminal. In addition, the server calculates the intersection-over-union ratio and the test accuracy based on the results returned by the model and the identification information of the test image set, and stores them in the database. The server feeds back the determined results to the terminal so that the terminal front desk displays the test accuracy. Through the above operations or through the above operations in conjunction with the experience of business personnel, at least part of the model optimization information can be given.

[0111] In addition, the terminal can also query error cases based on the confidence threshold and intersection-over-union ratio threshold set by the user, and display them to the user so that the user can determine the model optimization information based on the preset mapping relationship according to the intersection-over-union ratio, confidence and test accuracy information.

[0112] In addition, business personnel can also provide model optimization information for training image sets based on their own business experience, so as to optimize the training image sets used to train the model.

[0113] In this embodiment, taking the scenario in which the test image has the annotated positioning frame information and the object annotation information for the image within the annotated positioning frame as an example, the static indicators include: the recognition result confidence for the test image and the intersection-and-union ratio for the annotated positioning frame, and the intersection-and-union ratio of the annotated positioning frame represents the overlap ratio of the recognition positioning frame relative to the annotated positioning frame.

[0114] The disclosed embodiment dynamically integrates multiple static indicators such as IOU, confidence, confusion matrix and other static indicators, and adds a part for business personnel to review the model, which helps modeling personnel to locate problems in the model more quickly and speeds up the iterative optimization of the model. At the same time, it also makes the optimization direction of the model closer to business indicators, giving full play to the role of modeling personnel and business personnel.

[0115] Another aspect of the present disclosure provides a model optimization device, which is arranged in a terminal.

[0116] Fig.10 A block diagram of a model optimization device according to an embodiment of the present disclosure is schematically shown.

[0117] like Fig.10 As shown, the model optimization device 1000 may include: a request sending module 1010, an indicator threshold sending module 1020 and an accuracy display module 1030.

[0118] The request sending module 1010 is used to send a test request including a set identifier to the server side, so that the server side obtains the value of the static indicator of the test image in the test image set corresponding to the set identifier.

[0119] The indicator threshold sending module 1020 is used to send the indicator thresholds of at least some static indicators to the server side in response to the threshold input operation, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds.

[0120] The accuracy display module 1030 is used to display the test accuracy in response to the test accuracy from the server side, so as to determine the model optimization information.

[0121] The disclosed embodiment provides a testing system for a deep learning target detection algorithm based on web front-end technology, which realizes efficient testing of the model through collaboration between modeling personnel and business personnel, linkage adjustment of technical indicators, and tabular presentation of the accuracy of multiple test results.

[0122] Another aspect of the present disclosure provides a model optimization device, which is arranged on a server side.

[0123] Fig.11 A block diagram of a model optimization device according to another embodiment of the present disclosure is schematically shown.

[0124] like Fig.11 As shown, the device 1100 includes: a request receiving module 1110, an image processing module 1120, an accuracy determination module 1130 and an accuracy sending module 1140.

[0125] The request receiving module 1110 is used to receive a test request from a terminal, where the test request includes a set identifier of a test image set.

[0126] The image processing module 1120 is used to use the trained model to process the test image set corresponding to the set identifier to obtain the value of the static indicator of the test image in the test image set.

[0127] The accuracy determination module 1130 is used to determine the test accuracy for the test image set in response to the indicator threshold of at least part of the static indicators from the terminal, based on the value of the static indicators and the indicator threshold.

[0128] The accuracy sending module 1140 is used to send the test accuracy to the terminal so as to determine the model optimization information.

[0129] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and they will not be repeated here one by one.

[0130] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0131] For example, any multiple of the request sending module 1010, the index threshold sending module 1020, and the accuracy display module 1030 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the request sending module 1010, the index threshold sending module 1020, and the accuracy display module 1030 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the request sending module 1010, the indicator threshold sending module 1020 and the accuracy display module 1030 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0132] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0133] like Fig.12As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0134] In RAM 1203, various programs and data required for the operation of electronic device 1200 are stored. Processor 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1202 and / or RAM 1203. It should be noted that the program can also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0135] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the I / O interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage portion 1208 as needed.

[0136] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0137] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0138] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0139] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image model training method or model optimization method provided by the embodiment of the present disclosure.

[0140] When the computer program is executed by the processor 1201, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0142] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0143] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. These embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the various embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, all of which should fall within the scope of the present disclosure.

Claims

1. A model optimization method performed by a terminal, comprising: Sending a test request including a set identifier to a server side so that the server side obtains a value of a static indicator for a test image in a test image set corresponding to the set identifier, wherein the test image has an annotation positioning frame information and object annotation information for the image in the annotation positioning frame, and the static indicator includes: a recognition result confidence for the test image and an intersection-and-union ratio for the annotation positioning frame, wherein the intersection-and-union ratio of the annotation positioning frame represents an overlap ratio of the recognition positioning frame relative to the annotation positioning frame; In response to a threshold input operation, sending the indicator thresholds of at least part of the static indicators to the server side, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds; and In response to the test accuracy from the server, the test accuracy is displayed to determine model optimization information.

2. The method according to claim 1, further comprising: After sending the indicator thresholds of at least part of the static indicators to the server, receiving a subset of recognition error images for the test image set from the server; as well as The step of displaying the test accuracy in response to the test accuracy from the server to determine the model optimization information includes: In response to the subset of recognition error images and the test accuracy from the server side, at least a portion of the subset of recognition error images, the indicator threshold of the static indicator and the test accuracy are displayed so that a user can determine the model optimization information based on the subset of recognition error images, the indicator threshold of the static indicator and the test accuracy.

3. The method according to claim 2, further comprising: Displaying a preset mapping relationship, wherein the preset mapping relationship includes an indicator threshold of a static indicator and / or a corresponding relationship between the test accuracy and the model optimization information; and / or Model optimization information corresponding to the subset of recognition error images is determined based at least on a preset mapping relationship, an indicator threshold of the static indicator, and the test accuracy.

4. The method according to claim 2, wherein: The displaying of at least part of the subset of the recognition error images includes: for each of the at least part of the recognition error images displayed, displaying the recognition error image and at least one of the following: a recognition positioning box, an annotation positioning box, object annotation information and object recognition information corresponding to the recognition error image.

5. The method according to claim 4, wherein: If the recognition result confidence is less than a first threshold, and the intersection-over-union ratio is less than a second preset threshold, the model optimization information includes at least one of the following: abnormal annotation information of the training image set, the number of the training image set is less than a preset image number threshold, the model is underfitting, and at least one of increasing the input image size; If the recognition result confidence is less than the first threshold, and the intersection-over-union ratio is greater than the second preset threshold, the model optimization information includes at least one of the following: the number of training image sets is less than the preset image number threshold, the number of model training rounds is increased, the number of images input for each round of training is reduced, the training images are sliced, and there are images whose categories of training images overlap with other categories; If the recognition result confidence is greater than a first threshold, and the intersection-over-union ratio is less than the second preset threshold, the model optimization information includes at least one of the following: abnormal annotation information of the training image set, improving the training image quality of the training image set, and increasing the input image size; as well as If the types of at least some images in the subset of misidentified images are rare types in business applications, the model optimization information includes at least one of reducing or canceling the training image set belonging to the rare type and prompting manual processing.

6. The method according to claim 5, wherein: The static indicators also include a confusion matrix; and If the first proportion of false negative classes relative to negative classes is greater than a third threshold and / or the second proportion of false positive classes relative to positive classes is greater than a fourth threshold, the model optimization information includes at least one of the following: the number of training image sets is less than a preset image number threshold, the number of model training rounds is increased, the number of images input for each round of training is reduced, images are sliced, and there are at least one of images in which the categories of training images overlap with other categories.

7. The method according to any one of claims 1 to 6, further comprising: After presenting the test accuracy, In response to the optimization information input operation, the model optimization information is sent to the server.

8. The method according to any one of claims 1 to 6, wherein: The model optimization information includes at least one of: a type of a training image set, a number of training images in the training image set, a number of network layers, model parameters, and a size of a training image.

9. A model optimization method executed by a server, comprising: receiving a test request from a terminal, wherein the test request includes a set identifier of a test image set; The trained model is used to process a test image set corresponding to the set identifier to obtain a value of a static indicator for the test image in the test image set, wherein the test image has an annotation positioning frame information and object annotation information for the image in the annotation positioning frame; and the static indicator includes: a recognition result confidence for the test image and an intersection-and-union ratio for the annotation positioning frame, wherein the intersection-and-union ratio of the annotation positioning frame represents an overlap ratio of the recognition positioning frame relative to the annotation positioning frame; In response to an indicator threshold of at least part of the static indicator from the terminal, determining a test accuracy rate for the test image set based on a value of the static indicator and the indicator threshold; and The test accuracy is sent to the terminal to determine model optimization information.

10. The method according to claim 9, further comprising: After determining the test accuracy for the test image set, Determining a subset of recognition error images corresponding to the test accuracy; as well as The subset of recognition-error images is sent to the terminal so that the terminal displays the subset of recognition-error images.

11. A model optimization device, arranged in a terminal, comprising: A request sending module, used for sending a test request including a set identifier to a server side, so that the server side obtains the value of a static indicator of a test image in a test image set corresponding to the set identifier, wherein the test image has an annotation positioning frame information and object annotation information for the image in the annotation positioning frame, and the static indicator includes: a recognition result confidence for the test image and an intersection-and-union ratio for the annotation positioning frame, wherein the intersection-and-union ratio of the annotation positioning frame represents an overlap ratio of the recognition positioning frame relative to the annotation positioning frame; an indicator threshold sending module, configured to send the indicator thresholds of at least part of the static indicators to the server side in response to the threshold input operation, so that the server side determines the test accuracy for the test image set based on the values ​​of the static indicators of the test images and the indicator thresholds; and The accuracy display module is used to display the test accuracy in response to the test accuracy from the server side, so as to determine the model optimization information.

12. A model optimization device, arranged on a server side, comprising: A request receiving module, configured to receive a test request from a terminal, wherein the test request includes a set identifier of a test image set; An image processing module, used for processing a test image set corresponding to the set identifier using a trained model to obtain a value of a static index for a test image in the test image set, wherein the test image has information of a labeling positioning frame and object labeling information for the image in the labeling positioning frame; and the static index includes: a recognition result confidence for the test image and an intersection-and-union ratio for the labeling positioning frame, wherein the intersection-and-union ratio of the labeling positioning frame represents an overlap ratio of the recognition positioning frame relative to the labeling positioning frame; an accuracy determination module, configured to determine, in response to an indicator threshold of at least part of the static indicators from the terminal, a test accuracy for the test image set based on a value of the static indicator and the indicator threshold; and The accuracy sending module is used to send the test accuracy to the terminal so as to determine the model optimization information.

13. An electronic device comprising: one or more processors; A storage device for storing executable instructions, which, when executed by the processor, implements the model optimization method executed by the terminal according to any one of claims 1 to 8, or implements the model optimization method executed by the server according to any one of claims 9 to 10.

14. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implements a model optimization method executed by a terminal according to any one of claims 1 to 8, or implements a model optimization method executed by a server according to any one of claims 9 to 10.

Images