Model performance evaluation method and device, model training method and device and electronic equipment

Abstract

The invention provides a model performance evaluation method and device, a model training method and device and electronic equipment, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision and optical character recognition. According to the specific implementation scheme, in response to a model performance evaluation request for a target model, optical character recognition is conducted on a to-be-recognized object contained in a labeled image through the target model, a first structured character string is obtained, and a label of the labeled image is a second structured character string obtained by labeling the to-be-recognized object; performing similarity calculation on the first structured character string and the second structured character string by using a twin network to obtain a similarity value between the first structured character string and the second structured character string; and obtaining a performance evaluation result of the target model based on a similarity value between the first structured character string and the second structured character string.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning, computer vision, and optical character recognition technology, and specifically to a model performance evaluation method, a model training method, a device, and an electronic device. Background Art

[0002] Optical Character Recognition (OCR) is a technology that converts text in images into machine-readable text. It can automatically extract text from scanned documents, photos, and files, reducing the need for manual data entry and improving data processing efficiency. Summary of the invention

[0003] The present invention provides a model performance evaluation method, a model training method, a device and an electronic device.

[0004] According to one aspect of the present disclosure, a model performance evaluation method is provided, including: in response to a model performance evaluation request for a target model, using the target model to perform optical character recognition on an object to be identified contained in an annotated image to obtain a first structured string, and the label of the annotated image is a second structured string obtained by annotating the object to be identified; using a twin network to perform similarity calculation on the first structured string and the second structured string to obtain a similarity value between the first structured string and the second structured string; and based on the similarity value between the first structured string and the second structured string, obtaining a performance evaluation result of the target model.

[0005] According to another aspect of the present disclosure, a model training method is provided, including: obtaining multiple sample pairs, the sample pairs including two structured strings, the sample pair labels being used to characterize the degree of similarity between the two structured strings included in the sample pairs; and using the multiple sample pairs to train an initial network to obtain a twin network.

[0006] According to another aspect of the present disclosure, a model performance evaluation device is provided, including: a character recognition module, which is used to respond to a model performance evaluation request for a target model, and use the target model to perform optical character recognition on an object to be identified contained in an annotated image to obtain a first structured character string, and the label of the annotated image is a second structured character string obtained by annotating the object to be identified; a similarity determination module, which is used to use a twin network to calculate the similarity between the first structured character string and the second structured character string to obtain a similarity value between the first structured character string and the second structured character string; and a model evaluation module, which is used to obtain a performance evaluation result of the target model based on the similarity value between the first structured character string and the second structured character string.

[0007] According to another aspect of the present disclosure, a model training device is provided, including: a sample pair acquisition module, used to acquire multiple sample pairs, the sample pairs include two structured strings, and the sample pair labels are used to characterize the similarity between the two structured strings included in the sample pairs; and a model training module, used to train an initial network using the multiple sample pairs to obtain a twin network.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture to which a model performance evaluation method, a model training method, and an apparatus according to an embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 A flow chart of a model performance evaluation method according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A flowchart for determining a similarity value according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 5A An image of a formula object compiled from an initial structured character string according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 5B An image of a formula object compiled from a first structured character string according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5C An image of a formula object compiled from a first structured character string according to another embodiment of the present disclosure is schematically shown;

[0020] Figure 5D An image of a formula object compiled from a second structured character string according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5E An image of a formula object compiled from a second structured character string according to another embodiment of the present disclosure is schematically shown;

[0022] Figure 6 A schematic diagram of adjusting model parameters of an initial network according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 7 A schematic diagram schematically shows a method of removing target sample pairs from a training set according to an embodiment of the present disclosure;

[0024] Figure 8 A block diagram of a model performance evaluation device according to an embodiment of the present disclosure is schematically shown;

[0025] Fig. 9 A block diagram schematically shows a model training device according to an embodiment of the present disclosure; and

[0026] Fig.10 A block diagram of an electronic device suitable for implementing a model performance evaluation method and a model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] When using OCR technology to process documents, especially academic papers, it is usually necessary to recognize a large number of mathematical formulas. Since mathematical formulas include special characters such as Greek letters, Latin letters, and mathematical symbols, and the format of mathematical formulas is quite different from that of ordinary text, it is difficult to ensure the accuracy of mathematical formulas obtained using OCR technology. The recognition effect of OCR models in recognizing mathematical formulas can be tested by calculating the text edit distance to ensure that the mathematical formulas have high availability and accuracy.

[0029] However, the text edit distance is only calculated based on the surface differences of characters. Different typesetting and formats may have a significant impact on the calculation of text edit distance. Since mathematical formulas usually have multiple representation methods, there may be edit distances between different representation methods, thus affecting the evaluation results.

[0030] In addition, the symbols in mathematical formulas have specific mathematical meanings and priorities. Since the text edit distance does not take into account the particularity of the symbols, using the text edit distance to evaluate the performance of the model will further increase the difference and lead to inaccurate evaluation results.

[0031] In view of this, an embodiment of the present disclosure provides a model performance evaluation method, a model training method, an apparatus and an electronic device, wherein the model performance evaluation method comprises: in response to a model performance evaluation request for a target model, using the target model to perform optical character recognition on an object to be identified contained in an annotated image to obtain a first structured string, wherein the label of the annotated image is a second structured string obtained by annotating the object to be identified; using a twin network to perform similarity calculation on the first structured string and the second structured string to obtain a similarity value between the first structured string and the second structured string; and based on the similarity value between the first structured string and the second structured string, obtaining a performance evaluation result of the target model.

[0032] Figure 1 An exemplary system architecture to which a model performance evaluation method, a model training method, and an apparatus can be applied according to an embodiment of the present disclosure is schematically illustrated.

[0033] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the model performance evaluation method, model training method and device can be applied may include a terminal device, but the terminal device may implement the model performance evaluation method, model training method and device provided by the embodiments of the present disclosure without interacting with the server.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0036] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0037] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0038] It should be noted that the model performance evaluation method and model training method provided in the embodiments of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the model performance evaluation device and model training device provided in the embodiments of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0039] Alternatively, the model performance evaluation method and model training method provided in the embodiments of the present disclosure may also be generally performed by the server 105. Accordingly, the model performance evaluation device and model training device provided in the embodiments of the present disclosure may generally be arranged in the server 105. The model performance evaluation method and model training method provided in the embodiments of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the model performance evaluation device and model training device provided in the embodiments of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0042] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0043] Figure 2 The flowchart of the model performance evaluation method according to an embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the method includes operations S210 to S230.

[0045] In operation S210, in response to a model performance evaluation request for a target model, optical character recognition is performed on an object to be identified contained in an annotated image using the target model to obtain a first structured string, and a label of the annotated image is a second structured string obtained by annotating the object to be identified.

[0046] According to an embodiment of the present disclosure, a model performance evaluation request may be issued by a user to evaluate the performance of a target model in performing optical character recognition.

[0047] According to an embodiment of the present disclosure, the target model is a model for performing optical character recognition on an object to be recognized contained in an annotated image and outputting a first structured string, wherein the first structured string output by the target model may be a string in LaTeX format, and LaTeX is a typesetting system that can accurately and clearly typeset mathematical symbols and structures in mathematical formulas. The annotated image may be an image including a large amount of text and formulas, such as a textbook page, a paper page, etc., and the object to be recognized may be an object in the annotated image that is difficult to accurately recognize using a traditional optical character recognition method.

[0048] According to an embodiment of the present disclosure, the second structured character string may be determined by manually analyzing the object to be identified in the annotated image, and the second structured character string has the same format as the first structured character string.

[0049] In operation S220 , similarity calculation is performed on the first structured string and the second structured string using the twin network to obtain a similarity value between the first structured string and the second structured string.

[0050] According to an embodiment of the present disclosure, a twin network is a neural network for learning the similarities or differences between pairs of input data, wherein the twin network includes two sub-networks and a similarity calculation layer for determining the similarity between the input data, and the network structures and model parameters of the two sub-networks are the same, thereby ensuring that the input samples of the two sub-networks are mapped to the same feature space, that is, the two sub-networks extract features from their respective input data in the same way, so that the similarity between the respective input data can be determined by directly comparing the similarity between the vectors output by the two sub-networks.

[0051] According to an embodiment of the present disclosure, the two sub-networks included in the twin network are used to process the first structured string and the second structured string respectively to determine the features corresponding to the first structured string and the features corresponding to the second structured string. By comparing the similarities between the features corresponding to the first structured string and the features corresponding to the second structured string, the similarity value between the first structured string and the second structured string can be determined.

[0052] In operation S230 , a performance evaluation result of the target model is obtained based on the similarity value between the first structured character string and the second structured character string.

[0053] According to the embodiment of the present disclosure, since the second structured character string is determined by manually analyzing the object to be recognized in the annotated image, it can be considered that the second structured character string is a correct character string representation of the object to be recognized.

[0054] According to an embodiment of the present disclosure, based on the similarity value between the first structured string and the second structured string, the similarity value between the feature corresponding to the first structured string and the feature corresponding to the second structured string can be determined. When the similarity value is higher than a preset similarity threshold, it can be determined that the formulas represented by the first structured string and the second structured string are structurally similar and semantically identical. The performance evaluation result of the target model thus obtained is used to characterize that the performance of the target model meets the requirements.

[0055] According to an embodiment of the present disclosure, when the similarity value is lower than a preset similarity threshold, it can be determined that there is a semantic difference between the formulas represented by the first structured string and the second structured string respectively, and the performance result of the target model obtained is used to characterize that the performance of the target model does not meet the requirements.

[0056] According to an embodiment of the present disclosure, a target model is used to process an annotated image to obtain a first structured character string corresponding to an object to be identified. Using a twin network, similarity is calculated for the first structured character string and the second structured character string obtained by annotating the object to be identified, and a similarity value between the first structured character string and the second structured character string is obtained to determine the performance evaluation result of the target model. Since the second structured character string is obtained by annotating the object to be identified, the similarity value can characterize the similarity value between the first structured character string and the label of the object to be identified, and the performance evaluation result obtained based on the similarity value can accurately and objectively evaluate the performance of the target model. In addition, using similarity values ​​to evaluate model performance can avoid the problem of inaccurate semantic judgment and errors in model performance evaluation caused by not considering the particularity of symbols in the object to be identified when evaluating an optical character recognition model using, for example, edit distance, and only considering the differences between the character strings, thereby further improving the accuracy of model performance evaluation.

[0057] According to an embodiment of the present disclosure, a twin network is used to perform similarity calculation on a first structured string and a second structured string to obtain a similarity value between the first structured string and the second structured string, including: generating a first image based on the first structured string; generating a second image based on the second structured string; and inputting the first image and the second image into the twin network to obtain a similarity value between the first image and the second image, wherein the similarity value between the first structured string and the second structured string is expressed as a similarity value between the first image and the second image.

[0058] According to an embodiment of the present disclosure, a first structured string is compiled to generate a first image corresponding to the first structured string, and a second structured string is compiled to generate a second image corresponding to the second structured string.

[0059] According to an embodiment of the present disclosure, a first image and a second image are input into a twin network, and the first image is processed using one of two sub-networks included in the twin network, and the second image is processed using the other sub-network to respectively determine image features of the first image and image features of the second image.

[0060] According to an embodiment of the present disclosure, in the similarity calculation layer, a similarity determination method is used to determine the similarity value between the image features of the first image and the image features of the second image, and the similarity value is used as the similarity value between the first structured string and the second structured string, wherein the similarity determination method may include Euclidean distance, cosine similarity, contrast loss and other methods.

[0061] According to an embodiment of the present disclosure, a first image and a second image are generated based on a first structured string and a second structured string, respectively, and the first image and the second image are input into a twin network to determine the similarity value between the first image and the second image, and the similarity value between the first image and the second image is used as the similarity value between the first structured string and the second structured string. By using the twin network to process the first image and the second image, it is possible to reduce the problem of misjudgment caused by directly using the similarity of structured strings for model performance evaluation when the structured strings are greatly different but the semantics represented by the first image and the second image are the same, based on the characteristics of the structured strings, thereby improving the accuracy and stability of the model performance evaluation.

[0062] Figure 3 The flowchart of determining a similarity value according to an embodiment of the present disclosure is schematically shown.

[0063] like Figure 3 As shown, the twin network 310 includes a first sub-network 311, a second sub-network 312 and a similarity calculation layer 313, wherein the network structures and network parameters of the first sub-network 311 and the second sub-network 312 are completely the same. The first structured string 301 and the second structured string 302 are compiled respectively to obtain a first image 303 and a second image 304, and the first image 303 and the second image 304 are input into the twin network 310, and the first sub-network 311 of the twin network is used to process the first image 303 to determine the image feature 305 of the first image, and the second sub-network 312 of the twin network is used to process the second image 304 to determine the image feature 306 of the second image, and based on the image feature 305 of the first image and the image feature 306 of the second image, the similarity calculation layer 313 is used to determine the similarity value 307.

[0064] According to an embodiment of the present disclosure, the object to be identified includes a formula object.

[0065] According to the embodiments of the present disclosure, since formula objects often contain complex two-dimensional structures, characters may include various mathematical symbols, Greek letters, Latin letters, etc., and their forms vary. Formulas also contain a variety of special symbols and structural combinations such as superscripts and subscripts, fractions, square roots, integrals, etc., so formula objects are difficult to accurately recognize using traditional optical character recognition methods. Formula objects can be used as objects to be recognized, and after recognition using a target model, the recognition effect of the target model can be evaluated for model performance.

[0066] Figure 4 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown.

[0067] like Figure 4 As shown, the method includes operations S410 to S420.

[0068] In operation S410 , a plurality of sample pairs are obtained, each sample pair including two structured character strings, and a sample pair label is used to characterize the similarity between the two structured character strings included in the sample pair.

[0069] According to an embodiment of the present disclosure, when the semantics of the formula objects respectively corresponding to the two structured strings included in the sample pair are the same, the sample pair label is used to characterize that the two structured strings included in the sample pair are highly similar. When the semantics of the formula objects respectively corresponding to the two structured strings included in the sample pair are different, the sample pair label is used to characterize that the two structured strings included in the sample pair are less similar.

[0070] According to an embodiment of the present disclosure, the sample pair label can use the label of the binary classification problem, that is, when the sample pair label is used to represent that the two structured strings included in the sample pair have a high degree of similarity, the sample pair label can take 1, and when the sample pair label is used to represent that the two structured strings included in the sample pair have a low degree of similarity, the sample pair label can take 0.

[0071] In operation S420, the initial network is trained using the plurality of sample pairs to obtain a twin network.

[0072] According to an embodiment of the present disclosure, the network structure of the initial network is the same as that of the twin network, including two sub-networks with the same network structure, and an initial similarity calculation layer for determining the similarity between input data.

[0073] According to an embodiment of the present disclosure, a plurality of sample pairs are respectively input into an initial network. For two structured strings included in each sample pair, the first structured string is processed by a first subnetwork, and the second structured string is processed by a second subnetwork, so as to respectively determine the features of the two structured strings. Based on the features of the two structured strings, the similarity value between the two structured strings is determined by an initial similarity calculation layer.

[0074] According to an embodiment of the present disclosure, the model parameters of the initial network are adjusted based on the similarity value and the sample pair label until the similarity value output by the network matches the sample pair label, and the twin network is determined based on the current model parameters.

[0075] According to an embodiment of the present disclosure, the initial network is trained using a plurality of sample pairs and the similarity between two structured strings included in the sample pairs to obtain a twin network. The sample pairs can simulate the input data in the application process of the twin network, so that the similarity value output by the twin network after parameter adjustment matches the sample pair label, thereby improving the accuracy of the output similarity value when the twin network is used to evaluate the performance of the model.

[0076] According to an embodiment of the present disclosure, the model training method further includes: generating multiple sample pairs based on multiple initial structured character strings.

[0077] According to an embodiment of the present disclosure, the initial structured character string may include a structured character string obtained by manually annotating a formula object, and the initial structured character string may be determined by selecting a formula sample from a paper data set.

[0078] According to an embodiment of the present disclosure, based on each initial structured string, one or more strings different from the initial structured string are determined, and strings corresponding to the same initial structured string are combined multiple times to determine multiple sample pairs.

[0079] According to an embodiment of the present disclosure, based on multiple initial structured strings, one or more strings corresponding to each initial structured string are determined, and the strings corresponding to the same initial structured string are combined multiple times to determine multiple sample pairs. It is possible to construct a data set including a large number of sample pairs based on the multiple initial structured strings, thereby achieving data enhancement, increasing the number of data sets in the training process, reducing overfitting, and improving the generalization ability of the twin model.

[0080] According to an embodiment of the present disclosure, multiple sample pairs are generated based on multiple initial structured strings, including: based on preset fields, multiple field embedding processes are performed based on multiple embeddable positions included in the initial structured string to obtain multiple first structured strings; multiple field replacement processes are performed based on multiple replaceable positions included in the initial structured string to obtain multiple second structured strings; and multiple sample pairs are generated based on the multiple first structured strings and the multiple second structured strings.

[0081] According to an embodiment of the present disclosure, the preset fields include fields that do not affect the semantics of the formula object corresponding to the structured string, but only adjust the format or layout of the formula object, for example, the field "\mathrm{}" for setting text or mathematical symbols to Roman font, the field "\quad" for adding spaces, etc.

[0082] According to an embodiment of the present disclosure, the initial structured character string may include multiple fields, and the area between two adjacent fields may be determined as an embeddable position, thereby determining the multiple embeddable positions included in the initial structured character string.

[0083] According to an embodiment of the present disclosure, a preset field is embedded into any embeddable position to complete an embedding process. Multiple embeddable positions included in the initial structured string are subjected to multiple field embedding processes to obtain multiple first structured strings, wherein each first structured string can be obtained by performing one or more field embedding processes on the initial structured string, and the first structured string has the same semantics as the initial structured string.

[0084] According to an embodiment of the present disclosure, the variable in each field in the initial structured string can be determined as a replaceable position. For example, in the field "\partial x_1", "\partial" represents the partial derivative symbol. Replacing the letters therein will cause the replaced structured string to be unable to compile, so this type of field cannot be replaced. After replacing "x" and "1", the replaced structured string can be compiled normally, so this type of field is a variable.

[0085] According to an embodiment of the present disclosure, multiple field replacement processes are performed on multiple replaceable positions included in the initial structured string to obtain multiple second structured strings, wherein each second structured string can be obtained by performing one or more field replacement processes on the initial structured string, and the second structured string has different semantics from the initial structured string.

[0086] According to an embodiment of the present disclosure, multiple embeddable positions included in the second structured character string may be subjected to multiple field embedding processes to increase the complexity of the second structured character string.

[0087] According to an embodiment of the present disclosure, multiple deletable fields included in the initial structured string may be subjected to multiple field deletion processes to obtain multiple second structured strings, wherein the deletable fields are fields in the initial structured string that do not affect normal compilation of the new structured string after deletion. The second structured string obtained by the field deletion process has different semantics from the initial structured string.

[0088] According to an embodiment of the present disclosure, after the initial structured character string is processed multiple times to obtain multiple first structured character strings and multiple second structured character strings, multiple combinations are performed based on the multiple first structured character strings and the multiple second structured character strings to generate multiple sample pairs, wherein each sample pair may be obtained by combining two different first structured character strings or by combining a first structured character string with a second structured character string.

[0089] According to an embodiment of the present disclosure, the initial structured string is subjected to field embedding processing using a field that does not affect the semantics of the formula to obtain a first structured string, thereby ensuring that the semantics of the first structured string are the same as those of the initial structured string, and the first structured string is a positive sample relative to the initial structured string. The initial structured string is subjected to field replacement processing to obtain a second structured string, thereby making the semantics of the second structured string different from that of the initial structured string, and the second structured string is a negative sample relative to the initial structured string. By processing the fields of the initial structured string, generating multiple positive samples and multiple negative samples, and determining sample pairs, the number of sample pairs can be increased, thereby achieving data enhancement.

[0090] Figure 5A An image of a formula object compiled from an initial structured character string according to an embodiment of the present disclosure is schematically shown.

[0091] like Figure 5A As shown, the initial structured string “\frac{\partial^2}{\partial x_1\partialx_2}y” is compiled to obtain a formula object.

[0092] Figure 5B An image of a formula object compiled from a first structured character string according to an embodiment of the present disclosure is schematically shown.

[0093] like Figure 5B As shown, the preset field "\mathrm{}" is embedded in the embeddable position between "\frac{\partial^2}" and "\partial x_1" in the initial structured string, and the field "\partial x_1" after the preset field is written into the braces of the preset field as a parameter of the preset field, and the first structured string "\frac{\partial^2}{\mathrm{\partial x_1}\partial x_2}y" is obtained. The first structured string is compiled to obtain a formula object. Among them, since the field "\partial x_1" in the first structured string is a parameter of the preset field "\mathrm{}", the "x1" at the denominator of the formula object is in Roman font, which is different from the image of the formula object corresponding to the initial structured string, but the expression semantics are the same.

[0094] Figure 5C An image of a formula object compiled from a first structured character string according to another embodiment of the present disclosure is schematically shown.

[0095] like Figure 5CAs shown, the preset field "\mathrm{}" is embedded in the embeddable position between "\frac{\partial^2}" and "\partial x_1" in the initial structured string, and the field "\partial x_1" after the preset field is written into the curly brackets of the preset field as a parameter of the preset field, and then the preset field "\quad" is embedded in the embeddable position between the field "\partial x_1" and the field "\partial x_2", to obtain another first structured string "\frac{\partial^2}{\mathrm{\partial x_1}\quad\partial x_2} y", and the first structured string is compiled to obtain a formula object. Among them, since the field "\partialx_1" in the first structured string is a parameter of the preset field "\mathrm{}", the "x1" at the denominator of the formula object 503 is in Roman font, and since there is "\quad" between the field "\partial x_1" and the field "\partial x_2", there is a space between the partial differential symbols of x1 and x2 in the formula object, which is different from the image of the formula object corresponding to the initial structured string, but the expression semantics are the same.

[0096] Figure 5D An image of a formula object compiled from a second structured character string according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 5D As shown, the preset field "\mathrm{}" is embedded in the embeddable position between "\frac{\partial^2}" and "\partial x_1" in the initial structured string, and the field "\partial x_1" after the preset field is written into the curly brackets of the preset field as a parameter of the preset field, and the field "x_2" is changed to "x_3" through field replacement processing, and the second structured string "\frac{\partial^2}{\mathrm{\partial x_1}\quad\partial x_3} y" is obtained. The second structured string is compiled to obtain a formula object. Among them, since the field "\partial x_1" in the first structured string is a parameter of the preset field "\mathrm{}", the "x1" at the denominator of the formula object is in Roman font, which does not affect the semantics of the formula object, but the partial differential of x2 is changed to the partial differential of x3, and the semantics of the formula object is changed.

[0098] Figure 5EAn image of a formula object compiled from a second structured character string according to another embodiment of the present disclosure is schematically shown.

[0099] like Figure 5E As shown, the preset field "\mathrm{}" is embedded in the embeddable position between "\frac{\partial^2}" and "\partial x_1" in the initial structured string, and the field "\partial x_1" after the preset field is written into the curly brackets of the preset field as a parameter of the preset field, and the field "\partial^2" is replaced by field replacement and changed to "\beta {} ^2", and another second structured string "\frac{\beta {} ^2}{\mathrm{\partial x_1}\ partial x_3} y" is obtained. The second structured string is compiled to obtain a formula object. Among them, since the field "\partial x_1" in the first structured string is a parameter of the preset field "\mathrm{}", the "x1" at the denominator of the formula object is in Roman font, which does not affect the semantics of the formula object, but the partial symbol at the numerator is changed to β, and the semantics of the formula object is changed.

[0100] According to an embodiment of the present disclosure, sample pairs include positive sample pairs and negative sample pairs; wherein, based on multiple first structured strings and multiple second structured strings generated from multiple initial structured strings respectively, multiple sample pairs are generated, including: based on two first target structured strings, a positive sample pair is generated, and the two first target structured strings are determined from multiple first structured strings; and based on a second target structured string and a third target structured string, a negative sample pair is generated, and the second target structured string is determined from multiple first structured strings, and the third target structured string is determined from multiple second structured strings.

[0101] According to an embodiment of the present disclosure, the sample pair includes a positive sample pair and a negative sample pair, wherein the semantics of the two structured strings included in the positive sample pair are the same, and the semantics of the two structured strings included in the negative sample pair are different.

[0102] According to an embodiment of the present disclosure, two different first structured strings are determined from a plurality of first structured strings as two first target structured strings. Since the two first target structured strings have the same semantics as the initial structured string, a positive sample pair can be generated based on the two first target structured strings.

[0103] According to an embodiment of the present disclosure, a first structured string is determined from a plurality of first structured strings as a second target structured string, and a second structured string is determined from a plurality of second structured strings as a third target structured string. Since the second structured string has the same semantics as the initial structured string, and the third structured string has different semantics from the initial structured string, the second structured string has different semantics from the third structured string, and a negative sample pair can be generated based on the second target structured string and the third target structured string.

[0104] According to an embodiment of the present disclosure, a positive sample pair is constructed using two first structured strings with the same generated semantics, and a negative sample pair is constructed using a second structured string and a third structured string with different generated semantics. This ensures the correctness of the labels of the sample pairs, thereby improving the efficiency and effectiveness of model training and obtaining a twin network that meets the requirements.

[0105] According to an embodiment of the present disclosure, an initial network is trained using multiple sample pairs to obtain a twin network, including: generating two sample images based on two structured strings included in the sample pairs; inputting the two sample images into the initial network to obtain similarity values ​​of the sample pairs; obtaining loss values ​​based on the similarity values ​​of each of the multiple sample pairs and the labels of each of the multiple sample pairs; and adjusting the model parameters of the initial network based on the loss values ​​to finally obtain the twin network.

[0106] According to an embodiment of the present disclosure, two structured strings included in a sample pair are compiled, and two sample images corresponding to the two structured strings are generated respectively, and the two sample images are input into an initial network, and the first sample image is processed by a first sub-network, and the second sample image is processed by a second sub-network, respectively, to obtain sample features of the two sample images, and based on the sample features, the similarity values ​​of the two sample images are determined, and the similarity value is used as the similarity value of the sample pair.

[0107] According to an embodiment of the present disclosure, based on the similarity value of the sample pair and the label of the sample pair, the loss value for the sample pair is determined. Since the label of the sample pair adopts the labels 0 and 1 of the binary classification task, and when the sample pair label is used to characterize the high similarity of the two structured strings included in the sample pair, the sample pair label takes 1, and when the sample pair label is used to characterize the low similarity of the two structured strings included in the sample pair, the sample pair label takes 0. Therefore, when the label takes 1, the similarity value of the sample pair should be higher, and the closer to 1, the better the network performance of the initial network. When the label takes 0, the similarity value of the sample pair should be lower, and the closer to 0, the better the network performance of the initial network. The loss value of the sample pair can be determined by the absolute value of the difference between the sample pair and the similarity value.

[0108] According to an embodiment of the present disclosure, the model parameters of the initial network are adjusted based on the loss value to obtain a twin network.

[0109] According to an embodiment of the present disclosure, a loss value is determined based on the similarity value of the sample pair and the label of the sample pair, and the model parameters of the initial network are adjusted based on the loss value to determine the twin network. The performance of the model on the training data can be improved by finding a combination of model parameters that minimizes the loss value. Since the training data is the same as the input data during application, the performance of the twin network during model performance evaluation can be improved.

[0110] Figure 6 A schematic diagram of adjusting model parameters of an initial network according to an embodiment of the present disclosure is schematically shown.

[0111] like Figure 6 As shown, the sample pair data set includes N sample pairs, each sample pair includes two samples and a label of the sample pair, for example, sample pair N includes sample N-1 and sample N-2, and label N.

[0112] The two samples included in the sample pair are processed respectively using the initial network 601 to determine the similarity value 602 of the sample pair, and according to the similarity value 602 and the label of the sample pair, operation S603 is performed on the sample pair to calculate the loss value, and based on the loss value, the network parameters of the initial network are updated. In the case where the loss value is greater than the loss threshold, the model parameters of the initial network are adjusted based on the loss value until the new loss value obtained after the above operation is performed using the adjusted network is less than the loss threshold, and the adjustment of the model parameters is stopped to obtain a twin network.

[0113] According to an embodiment of the present disclosure, the model training method also includes: when the loss value is greater than a preset threshold, determining at least one target sample pair related to the loss value; and eliminating at least one target sample pair from multiple sample pairs.

[0114] According to the embodiments of the present disclosure, after multiple rounds of model parameter adjustment, it can be considered that the model has achieved good performance. In this case, the reason why the loss value is greater than the preset threshold may not be a problem with the model performance, but the presence of noise in the sample pairs. The sample pairs corresponding to the noise can be judged by the size of the loss values ​​obtained from different sample pairs, and the sample pairs can be eliminated to complete data cleaning.

[0115] According to an embodiment of the present disclosure, the above-mentioned model training operation is performed using a model that has undergone multiple rounds of model parameter adjustment, or a twin network obtained after the parameter adjustment is completed, and when the loss value is greater than a preset threshold, the sample pair that obtains the loss value greater than the preset threshold is determined as a target sample pair, and the target sample pair is eliminated from multiple sample pairs, wherein, preferably, the preset threshold can be 0.8.

[0116] According to an embodiment of the present disclosure, based on the loss value obtained by the model after multiple rounds of model parameter adjustment, the dirty data that may exist in multiple sample pairs is checked and cleaned, which can reduce the possibility of including dirty data in the training data, thereby improving the accuracy of the training data, and improving the efficiency of model training and the performance of the twin network obtained after the training is completed.

[0117] Figure 7 The diagram schematically shows a schematic diagram of eliminating target sample pairs from a training set according to an embodiment of the present disclosure.

[0118] like Figure 7 As shown, the sample pair data set includes N sample pairs, each sample pair includes two samples and a label of the sample pair, for example, sample pair N includes sample N-1 and sample N-2, and label N.

[0119] The two samples included in the sample pair are processed respectively using the twin network 701 to determine the similarity value 702 of the sample pair. According to the similarity value 702 and the label of the sample pair, operation S703 is performed to calculate the loss value of the sample pair, and operation S704 is performed to determine the relationship between the loss value and a preset threshold. When the loss value is greater than the preset threshold, the sample pair that obtains the loss value is determined as the target sample pair, and based on the target sample pair, the target sample pair is removed from the sample pair data set.

[0120] Figure 8 A block diagram of a model performance evaluation device according to an embodiment of the present disclosure is schematically shown.

[0121] like Figure 8 As shown, the model performance evaluation device 800 of this embodiment includes a character recognition module 810 , a similarity determination module 820 and a model evaluation module 830 .

[0122] The character recognition module 810 is used to respond to a model performance evaluation request for a target model, use the target model to perform optical character recognition on the object to be recognized contained in the annotated image, and obtain a first structured string. The label of the annotated image is a second structured string obtained by annotating the object to be recognized.

[0123] The similarity determination module 820 is used to calculate the similarity between the first structured string and the second structured string using the twin network to obtain a similarity value between the first structured string and the second structured string.

[0124] The model evaluation module 830 is used to obtain a performance evaluation result of the target model based on the similarity value between the first structured character string and the second structured character string.

[0125] According to an embodiment of the present disclosure, the similarity determination module 820 includes a first image generation submodule, a second image generation submodule and a similarity determination submodule.

[0126] The first image generating submodule is used to generate a first image based on the first structured character string.

[0127] The second image generating submodule is used to generate a second image based on the second structured character string.

[0128] The similarity determination submodule is used to input the first image and the second image into the twin network to obtain a similarity value between the first image and the second image, wherein the similarity value between the first structured string and the second structured string is expressed as a similarity value between the first image and the second image.

[0129] According to an embodiment of the present disclosure, the object to be identified includes a formula object.

[0130] Fig. 9 A block diagram of a model training device according to an embodiment of the present disclosure is schematically shown.

[0131] like Fig. 9 As shown, the model performance evaluation device 900 of this embodiment includes a sample pair acquisition module 910 and a model training module 920.

[0132] A sample pair acquisition module 910 is used to acquire a plurality of sample pairs, wherein the sample pairs include two structured strings, and the sample pair labels are used to indicate the similarity between the two structured strings included in the sample pairs; and

[0133] The model training module 920 is used to train the initial network using multiple sample pairs to obtain a twin network.

[0134] According to an embodiment of the present disclosure, the model training device 900 also includes a sample pair generation module.

[0135] The sample pair generation module is used to generate multiple sample pairs based on multiple initial structured strings.

[0136] According to an embodiment of the present disclosure, the sample pair generation module includes a field embedding submodule, a field replacement submodule and a sample pair generation submodule.

[0137] The field embedding submodule is used to perform multiple field embedding processes based on preset fields and multiple embeddable positions included in the initial structured character string to obtain multiple first structured character strings.

[0138] The field replacement submodule is used to perform multiple field replacement processes based on multiple replaceable positions included in the initial structured character string to obtain multiple second structured character strings.

[0139] The sample pair generation submodule is used to generate a plurality of sample pairs based on a plurality of first structured character strings and a plurality of second structured character strings.

[0140] According to an embodiment of the present disclosure, the sample pairs include positive sample pairs and negative sample pairs; the sample pair generation submodule includes a positive sample pair generation unit and a negative sample pair generation unit.

[0141] The positive sample pair generating unit is used to generate a positive sample pair based on two first target structured character strings, where the two first target structured character strings are determined from a plurality of first structured character strings.

[0142] The negative sample pair generating unit is used to generate a negative sample pair based on a second target structured string and a third target structured string, wherein the second target structured string is determined from a plurality of first structured strings, and the third target structured string is determined from a plurality of second structured strings.

[0143] According to an embodiment of the present disclosure, the model training module 920 includes an image generation submodule, an image input submodule, a loss value determination submodule and a parameter adjustment submodule.

[0144] The image generation submodule is used to generate two sample images based on the two structured strings included in the sample pair.

[0145] The image input submodule is used to input two sample images into the initial network to obtain the similarity value of the sample pair.

[0146] The loss value determination submodule is used to obtain the loss value based on the similarity values ​​of the multiple sample pairs and the labels of the multiple sample pairs.

[0147] The parameter adjustment submodule is used to adjust the model parameters of the initial network based on the loss value to finally obtain the twin network.

[0148] According to an embodiment of the present disclosure, the model training device 900 further includes a target sample pair determination module and a target sample pair elimination module.

[0149] The target sample pair determination module is used to determine at least one target sample pair related to the loss value when the loss value is greater than a preset threshold.

[0150] The target sample pair elimination module is used to eliminate at least one target sample pair from multiple sample pairs.

[0151] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0152] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0153] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0154] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0155] Fig.10 A block diagram of an electronic device suitable for implementing a model performance evaluation method and a model training method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0156] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0157] A number of components in the device 1000 are connected to an input / output (I / O) interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a model performance evaluation method and a model training method. For example, in some embodiments, the model performance evaluation method and the model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the model performance evaluation method and the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the model performance evaluation method and the model training method in any other appropriate manner (for example, by means of firmware).

[0159] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0165] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0166] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model performance evaluation method, comprising: In response to a model performance evaluation request for a target model, optical character recognition is performed on an object to be identified contained in an annotated image using the target model to obtain a first structured string, wherein the label of the annotated image is a second structured string obtained by annotating the object to be identified; Using a twin network to perform similarity calculation on the first structured string and the second structured string to obtain a similarity value between the first structured string and the second structured string; as well as Based on the similarity value between the first structured character string and the second structured character string, a performance evaluation result of the target model is obtained.

2. The method according to claim 1, wherein: The using the twin network to perform similarity calculation on the first structured string and the second structured string to obtain a similarity value between the first structured string and the second structured string includes: Based on the first structured character string, generate a first image; Based on the second structured character string, generating a second image; and The first image and the second image are input into the twin network to obtain a similarity value between the first image and the second image, wherein the similarity value between the first structured string and the second structured string is represented as a similarity value between the first image and the second image.

3. The method according to claim 1, wherein: The object to be identified includes a formula object.

4. A model training method, comprising: Acquire multiple sample pairs, each of which includes two structured strings, and each of which includes a sample pair label used to represent a similarity between the two structured strings included in the sample pair; as well as The initial network is trained using the multiple sample pairs to obtain a twin network.

5. The method according to claim 4, further comprising: The plurality of sample pairs are generated based on a plurality of initial structured character strings.

6. The method according to claim 5, wherein: The generating the plurality of sample pairs based on the plurality of initial structured character strings comprises: Based on the preset fields, multiple field embedding processes are performed based on multiple embeddable positions included in the initial structured character string to obtain multiple first structured character strings; Performing multiple field replacement processes based on multiple replaceable positions included in the initial structured character string to obtain multiple second structured character strings; and A plurality of the sample pairs are generated based on the plurality of first structured character strings and the plurality of second structured character strings.

7. The method according to claim 6, wherein: The sample pairs include positive sample pairs and negative sample pairs; The step of generating the plurality of sample pairs based on the plurality of first structured character strings and the plurality of second structured character strings respectively generated from the plurality of initial structured character strings comprises: generating the positive sample pair based on two first target structured character strings, wherein the two first target structured character strings are determined from the plurality of first structured character strings; and The negative sample pair is generated based on a second target structured string and a third target structured string, wherein the second target structured string is determined from the plurality of first structured strings, and the third target structured string is determined from the plurality of second structured strings.

8. The method according to claim 4, wherein: The method of using the multiple sample pairs to train the initial network to obtain the twin network includes: Based on the two structured character strings included in the sample pair, two sample images are generated; Inputting the two sample images into the initial network to obtain a similarity value of the sample pair; Obtaining a loss value based on the similarity values ​​of the plurality of sample pairs and the labels of the plurality of sample pairs; and The model parameters of the initial network are adjusted based on the loss value to finally obtain the twin network.

9. The method according to claim 8, further comprising: When the loss value is greater than a preset threshold, determining at least one target sample pair associated with the loss value; as well as The at least one target sample pair is eliminated from the multiple sample pairs.

10. A model performance evaluation device, comprising: a character recognition module, configured to, in response to a model performance evaluation request for a target model, perform optical character recognition on an object to be recognized contained in an annotated image using the target model to obtain a first structured string, wherein the label of the annotated image is a second structured string obtained by annotating the object to be recognized; a similarity determination module, configured to calculate the similarity between the first structured string and the second structured string using a twin network to obtain a similarity value between the first structured string and the second structured string; as well as A model evaluation module is used to obtain a performance evaluation result of the target model based on a similarity value between the first structured character string and the second structured character string.

11. A model training device, comprising: A sample pair acquisition module, used to acquire a plurality of sample pairs, wherein the sample pairs include two structured character strings, and the sample pair labels are used to characterize the similarity between the two structured character strings included in the sample pairs; as well as The model training module is used to train the initial network using the multiple sample pairs to obtain a twin network.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images