Identification method and device, segmentation model training method and device
The combination of slicing mathematical expressions and recognition models is combined by slicing the slicing point heat map training model, which solves the drift problem in long mathematical expression recognition and improves recognition accuracy and performance.
Patent Information
- Application Number
- CN202010740413.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-07-28
AI Technical Summary
When faced with long mathematical expression recognition technology, existing mathematical expression recognition technology is prone to drift problems, resulting in the accumulation of recognition errors and catastrophic errors in the entire sequence.
The slicing model is used to segment the recognized image, generate a slicing map, and use the recognition model to identify the slicing map. The slicing model is trained through the slicing point heat map to avoid drift problems caused by excessive content.
It effectively reduces the difficulty of identifying longer mathematical expressions, improves recognition performance, and ensures accurate division and recognition of long mathematical expressions.
Smart Images

Figure CN114005114B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of computer technology, and in particular to identification methods. One or more embodiments of this specification also relate to identification devices, segmentation model training methods, segmentation model training devices, computing devices, and computer-readable storage media. Background Art
[0002] With the advancement of technology and the expansion of application scenarios, text recognition technology has gradually transitioned from simpler document-type text recognition to more complex mathematical expression recognition; however, in the current mathematical expression recognition technology, when faced with longer mathematical expressions, it is often encountered that the sequence of mathematical expressions is too long, resulting in drift during recognition, which causes subsequent recognition to accumulate errors, leading to catastrophic misrecognition of the entire sequence.
[0003] Therefore, it is necessary to provide a recognition method that can reduce the sequence length of mathematical expressions and avoid recognition errors caused by too long sequences. Summary of the invention
[0004] In view of this, the present specification provides an identification method. One or more embodiments of the present specification also relate to an identification device, a segmentation model training method, a segmentation model training device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, there is provided an identification method, including:
[0006] Get the image to be identified;
[0007] Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0008] Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0009] Determining a target recognition result of the image to be recognized based on the initial recognition result;
[0010] The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation graph of the image to be identified.
[0011] The recognition model is suitable for recognizing the segmentation graph and outputting an initial recognition result of the segmentation graph.
[0012] According to a second aspect of an embodiment of this specification, a training method for a segmentation model is provided, comprising:
[0013] Get sample images;
[0014] Inputting the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result;
[0015] Preprocessing the sample image to obtain a projection image of the sample image;
[0016] Determine a segmentation point heat map of the sample image according to the first recognition result, the position image, and the image projection image;
[0017] The initial segmentation model is trained based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
[0018] According to a third aspect of the embodiments of this specification, there is provided an identification method, including:
[0019] Displaying an image input interface to the user based on a call request from the user;
[0020] Receiving a picture to be identified input by the user based on the image input interface;
[0021] Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0022] Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0023] A target recognition result of the image to be recognized is determined based on the initial recognition result, and the target recognition result is returned to the user.
[0024] According to a fourth aspect of the embodiments of this specification, there is provided an identification method, including:
[0025] Receiving a call request sent by a user, wherein the call request carries a picture to be identified;
[0026] Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0027] Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0028] A target recognition result of the image to be recognized is determined based on the initial recognition result, and the target recognition result is returned to the user.
[0029] According to a fifth aspect of the embodiments of this specification, there is provided an identification device, including:
[0030] A first acquisition module is configured to acquire a picture to be identified;
[0031] A first obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0032] A second obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0033] A first determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result;
[0034] The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation graph of the image to be identified.
[0035] The recognition model is suitable for recognizing the segmentation graph and outputting an initial recognition result of the segmentation graph.
[0036] According to a sixth aspect of the embodiments of this specification, a training device for a segmentation model is provided, including:
[0037] A second acquisition module is configured to acquire a sample image;
[0038] A third obtaining module is configured to input the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result;
[0039] A fourth obtaining module is configured to pre-process the sample image to obtain a picture projection image of the sample image;
[0040] A second determination module is configured to determine a segmentation point heat map of the sample image according to the first recognition result, the position image and the image projection image;
[0041] The fifth acquisition module is configured to train the initial segmentation model based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
[0042] According to a seventh aspect of the embodiments of this specification, there is provided an identification device, including:
[0043] An interface display module, configured to display an image input interface to the user based on a user's call request;
[0044] A first receiving module is configured to receive a picture to be identified input by the user based on the image input interface;
[0045] A sixth obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0046] A seventh obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0047] The third determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0048] According to an eighth aspect of the embodiments of this specification, there is provided an identification device, including:
[0049] A second receiving module is configured to receive a call request sent by a user, wherein the call request carries a picture to be identified;
[0050] An eighth obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0051] A ninth obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0052] The third determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0053] According to a ninth aspect of the embodiments of this specification, a computing device is provided, including:
[0054] Memory and processor;
[0055] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the recognition method or the training method of the segmentation model when executing the computer-executable instructions.
[0056] According to the tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the recognition method or the training method of the segmentation model are implemented.
[0057] An embodiment of the present specification implements a recognition method and device, wherein the recognition method includes obtaining a picture to be recognized; inputting the picture to be recognized into a segmentation model to obtain a segmentation graph of the picture to be recognized; inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; determining a target recognition result of the picture to be recognized based on the initial recognition result;
[0058] The recognition method selects appropriate segmentation points to segment the image to be recognized based on the segmentation model, and recognizes the segmented image of the image to be recognized through the recognition model, thereby effectively avoiding the drift problem caused by the excessive length of the content in the image to be recognized. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 It is a schematic diagram of a specific application scenario of an identification method provided by an embodiment of this specification;
[0060] Figure 2 is a flow chart of a first identification method provided by an embodiment of this specification;
[0061] Figure 3 It is a specific schematic diagram of a segmentation point, a segmentation point heat map, and a segmentation map in a processing process of an identification method provided by an embodiment of this specification;
[0062] Figure 4 is a flow chart of a segmentation model training method provided by an embodiment of this specification;
[0063] Figure 5 It is a schematic diagram of a specific processing process of an identification method provided by an embodiment of this specification;
[0064] Figure 6 It is a flowchart of determining a segmentation point heat map and realigning labels in a recognition method provided by an embodiment of this specification;
[0065] Figure 7 is a flow chart of a second identification method provided by an embodiment of this specification;
[0066] Figure 8 is a flow chart of a third identification method provided by an embodiment of this specification;
[0067] Fig. 9 is a schematic diagram of the structure of a first identification device provided by an embodiment of this specification;
[0068] Fig.10 It is a structural schematic diagram of a segmentation model training device provided by an embodiment of this specification;
[0069] Fig.11 is a schematic diagram of the structure of a second identification device provided by an embodiment of this specification;
[0070] Fig.12 is a schematic diagram of the structure of a third identification device provided by an embodiment of this specification;
[0071] Fig.13 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0072] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0073] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0074] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0075] First, the terms involved in one or more embodiments of this specification are explained.
[0076] ResNet: Residual network, built by residual blocks.
[0077] Backbone: Neural network model.
[0078] Attention Mechanism: Attention Mechanism is a data processing method in machine learning, which is widely used in various types of machine learning tasks such as natural language processing, image recognition and speech recognition.
[0079] Bahdanau attention: Attention mechanism.
[0080] DenseNet: Dense Network.
[0081] In this specification, three recognition methods are provided. One or more embodiments of this specification simultaneously relate to a recognition device, a segmentation model training method, a segmentation model training device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0082] See also Figure 1 , Figure 1 A schematic diagram showing a specific application scenario of an identification method provided according to an embodiment of the present specification.
[0083] Specifically, the mathematical expression in the picture a is recognized by the recognition method.
[0084] Figure 1 The application scenarios include terminals and servers.
[0085] Specifically, the user sends picture a to the server through the terminal. After receiving picture a, the server inputs picture a into the pre-trained segmentation model to obtain multiple segmentation graphs of picture a: segmentation Figure 1 , split Figure 2 ...cut graph N;
[0086] Divide Figure 1 , split Figure 2 ...The segmentation graphs N are sequentially input into the pre-trained recognition model based on the attention mechanism to obtain the segmentation Figure 1 , split Figure 2 ...Recognition results of segmented graph N: Result 1, Result 2... Result N;
[0087] Then, result 1, result 2, ... result N are sequentially connected to obtain the final target recognition result of picture a, that is, an editable form of the mathematical expression contained in picture a, such as text form.
[0088] The recognition method provided in the embodiment of this specification provides a segmentation model that can automatically segment long mathematical expressions in pictures, thereby achieving reasonable and efficient segmentation of longer mathematical expressions in pictures, thereby greatly reducing the recognition difficulty of the recognition model and improving the recognition performance of longer mathematical expressions.
[0089] See also Figure 2 , Figure 2 A flowchart of a first identification method provided according to an embodiment of this specification is shown, comprising the following steps:
[0090] Step 202: Obtain the image to be identified.
[0091] Among them, the pictures to be identified include but are not limited to pictures containing mathematical expressions, chemical expressions or physical expressions, that is, the pictures to be identified include mathematical expressions, chemical expressions or physical expressions.
[0092] In the specific implementation, the image to be identified is a screenshot, photo, etc. that only contains various expressions, and does not contain other non-expression content. In actual applications, in order to achieve faster and more accurate recognition of the content in the image to be identified, each image to be identified can contain only one content, that is, one expression, to avoid multiple expressions in one image to be identified, which would bring a burden to the subsequent expression recognition.
[0093] In practical applications, obtaining a picture to be identified can be understood as obtaining a picture to be identified that only contains a relatively long mathematical expression. Generally, a mathematical expression is about 2 centimeters long. If a mathematical expression is more than 8 centimeters long, it can be considered to be a relatively long mathematical expression.
[0094] Step 204: input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified.
[0095] Specifically, after obtaining the image to be identified, the image to be identified is input into a pre-trained segmentation model for segmentation to obtain multiple segmentation graphs of the image to be identified.
[0096] The training method of the segmentation model includes:
[0097] Get sample images;
[0098] Inputting the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result;
[0099] Preprocessing the sample image to obtain a projection image of the sample image;
[0100] Determine a segmentation point heat map of the sample image according to the first recognition result, the position image, and the image projection image;
[0101] The initial segmentation model is trained based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
[0102] Among them, the sample pictures include but are not limited to pictures containing various expressions, and in model training, the number of sample pictures is generally multiple; the first recognition result of the sample picture is the sequence of each element of the expression contained in the sample picture; the position image of the first recognition result is a heat map of each element of the expression contained in the sample picture, and the position of each element can be determined through the heat map.
[0103] For example, the expression contained in the sample image is: x+y=z, then the first recognition result of the sample image is [x, +, y, =, z], and the position image of the first recognition result is the heat map of each element in the first recognition result [x, +, y, =, z]. In practical applications, the heat map can be understood as an image that displays the first recognition result of the sample image in a special highlighted form.
[0104] Specifically, before outputting the first recognition result of the sample image and the position image of the first recognition result based on the recognition model, the recognition model needs to be pre-trained, and the specific implementation method is as follows:
[0105] The training method of the recognition model includes:
[0106] Obtain the sample image and the sample label corresponding to the sample image;
[0107] Construct an initial recognition model based on the attention mechanism;
[0108] The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
[0109] Among them, the sample label corresponding to the sample image is the specific content in the sample image, that is, the sequence of each element of the expression contained in the sample image, that is, the first recognition result of the above embodiment.
[0110] Specifically, multiple sample images and the sample labels corresponding to each sample image are first obtained, and then an initial recognition model based on the attention mechanism is constructed. The initial recognition model based on the attention mechanism is an encoder-decoder structure. The encoder uses ResNet or DenseNet as the backbone, and the decoder uses a conventional Bahdanau attention decoder.
[0111] After the initial recognition model is constructed, the initial recognition model is trained based on the sample images and the sample labels corresponding to each sample image to obtain a recognition model.
[0112] During specific implementation, a picture is input into the recognition model, and the recognition model can output the specific content in the picture, such as the first recognition result in the above embodiment.
[0113] In an embodiment of the present specification, an initial recognition model based on an attention mechanism is trained based on sample images and sample labels corresponding to each sample image to obtain a recognition model based on an attention mechanism. The recognition model can automatically align characters and structural positions on the image through the attention mechanism, and gradually recognize and output the characters and structural positions on the image. Therefore, the use of the pre-trained recognition model can more quickly and accurately recognize expressions in sample images.
[0114] In specific implementation, after obtaining the sample image, while obtaining the first recognition result of the sample image and the position image of the first recognition result based on the recognition model, the sample image is also preprocessed to obtain the image projection image of the sample image. The specific implementation method is as follows:
[0115] The preprocessing of the sample image to obtain the image projection image of the sample image includes:
[0116] Performing corrosion and binarization processing on the sample image to obtain a binarized sample image;
[0117] The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
[0118] Among them, the sample image is corroded, that is, the highlight area or white part in the sample image is reduced and refined. The sample image after corrosion will be much smaller than the original image, which will speed up the training speed of the subsequent segmentation model training; after the sample image is corroded, the corroded sample image is binarized, that is, the grayscale value of the pixel on the corroded sample image is set to 0 or 255, so that the corroded sample image presents an obvious black and white effect; then the pixel average value of the binarized sample image is taken in the horizontal direction to obtain the image projection map (i.e., horizontal projection map) of the sample image.
[0119] In the embodiments of the present specification, a series of graphics operations such as corrosion and binarization are performed on the sample image, and then the pixel mean is removed in the horizontal direction to obtain a horizontal projection image of the sample image. Subsequently, the segmentation points of the sample image are accurately determined through the horizontal projection image and the position image of the first recognition result, and the sample image is quickly and accurately segmented based on the segmentation points.
[0120] In specific implementation, after obtaining the first recognition result of the sample image, the position image of the first recognition result, and the image projection image, the segmentation point heat map of the sample image can be determined based on the first recognition result, the position image of the first recognition result, and the image projection image. The specific implementation method is as follows:
[0121] The step of determining the segmentation point heat map of the sample image according to the first recognition result, the position image and the image projection image includes:
[0122] determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images;
[0123] Determining the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image;
[0124] A segmentation point heat map of the sample image is determined according to the segmentation points.
[0125] Among them, the alternative characters include but are not limited to any character in the first recognition result, such as Chinese characters, commas, periods, etc.; taking the first recognition result of [x, +, y, =, z] as an example, the alternative characters can be at least one of the first recognition results, for example, the alternative characters are "+, =".
[0126] Specifically, each first recognition result corresponds to a position image. After the candidate characters are determined, the position of the selected candidate characters can be determined according to the position image corresponding to the first recognition result.
[0127] Before determining the candidate characters according to the first recognition result, the accuracy of the first recognition result needs to be confirmed to ensure the accuracy of the first recognition result of the sample image output by the recognition model, so as to accurately train the segmentation model in the subsequent step. The specific implementation method is as follows:
[0128] Determining candidate characters according to the first recognition result includes:
[0129] Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
[0130] In practical applications, only when the first recognition result of the sample image output by the recognition model is accurate, the alternative characters selected based on the first recognition result and the segmentation of the sample image based on the alternative characters will be accurate and valuable. If the first recognition result is inaccurate, then the subsequent alternative characters selected based on the first recognition result and the segmentation of the sample image based on the alternative characters will also produce errors, which will ultimately cause the training of the segmentation model to be biased and inaccurate. When the first recognition result is not the target recognition result of the sample image (that is, the real content), the processing flow is terminated and the recognition model is continued to be trained. While ensuring the recognition accuracy of the recognition model, the training process of the segmentation model is continued to ensure the safety and accuracy of the overall training process of the segmentation model.
[0131] Specifically, the first recognition result may be determined manually, or the content of the sample image may be recognized through other recognition models, and the recognition results may be compared two or three times to determine the accuracy of the first recognition result.
[0132] In specific implementation, when it is determined that the first recognition result is the target recognition result of the sample image, alternative characters are selected from the first recognition result, and then the position of each alternative character is determined based on the position image, wherein the alternative characters can be selected according to the actual application, and the specific number of selections and which one is selected as the alternative character are not limited here; in actual applications, the selection of alternative characters can be diversified, and if the sample image is a mathematical expression, in theory, the characters in the primary structure of the mathematical expression can be used as alternative characters, for example, if the mathematical expression is x / y+y=z, then the primary structure is: x / y, +, y, =, z, and " / " is the secondary structure.
[0133] Finally, the segmentation point of the sample image is determined based on the correlation between the position of the candidate characters and the image projection map, and then the segmentation point heat map of the sample image is determined based on the segmentation point. In practical applications, the segmentation point heat map has a height of 1 and a width of one-fourth of the original sample image. The value at the segmentation point position is 1, and the value at other positions is 0.
[0134] Specifically, the determining of the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image includes:
[0135] The position image of the candidate characters is overlapped with the picture projection image, and the last breakpoint of the overlapping portion is determined as the segmentation point of the sample picture.
[0136] In specific implementation, the position image of the candidate characters is determined, the position image of the candidate characters is overlapped with the image projection image, and the last breakpoint of the overlapped part (ie, the point with a pixel mean value of 255) is used as the segmentation point of the sample image.
[0137] In the embodiments of this specification, based on the mutual restriction between the position image of the alternative character and the picture projection map, the splitting point of the sample picture is determined based on the coincidence relationship, making the acquisition of the splitting point more accurate.
[0138] After determining the splitting point, the splitting point heat map of the sample picture can be determined based on this splitting point, and then the initial splitting model is trained based on the sample picture and the splitting point heat map of each sample picture to obtain the splitting model.
[0139] And when the splitting point heat map is determined, the sample picture can also be split based on the splitting point heat map to obtain the split picture of the sample picture, and its specific implementation method is as follows:
[0140] Determine the split picture of the sample picture according to the splitting point heat map.
[0141] See Figure 3 , Figure 3 which is a specific schematic diagram of the splitting point, the splitting point heat map, and the split picture in the processing process of an identification method provided by an embodiment of this specification.
[0142] Figure 3 In, the line after the mathematical expression "then" is the splitting point, the bottom line of the mathematical expression is the splitting point heat map, and the picture before the splitting point is the first split picture of the mathematical expression, and the picture after the splitting point is the second split picture of the mathematical expression.
[0143] In the embodiments of this specification, in the processing flow of training the splitting model, based on the joint verification of the position image of the first recognition result of the sample picture and the picture horizontal pixel value projection map (i.e., the picture projection map), a suitable splitting point is selected to split the sample picture containing the expression, and the splitting point heat map is determined for training the splitting model, so that the trained splitting model can effectively avoid the problem of attention drift when splitting the picture containing the expression in the future, making the splitting of the picture faster and more accurate.
[0144] In another embodiment of this specification, after determining the split picture of the sample picture according to the splitting point heat map, it further includes:
[0145] Determine the local recognition result in the split picture based on the position image of the first recognition result;
[0146] Determine the local recognition result in the split picture as the sample label corresponding to the split picture.
[0147] Specifically, the sample labels corresponding to the sample images are a sequence. Before the sample images are recognized by the recognition model to obtain the first recognition result, the positions of the sample labels corresponding to the sample images in the sample images are uncertain. After the sample images are divided into multiple segmented images, the labels corresponding to each segmented image are also uncertain.
[0148] In specific implementation, after obtaining the first recognition result of the sample image and the position image of the first recognition result based on the recognition model, the local recognition result of each cut graph of the sample image can be determined based on the position image of the first recognition result, that is, which first recognition results exist on each cut graph; and then the local recognition result on each cut graph is used as the sample label corresponding to the cut graph.
[0149] Still taking the mathematical expression in the sample image as: x+y=z as an example, if the alternative character is "=", the first segmentation graph of the sample image is: x+y=, and the second segmentation graph is: z, then it can be determined that the local recognition result in the first segmentation graph is: x, +, y, =, and the local recognition result in the second segmentation graph is: z, then x, +, y, = are determined as the sample labels corresponding to the first segmentation graph, and z is determined as the sample label of the second segmentation graph.
[0150] In the embodiment of the present specification, the position image based on the first recognition result of the sample image can quickly and accurately determine the corresponding sample label for the cut map of the sample image, so that the recognition model can be further trained based on the cut map and the sample label corresponding to each cut map.
[0151] Specifically, after obtaining the recognition model, the method further includes:
[0152] The recognition model is continuously trained based on the segmentation graph and the sample labels corresponding to the segmentation graph.
[0153] In an embodiment of the present specification, after obtaining a segmentation graph of a sample image, the segmentation graph and the sample labels corresponding to the segmentation graph can be added to the training samples of a recognition model to achieve continued training of the recognition model. Continuing training of the recognition model through the segmentation graph and the corresponding sample labels effectively enriches the semantic richness of the recognition model, thereby improving the recognition accuracy of the recognition model.
[0154] Step 206: Input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph.
[0155] Specifically, the recognition model of the embodiment of the present specification may be a pre-trained recognition model when used for the first time, and may be a re-trained recognition model based on the segmentation graph and the sample labels corresponding to the segmentation graph when used subsequently.
[0156] During specific implementation, each segmentation graph is sequentially input into the recognition model to obtain the initial recognition result of each segmentation graph sequentially.
[0157] Step 208: Determine a target recognition result of the image to be recognized based on the initial recognition result.
[0158] In practical applications, the embodiments of this specification are aimed at images containing relatively long mathematical expressions. Therefore, when the image to be identified is segmented by the segmentation model, the image to be identified will be segmented into at least two or more segmentation graphs, and then the initial recognition result of each segmentation graph is obtained through the recognition model.
[0159] Then the final recognition result of the image to be recognized is formed by combining the initial recognition results of all the segmented images after segmentation. The specific implementation method is as follows:
[0160] Determining the target recognition result of the image to be recognized based on the initial recognition result includes:
[0161] The initial recognition results are sequentially connected to generate a target recognition result of the image to be recognized.
[0162] Specifically, after obtaining the initial recognition result of each segmented image, each initial recognition result is sequentially connected in turn to form a final target recognition result of the image to be recognized.
[0163] The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation map of the image to be identified. Specifically, although the segmentation model uses sample images and segmentation point heat maps corresponding to the sample images during training, in specific applications, the output is a segmentation map of the image to be identified after segmentation according to the segmentation point heat map.
[0164] The recognition model is suitable for recognizing the segmentation graph and outputting an initial recognition result of the segmentation graph.
[0165] The recognition method provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0166] See also Figure 4 , Figure 4A flow chart of a segmentation model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps:
[0167] Step 402: Obtain a sample image.
[0168] The sample images include but are not limited to images containing mathematical expressions, chemical expressions or physical expressions.
[0169] Step 404: input the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result.
[0170] Step 406: pre-process the sample image to obtain a projection image of the sample image.
[0171] Step 408: Determine a segmentation point heat map of the sample image according to the first recognition result, the position image, and the image projection image.
[0172] Step 410: Train an initial segmentation model based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs a segmentation map of the sample image.
[0173] Optionally, the training method of the recognition model includes:
[0174] Obtain the sample image and the sample label corresponding to the sample image;
[0175] Construct an initial recognition model based on the attention mechanism;
[0176] The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
[0177] Optionally, determining the segmentation point heat map of the sample image according to the first recognition result, the position image, and the image projection image includes:
[0178] determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images;
[0179] Determining the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image;
[0180] A segmentation point heat map of the sample image is determined according to the segmentation points.
[0181] Optionally, determining candidate characters according to the first recognition result includes:
[0182] Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
[0183] Optionally, the determining the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image includes:
[0184] The position image of the candidate characters is overlapped with the picture projection image, and the last breakpoint of the overlapping portion is determined as the segmentation point of the sample picture.
[0185] Optionally, the preprocessing the sample image to obtain a picture projection image of the sample image includes:
[0186] Performing corrosion and binarization processing on the sample image to obtain a binarized sample image;
[0187] The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
[0188] Optionally, after determining the segmentation point heat map of the sample image according to the segmentation points, the method further includes:
[0189] A segmentation map of the sample image is determined according to the segmentation point heat map.
[0190] Optionally, after determining the segmentation map of the sample image according to the segmentation point heat map, the method further includes:
[0191] Determine a local recognition result in the segmentation image based on the position image of the first recognition result;
[0192] The local recognition result in the segmentation graph is determined as the sample label corresponding to the segmentation graph.
[0193] Optionally, after obtaining the recognition model, the method further includes:
[0194] The recognition model is continuously trained based on the segmentation graph and the sample labels corresponding to the segmentation graph.
[0195] The technical solution of the segmentation model training method provided in the embodiment of this specification belongs to the same concept as the technical solution of the segmentation model training method in the recognition method in the above-mentioned embodiment. The details not described in detail in the technical solution of the segmentation model training method provided in this embodiment can be found in the description of the technical solution of the segmentation model training method in the recognition method in the above-mentioned embodiment, and will not be repeated here.
[0196] The training method of the segmentation model provided in the embodiments of this specification realizes a method for selecting segmentation points of long expressions based on image information and attention maps (i.e., position images of content in images), as well as a method for realizing image segmentation based on the segmentation points. The segmentation points can be automatically extracted from long expressions in training sample images to divide long expressions into a series of semantically complete sub-expressions, so that the segmentation model after subsequent training can reasonably and efficiently segment images containing long expressions, thereby greatly reducing the recognition difficulty of the attention mechanism and improving the recognition performance of long expressions.
[0197] See also Figure 5 , Figure 5 A schematic diagram of a specific processing process of an identification method provided according to an embodiment of the present specification is shown.
[0198] Step 1: Train a recognition model based on the attention mechanism based on the input image and the corresponding label of the input image, so that the recognition model can output the recognition result of each input image and the attention map of each recognition result (i.e., the position image of the above embodiment).
[0199] Step 2: Analyze the segmentation points based on the input images, the corresponding labels of each input image, the recognition results, and the attention map of the recognition results to determine the segmentation points.
[0200] Step 3: Segment the sample image according to the segmentation point and obtain the segmentation Figure 1 , split Figure 2 And the segmentation point heat map.
[0201] Step 4: Based on segmentation Figure 1 , split Figure 2 The attention map of the recognition result in determines the segmentation Figure 1 Realignment label 1, segmentation Figure 2 Realign the labels 2.
[0202] Step 5: Based on segmentation Figure 1 , realign label 1 and the segmentation map, re-label it 2 to continue training the recognition model based on the attention mechanism, and train the pre-segmentation network based on the input image and the segmentation point heat map to obtain the segmentation model.
[0203] The recognition method provided in the embodiment of this specification does not require character-level supervision of the expressions contained in the input image. The labels of the segmented image are generated by realigning the labels of the original input image, which greatly reduces the amount of annotation. The recognition method in the embodiment of this specification does not require modification of the modeling of the attention mechanism, and directly solves the problem fundamentally from the data level. A pre-segmentation network (i.e., a segmentation model) that is independent of the recognition model is proposed to automatically segment long expressions in the input image, thereby solving the problem of difficulty in recognizing long expressions in the input image and ensuring the performance of the attention mechanism on short expressions. In the training set of input images containing long data expressions, the recognition method in the embodiment of this specification automatically mines long samples that can be segmented in the training set and iterates training, which can eventually improve long expressions with more than 20 characters by 1.5% to 2%.
[0204] See also Figure 6 , Figure 6 A flowchart of determining a segmentation point heat map and realigning labels in a recognition method provided according to an embodiment of the present specification is shown, which specifically includes the following steps:
[0205] Step 602: Input a picture.
[0206] Here, pictures can be understood as pictures containing long mathematical expressions.
[0207] Step 604: Obtain a horizontal projection of the image.
[0208] Specifically, graphics operations such as corrosion and binarization are performed on the input image, and the pixel values of the binarized image are averaged in the horizontal direction to obtain a horizontal projection of the image (ie, the image projection of the above embodiment).
[0209] Step 606: Input the image into the trained recognition model to obtain the recognition result of the image and the attention map of the recognition result.
[0210] Step 608: Determine whether the recognition result of the image is correct, if so, execute step 610.
[0211] Step 610: Select candidate characters based on the recognition result of the image.
[0212] Step 612: Determine the positions of the candidate characters and the attention maps of the candidate characters based on the attention map of the recognition result.
[0213] Step 614: Determine the segmentation point according to the attention map and the horizontal projection map of the candidate characters.
[0214] Specifically, the breakpoints on the horizontal projection image that are covered by the attention maps of the candidate characters are used as segmentation points.
[0215] Step 616: segment the image based on the segmentation points and generate a segmentation point heat map.
[0216] Step 618: Obtain a segmentation map and a segmentation point heat map of the image.
[0217] Step 620: realign the labels according to the segmentation graph and the recognition results of the image.
[0218] Step 622: Obtain the realignment label of the segmented graph.
[0219] In the embodiments of the present specification, the segmentation points of the image are accurately determined based on the horizontal projection map, attention map, recognition results and labels of the image, and then the segmentation map, segmentation point heat map and realignment labels of the image are obtained according to the segmentation points. Subsequently, the segmentation model can be trained based on the image and the corresponding segmentation point heat map to ensure that the segmentation model obtained by training can accurately segment long mathematical expressions in the image, and the recognition model can be trained based on the segmentation map and the corresponding realignment labels to effectively enrich the semantics of the original recognition model.
[0220] See also Figure 7 , Figure 7 A flowchart of a second identification method provided according to an embodiment of this specification is shown, comprising the following steps:
[0221] Step 702: Displaying an image input interface to the user based on the user's call request.
[0222] Specifically, when receiving a call request from a user, an image input interface is determined according to the call request, and the image input interface is displayed to the user, and the user can input a picture to be identified through the image input interface.
[0223] In actual applications, users include but are not limited to merchants or individual users.
[0224] Step 704: receiving the image to be identified input by the user based on the image input interface.
[0225] Step 706: Input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified.
[0226] Step 708: Input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph.
[0227] Step 710: Determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0228] It should be noted that the technical solution of the second identification method provided in the embodiment of this specification belongs to the same concept as the technical solution of the first identification method mentioned above. The details not described in detail in the technical solution of the identification method provided in this embodiment can be found in the description of the technical solution of the identification method in the above embodiment and will not be repeated here.
[0229] The recognition method provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0230] See also Figure 8 , Figure 8 A flowchart of a third identification method provided according to an embodiment of this specification is shown, comprising the following steps:
[0231] Step 802: Receive a call request sent by a user, wherein the call request carries a picture to be identified.
[0232] Step 804: input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified.
[0233] Step 806: Input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph.
[0234] Step 808: Determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0235] In actual applications, the recognition method is applied to the local server and an API interface is provided to the user. After receiving the API call request sent by the user, based on the image to be recognized carried in the user's call request, the target recognition result of the image to be recognized is quickly and accurately obtained through the recognition method, and the target recognition result is returned to the user to improve the user experience.
[0236] It should be noted that the technical solution of the third identification method provided in the embodiment of this specification belongs to the same concept as the technical solution of the first identification method mentioned above. The details not described in detail in the technical solution of the identification method provided in this embodiment can be found in the description of the technical solution of the identification method in the above embodiment and will not be repeated here.
[0237] The recognition method provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0238] Corresponding to the above method embodiment, this specification also provides a first identification device embodiment, Fig. 9 FIG. 1 shows a schematic diagram of the structure of a first identification device provided by an embodiment of this specification. Fig. 9 As shown, the device comprises:
[0239] A first acquisition module 902 is configured to acquire a picture to be identified;
[0240] The first obtaining module 904 is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0241] A second obtaining module 906 is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0242] A first determination module 908 is configured to determine a target recognition result of the image to be recognized based on the initial recognition result;
[0243] The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation graph of the image to be identified.
[0244] The recognition model is suitable for recognizing the segmentation graph and outputting an initial recognition result of the segmentation graph.
[0245] Optionally, the first obtaining module 904 is further configured to train the segmentation model, and the training method of the segmentation model includes:
[0246] Get sample images;
[0247] Inputting the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result;
[0248] Preprocessing the sample image to obtain a projection image of the sample image;
[0249] Determine a segmentation point heat map of the sample image according to the first recognition result, the position image, and the image projection image;
[0250] The initial segmentation model is trained based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
[0251] Optionally, the second obtaining module 906 is further configured to train the recognition model, and the training method of the recognition model includes:
[0252] Obtain the sample image and the sample label corresponding to the sample image;
[0253] Construct an initial recognition model based on the attention mechanism;
[0254] The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
[0255] Optionally, the second obtaining module 906 is further configured to:
[0256] determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images;
[0257] Determining the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image;
[0258] A segmentation point heat map of the sample image is determined according to the segmentation points.
[0259] Optionally, the second obtaining module 906 is further configured to:
[0260] Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
[0261] Optionally, the second obtaining module 906 is further configured to:
[0262] The position image of the candidate characters is overlapped with the picture projection image, and the last breakpoint of the overlapping portion is determined as the segmentation point of the sample picture.
[0263] Optionally, the second obtaining module 906 is further configured to:
[0264] Performing corrosion and binarization processing on the sample image to obtain a binarized sample image;
[0265] The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
[0266] Optionally, the device further includes:
[0267] The fourth determination module is configured to determine the segmentation map of the sample image according to the segmentation point heat map.
[0268] Optionally, the device further includes:
[0269] A fifth determination module is configured to determine a local recognition result in the segmentation image based on the position image of the first recognition result;
[0270] The sixth determination module is configured to determine the local recognition result in the segmentation graph as the sample label corresponding to the segmentation graph.
[0271] Optionally, the device further includes:
[0272] The first training module is configured to continue training the recognition model based on the segmentation graph and the sample labels corresponding to the segmentation graph.
[0273] Optionally, the first determining module 908 is further configured to:
[0274] The initial recognition results are sequentially connected to generate a target recognition result of the image to be recognized.
[0275] Optionally, the image to be identified includes a mathematical expression, a chemical expression or a physical expression.
[0276] The recognition device provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0277] The above is a schematic scheme of the first identification device of this embodiment. It should be noted that the technical scheme of the identification device and the technical scheme of the first identification method mentioned above are of the same concept, and the details of the technical scheme of the identification device not described in detail can be referred to the description of the technical scheme of the first identification method mentioned above.
[0278] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for a segmentation model. Fig.10 FIG. 1 is a schematic diagram showing a structure of a training device for a segmentation model provided by an embodiment of the present specification. Fig.10 As shown, the device comprises:
[0279] The second acquisition module 1002 is configured to acquire a sample image;
[0280] The third obtaining module 1004 is configured to input the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result;
[0281] The fourth obtaining module 1006 is configured to pre-process the sample image to obtain a picture projection image of the sample image;
[0282] A second determination module 1008 is configured to determine a segmentation point heat map of the sample image according to the first recognition result, the position image and the image projection image;
[0283] The fifth obtaining module 1010 is configured to train an initial segmentation model based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs a segmentation map of the sample image.
[0284] Optionally, the third obtaining module 1004 is further configured to train the recognition model, and the training method of the recognition model includes:
[0285] Obtain the sample image and the sample label corresponding to the sample image;
[0286] Construct an initial recognition model based on the attention mechanism;
[0287] The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
[0288] Optionally, the second determining module 1008 is further configured to:
[0289] determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images;
[0290] Determining the segmentation points of the sample image based on the association relationship between the position image of the candidate characters and the image projection image;
[0291] A segmentation point heat map of the sample image is determined according to the segmentation points.
[0292] Optionally, the second determining module 1008 is further configured to:
[0293] Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
[0294] Optionally, the second determining module 1008 is further configured to:
[0295] The position image of the candidate characters is overlapped with the picture projection image, and the last breakpoint of the overlapping portion is determined as the segmentation point of the sample picture.
[0296] Optionally, the fourth obtaining module 1006 is configured to:
[0297] Performing corrosion and binarization processing on the sample image to obtain a binarized sample image;
[0298] The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
[0299] Optionally, the device further includes:
[0300] The seventh determination module is configured to determine the segmentation map of the sample image according to the segmentation point heat map.
[0301] Optionally, the device further includes:
[0302] An eighth determination module is configured to determine a local recognition result in the segmentation image based on the position image of the first recognition result;
[0303] The ninth determination module is configured to determine the local recognition result in the segmentation graph as the sample label corresponding to the segmentation graph.
[0304] Optionally, the device further includes:
[0305] The second training module is configured to continue training the recognition model based on the segmentation graph and the sample labels corresponding to the segmentation graph.
[0306] The training device for the segmentation model provided in the embodiment of the present specification realizes a segmentation point selection for long expressions based on image information and an attention map (i.e., a position image of the content in the image), and a method for image segmentation based on the segmentation points. The segmentation points can be automatically extracted from the long expressions in the training sample images to divide the long expressions into a series of semantically complete sub-expressions, so that the segmentation model after subsequent training can reasonably and efficiently segment the images containing long expressions, thereby greatly reducing the recognition difficulty of the attention mechanism and improving the recognition performance of long expressions.
[0307] The above is a schematic scheme of a training device for a segmentation model of this embodiment. It should be noted that the technical scheme of the training device for the segmentation model and the technical scheme of the training method for the segmentation model described above belong to the same concept, and the details of the technical scheme of the training device for the segmentation model that are not described in detail can all be referred to the description of the technical scheme of the training method for the segmentation model described above.
[0308] Corresponding to the above method embodiment, this specification also provides a second identification device embodiment, Fig.11 FIG. 2 shows a schematic diagram of the structure of a second identification device provided by an embodiment of this specification. Fig.11 As shown, the device comprises:
[0309] The interface display module 1102 is configured to display an image input interface to the user based on a user's call request;
[0310] The first receiving module 1104 is configured to receive the image to be recognized input by the user based on the image input interface;
[0311] The sixth obtaining module 1106 is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0312] A seventh obtaining module 1108 is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0313] The third determination module 1110 is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0314] The recognition device provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0315] The above is a schematic scheme of the second identification device of this embodiment. It should be noted that the technical scheme of the identification device and the technical scheme of the second identification method mentioned above are of the same concept, and the details of the technical scheme of the identification device not described in detail can be referred to the description of the technical scheme of the second identification method mentioned above.
[0316] Corresponding to the above method embodiment, this specification also provides a third identification device embodiment, Fig.12 FIG. 2 shows a schematic diagram of the structure of a third identification device provided by an embodiment of this specification. Fig.12 As shown, the device comprises:
[0317] The second receiving module 1202 is configured to receive a call request sent by a user, wherein the call request carries a picture to be identified;
[0318] An eighth obtaining module 1204 is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified;
[0319] A ninth obtaining module 1206 is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph;
[0320] The third determination module 1208 is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user.
[0321] The recognition device provided in the embodiment of the present specification realizes a scheme for automatically finding segmentation points for images containing long expressions and segmenting them into segmentation graphs, and determining labels (i.e., realignment labels) for the segmentation graphs. The segmentation graphs and realignment labels obtained by the method are used to continue training the recognition model, which can effectively enrich the semantics of the original recognition model. In addition, the recognition method proposes a segmentation model, which is trained using a determined segmentation point heat map. Subsequently, based on the trained segmentation model, the information of the segmentation points in the image can be accurately determined to achieve fast and accurate segmentation of the image.
[0322] The above is a schematic scheme of the third identification device of this embodiment. It should be noted that the technical scheme of the identification device and the technical scheme of the third identification method mentioned above are of the same concept, and the details not described in detail in the technical scheme of the identification device can be referred to the description of the technical scheme of the third identification method mentioned above.
[0323] Fig.13 The block diagram of a computing device 1300 according to one embodiment of the present specification is shown. The components of the computing device 1300 include but are not limited to a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and the database 1350 is used to store data.
[0324] The computing device 1300 also includes an access device 1340 that enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1340 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0325] In one embodiment of the present specification, the above components of the computing device 1300 and Fig.13 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.13 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0326] Computing device 1300 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1300 may also be a mobile or stationary server.
[0327] The processor 1320 is used to execute the following computer executable instructions, wherein the processor implements the steps of the recognition method or the training method of the segmentation model when executing the computer executable instructions.
[0328] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned recognition method or the training method of the segmentation model belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned recognition method or the training method of the segmentation model.
[0329] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the recognition method or the training method of the segmentation model.
[0330] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned recognition method or the training method of the segmentation model belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned recognition method or the training method of the segmentation model.
[0331] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0332] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0333] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0334] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0335] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. An identification method, comprising: Get the image to be identified; Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; Determining a target recognition result of the image to be recognized based on the initial recognition result; The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation graph of the image to be identified. The recognition model is suitable for recognizing the segmentation graph and outputting an initial recognition result of the segmentation graph; The segmentation model is obtained by training an initial segmentation model based on a sample image and a segmentation point heat map. The segmentation point heat map is determined by segmenting the sample image based on the segmentation points. The segmentation points are selected by jointly verifying the position image and the image projection map of the first recognition result of the sample image.
2. According to the recognition method of claim 1, the training method of the segmentation model comprises: Get sample images; Inputting the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result; Preprocessing the sample image to obtain a projection image of the sample image; determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images; Overlapping the position image of the candidate character with the picture projection image, and determining the last breakpoint of the overlapping portion as the segmentation point of the sample picture; Determine a segmentation point heat map of the sample image according to the segmentation points; The initial segmentation model is trained based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
3. According to the recognition method of claim 2, the training method of the recognition model comprises: Obtaining sample images and sample labels corresponding to the sample images; Construct an initial recognition model based on the attention mechanism; The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
4. The recognition method according to claim 2, wherein determining candidate characters according to the first recognition result comprises: Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
5. According to the recognition method of claim 2, the preprocessing of the sample image to obtain the image projection image of the sample image comprises: Performing corrosion and binarization processing on the sample image to obtain a binarized sample image; The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
6. The recognition method according to claim 2, after determining the segmentation point heat map of the sample image according to the segmentation points, further comprising: A segmentation map of the sample image is determined according to the segmentation point heat map.
7. The recognition method according to claim 6, after determining the segmentation map of the sample image according to the segmentation point heat map, further comprising: Determine a local recognition result in the segmentation image based on the position image of the first recognition result; The local recognition result in the segmentation graph is determined as the sample label corresponding to the segmentation graph.
8. The recognition method according to claim 7, after obtaining the recognition model, further comprising: The recognition model is continuously trained based on the segmentation graph and the sample labels corresponding to the segmentation graph.
9. The recognition method according to claim 1, wherein determining the target recognition result of the image to be recognized based on the initial recognition result comprises: The initial recognition results are sequentially connected to generate a target recognition result of the image to be recognized. 10 . The recognition method according to claim 1 , wherein the image to be recognized includes a mathematical expression, a chemical expression or a physical expression.
11. A training method for a segmentation model, comprising: Get sample images; Inputting the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result; Preprocessing the sample image to obtain a projection image of the sample image; determining candidate characters according to the first recognition result, and determining position images of the candidate characters based on the position images; Overlapping the position image of the candidate character with the picture projection image, and determining the last breakpoint of the overlapping portion as the segmentation point of the sample picture; Determine a segmentation point heat map of the sample image according to the segmentation points; The initial segmentation model is trained based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
12. The segmentation model training method according to claim 11, wherein the recognition model training method comprises: Obtain the sample image and the sample label corresponding to the sample image; Construct an initial recognition model based on the attention mechanism; The initial recognition model is trained based on the sample image and the sample label corresponding to the sample image to obtain the recognition model, wherein the recognition model outputs a first recognition result of the sample image and a position image of the first recognition result.
13. The segmentation model training method according to claim 11, wherein determining candidate characters according to the first recognition result comprises: Determine whether the first recognition result is the target recognition result of the sample image, and if so, determine the candidate characters according to the first recognition result.
14. According to the training method of the segmentation model of claim 11, the preprocessing of the sample image to obtain the image projection image of the sample image comprises: Performing corrosion and binarization processing on the sample image to obtain a binarized sample image; The pixel average value of the binarized sample image is taken in the horizontal direction to obtain a picture projection image of the sample image.
15. The method for training a segmentation model according to claim 11, after determining the segmentation point heat map of the sample image according to the segmentation points, further comprising: A segmentation map of the sample image is determined according to the segmentation point heat map.
16. The segmentation model training method according to claim 15, after determining the segmentation map of the sample image according to the segmentation point heat map, further comprising: Determine a local recognition result in the segmentation image based on the position image of the first recognition result; The local recognition result in the segmentation graph is determined as the sample label corresponding to the segmentation graph.
17. The segmentation model training method according to claim 16, after obtaining the recognition model, further comprising: The recognition model is continuously trained based on the segmentation graph and the sample labels corresponding to the segmentation graph.
18. A recognition method comprising: Displaying an image input interface to the user based on a call request from the user; Receiving a picture to be identified input by the user based on the image input interface; Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; Determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user; Among them, the segmentation model is obtained by training the initial segmentation model according to the sample image and the segmentation point heat map, the segmentation point heat map is determined by segmenting the sample image according to the segmentation points, and the segmentation points are selected by jointly verifying the position image and the picture projection map of the first recognition result of the sample image.
19. A recognition method comprising: Receiving a call request sent by a user, wherein the call request carries a picture to be identified; Inputting the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; Inputting the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; Determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user; Among them, the segmentation model is obtained by training the initial segmentation model according to the sample image and the segmentation point heat map, the segmentation point heat map is determined by segmenting the sample image according to the segmentation points, and the segmentation points are selected by jointly verifying the position image and the picture projection map of the first recognition result of the sample image.
20. An identification device, comprising: A first acquisition module is configured to acquire a picture to be identified; A first obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; A second obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; A first determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result; The segmentation model is suitable for segmenting the image to be identified and outputting a segmentation graph of the image to be identified. The recognition model is suitable for recognizing the segmentation graph and outputting the initial recognition result of the segmentation graph. The segmentation model is obtained by training an initial segmentation model based on a sample image and a segmentation point heat map. The segmentation point heat map is determined by segmenting the sample image based on the segmentation points. The segmentation points are selected by jointly verifying the position image and the image projection map of the first recognition result of the sample image.
21. A training device for a segmentation model, comprising: A second acquisition module is configured to acquire a sample image; A third obtaining module is configured to input the sample image into a recognition model to obtain a first recognition result of the sample image and a position image of the first recognition result; A fourth obtaining module is configured to pre-process the sample image to obtain a picture projection image of the sample image; A second determination module is configured to determine candidate characters according to the first recognition result, and determine a position image of the candidate characters based on the position image; Overlapping the position image of the candidate character with the picture projection image, and determining the last breakpoint of the overlapping portion as the segmentation point of the sample picture; Determine a segmentation point heat map of the sample image according to the segmentation points; The fifth acquisition module is configured to train the initial segmentation model based on the sample image and the segmentation point heat map of the sample image to obtain the segmentation model, wherein the segmentation model outputs the segmentation map of the sample image.
22. An identification device, comprising: An interface display module, configured to display an image input interface to the user based on a user's call request; A first receiving module is configured to receive a picture to be identified input by the user based on the image input interface; A sixth obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; A seventh obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; A third determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user; Among them, the segmentation model is obtained by training the initial segmentation model according to the sample image and the segmentation point heat map, the segmentation point heat map is determined by segmenting the sample image according to the segmentation points, and the segmentation points are selected by jointly verifying the position image and the picture projection map of the first recognition result of the sample image.
23. An identification device, comprising: A second receiving module is configured to receive a call request sent by a user, wherein the call request carries a picture to be identified; An eighth obtaining module is configured to input the image to be identified into a segmentation model to obtain a segmentation graph of the image to be identified; A ninth obtaining module is configured to input the segmentation graph into a recognition model to obtain an initial recognition result of the segmentation graph; A third determination module is configured to determine a target recognition result of the image to be recognized based on the initial recognition result, and return the target recognition result to the user; Among them, the segmentation model is obtained by training the initial segmentation model according to the sample image and the segmentation point heat map, the segmentation point heat map is determined by segmenting the sample image according to the segmentation points, and the segmentation points are selected by jointly verifying the position image and the picture projection map of the first recognition result of the sample image.
24. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein when the processor executes the computer-executable instructions, it implements the recognition method of any one of claims 1-10, the training method of the segmentation model of any one of claims 11-17, the recognition method of claim 18, or the steps of the recognition method of claim 19.
25. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the recognition method described in any one of claims 1 to 10, the training method of the segmentation model described in any one of claims 11 to 17, the recognition method of claim 18, or the steps of the recognition method of claim 19.
