A method and device for recognizing mathematical symbols in ancient books

Abstract

The present invention discloses a method and device for recognizing mathematical symbols in ancient books. The method comprises: performing an opening operation on an original image based on pixel values ​​to obtain a first image; performing a first convolution process on the first image using a first convolution kernel to obtain a first processing result; adjusting the first image based on the text recognized by the first processing result to obtain a second image; performing a second convolution process on the second image using a second convolution kernel to obtain a second processing result; determining the position and relevance of mathematical symbols based on the first processing result and the second processing result; the receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel, and the edge weight of the first convolution kernel is larger than the internal weight, and the edge weight of the second convolution kernel is smaller than the internal weight; and when the convolution result of the first convolution kernel and the second convolution kernel is lower than the threshold value, the convolution result is set to 0. The present invention can accurately recognize the text and mathematical symbols in ancient book images.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and device for recognizing mathematical symbols in ancient books. Background Art

[0002] Ancient books on mathematics contain a large number of ancient mathematical symbols. Existing mathematical symbol recognition methods can usually only recognize modern mathematical symbols. Since these ancient mathematical symbols are different from modern mathematical symbols, they cannot be recognized by existing mathematical symbol recognition methods. Summary of the invention

[0003] At least one aspect or advantage of the invention will be set forth in part in the following description, or may be obvious from the description, or may be learned by practice of the disclosed subject matter.

[0004] According to a first aspect of the present invention, there is provided a method for recognizing mathematical symbols in ancient books, comprising:

[0005] Performing an opening operation on the original image based on pixel values ​​to obtain a first image;

[0006] Performing a first convolution process on the first image using a first convolution kernel, and recognizing the image after the first convolution process to obtain a first processing result;

[0007] Adjusting the first image based on the text recognized by the first processing result so that the recognition result included in the first processing result is masked to obtain a second image;

[0008] Performing a second convolution process on the second image using a second convolution kernel, and recognizing the image after the second convolution process to obtain a second processing result;

[0009] determining the location and relevance of the mathematical symbol based on the first processing result and the second processing result;

[0010] The receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel, and the edge weight of the first convolution kernel is larger than the internal weight, and the edge weight of the second convolution kernel is smaller than the internal weight;

[0011] When the convolution result of the first convolution kernel and the second convolution kernel is lower than the threshold value, the convolution result is set to 0.

[0012] According to an embodiment of the present invention, based on the statistics of the pixels of the original image, the pixels with the largest proportion are determined as the background;

[0013] Remove the background of the original image to obtain a first intermediate image;

[0014] Based on pixel statistics of the first intermediate image, determine the pixel with the largest proportion as the text color;

[0015] Taking the text color as the target image area, an opening operation is performed on the first intermediate image to obtain a first image.

[0016] According to one embodiment of the present invention, the size of the first convolution kernel is at least 5*5; the size of the second convolution kernel does not exceed 5*5.

[0017] According to one embodiment of the present invention, a first text sequence and a position of the text in the image are obtained based on the first processing result;

[0018] determining a distribution of the descriptive text in the image based on a position of the first text sequence in the image;

[0019] Obtaining distribution of a second character sequence and a corresponding character sequence in the image based on the second processing result;

[0020] Determining boundaries of a plurality of groups of character sequences contained in the image based on distribution of the second character sequence and the corresponding character sequence in the image;

[0021] Determining a plurality of groups of regions containing mathematical symbols based on the boundaries of a plurality of groups of text sequences;

[0022] Based on the second text sequence contained in the plurality of groups of regions, a replacement character for replacing the text sequence is determined.

[0023] According to one embodiment of the present invention, in response to the dictionary determining that the semantic attribute of the character of the second text sequence is a function character, the corresponding character is replaced with a function corresponding to the dictionary;

[0024] In response to the character semantics of the second text sequence determined by the dictionary to be variable characters, characters that are different from functions are used to replace corresponding characters.

[0025] According to one embodiment of the present invention, based on the image area corresponding to the first processing result and the second processing result, an unrecognized image area in the first image is determined;

[0026] Based on the first image recognition result and the second image recognition result, the recognized area is removed to obtain a third image;

[0027] Cropping the third image to obtain a plurality of sub-images to be identified;

[0028] Processing each sub-image to obtain a recognition result of the mathematical symbols in the sub-image;

[0029] The recognition result of the mathematical symbol in the sub-image includes the implicit mathematical symbol and its offset relative to the origin of the first image.

[0030] According to one embodiment of the present invention, based on the distribution of the second text sequence and the corresponding text sequence in the image, boundaries of several groups of text sequences contained in the image are determined;

[0031] Determining a plurality of groups of regions containing mathematical symbols based on the boundaries of a plurality of groups of text sequences;

[0032] Based on the second text sequence and the implicit mathematical symbol contained in the plurality of groups of regions, a replacement character is determined for replacing the text sequence.

[0033] According to one embodiment of the present invention, in response to determining according to the dictionary that the semantic attribute of the characters of the second text sequence is a function character, the corresponding characters are replaced with a function formula;

[0034] In response to the character semantics of the second text sequence determined according to the dictionary being variable characters, corresponding characters are replaced with characters that are different from the function formula.

[0035] According to one embodiment of the present invention, function symbols are used instead of implicit mathematical symbols.

[0036] According to a second aspect of the present invention, there is provided a device for recognizing mathematical symbols in ancient books, comprising:

[0037] A first image acquisition unit, configured to perform an opening operation on the original image based on pixel values ​​to obtain a first image;

[0038] A first processing unit, configured to perform a first convolution process on the first image using a first convolution kernel, and recognize the image after the first convolution process to obtain a first processing result;

[0039] A second image acquisition unit adjusts the first image based on the text recognized by the first processing result, so that the recognition result included in the first processing result is masked to obtain a second image;

[0040] Performing a second processing unit and a second convolution process on the second image using a second convolution kernel, and recognizing the image after the second convolution process to obtain a second processing result;

[0041] The mathematical symbol recognition unit determines the position and relevance of the mathematical symbol based on the first processing result and the second processing result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a flowchart of a method for recognizing mathematical symbols in ancient books in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, rather than implying any limitation on the scope of the present disclosure.

[0044] According to one embodiment of the present invention, a method for recognizing mathematical symbols in ancient books includes steps 1100-1500.

[0045] Step 1100: Perform an opening operation on the original image based on pixel values ​​to obtain a first image.

[0046] The original image refers to an image of an ancient book containing text and mathematical symbols. Ancient books refer to books that are relatively old. Under the influence of the environment, there will be some stains on the pages of ancient books that are irrelevant to the written content. For example, stains appear due to aging of paper. The above stains will cause noise areas to exist in the original image. Since the color of the noise area is different from the color of the text, by performing an opening operation on the original image based on the pixel value, the noise in the original image can be removed and the text and mathematical symbols in the original image can be retained.

[0047] Opening is a basic operation in mathematical morphology, which is defined by performing an erosion operation followed by a dilation operation. This process can effectively remove small objects, burrs, isolated dots, and bridges (i.e., dots connecting two areas) in an image while keeping the shape and position of larger objects in the image basically unchanged.

[0048] The purpose of the opening operation is to remove noise and smooth the boundaries of the image. The opening operation can be implemented through functions in image processing libraries such as OpenCV, such as using the cv2.morpho logyEx() function and specifying the cv2.MORPH_OPEN flag to perform the opening operation. In addition, the effect of the opening operation can be controlled by adjusting the size and shape of the structural element. Different structural element selections can extract different features from the image.

[0049] Step 1200: Perform a first convolution process on the first image using a first convolution kernel, and recognize the image after the first convolution process to obtain a first processing result.

[0050] The convolution kernel can be represented as a matrix, and the elements in the matrix represent the weights of the pixels. At the beginning of the convolution operation, the convolution kernel is placed in the upper left corner of the image. Then, the convolution kernel slides on the image with a certain step size, either from left to right or from top to bottom. The step size represents the distance the convolution kernel moves each time. The convolution operation refers to the process of weighted summing of pixels in a part of the input image to obtain the corresponding pixel value in the output image.

[0051] In this embodiment, the edge weight of the first convolution kernel is greater than the internal weight. The first convolution kernel can refer to the following matrix. For example:

[0052]

[0053] The best convolution kernel can be 10*10. When processing, Chinese characters involving symbols will be difficult to recognize due to weights, while normal written text will not affect recognition due to its large area. By sampling the first convolution kernel and performing the first convolution processing on the first image, the text information in the image can be obtained.

[0054] Step 1300: Adjust the first image based on the text recognized by the first processing result so that the recognition result included in the first processing result is masked to obtain a second image.

[0055] Image masking is to use a selected image to block all or part of the image to be processed to control the area or process of image processing. Image masking can be used to extract the area of ​​interest in the image. Since the recognition result contained in the first processing result is the text in the first image, the text is blocked after masking, and mathematical symbols remain. The second image is an image containing mathematical symbols.

[0056] Step 1400: Perform a second convolution process on the second image using a second convolution kernel, and recognize the image after the second convolution process to obtain a second processing result.

[0057] In this embodiment, the receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel. The edge weight of the second convolution kernel is smaller than the internal weight.

[0058] The receptive field of a convolution kernel refers to the area of ​​the input data that the convolution kernel can cover during the convolution operation. The receptive field of the convolution kernel is reflected by the dimension of the matrix. Specifically, the receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel, that is, the size of the first convolution kernel is larger than the size of the second convolution kernel. For example, the size of the first convolution kernel is 7*7, and the size of the second convolution kernel is 5*5.

[0059] For mathematical symbols in ancient books, including mathematical symbols presented by Chinese characters and mathematical symbols in non-symbol Chinese characters, since the number of pixels of mathematical symbols is generally low, more attention should be paid to the internal weights when processing. In this case, the second convolution kernel can be:

[0060]

[0061] The above method can extract mathematical symbols presented by Chinese characters. However, it is not possible to process symbols whose length and width are greater than a non-symbol Chinese character. The second processing result is a mathematical symbol presented by Chinese characters.

[0062] Step 1500: Determine the location and association of mathematical symbols based on the first processing result and the second processing result.

[0063] The position of a mathematical symbol indicates the position of the mathematical symbol in the image, which can be represented by image coordinates. For the recognized mathematical symbols, some of them exist alone. For another part of the mathematical symbols, at least one of their adjacent positions is also a mathematical symbol, and there is a correlation between these mathematical symbols.

[0064] The correlation between mathematical symbols can be determined based on the position of each mathematical symbol. For example, for a mathematical symbol, if there is no other mathematical symbol within a certain range of its own position, then the mathematical symbol is independent and has no correlation with other mathematical symbols. If there is another mathematical symbol within a certain range of the position of a mathematical symbol, then the two mathematical symbols are correlated.

[0065] In this embodiment, when the convolution result of the first convolution kernel and the second convolution kernel is lower than the threshold value, the convolution result is set to 0. For example, the threshold value is set to 100, and if the pixel value obtained after the convolution operation by the first convolution kernel is 85, then the result is further set to 0.

[0066] The present invention performs an opening operation on the original image to remove noise in the original image, thereby obtaining a first image containing text and mathematical symbols. Then, the first image is convolved using a first convolution kernel to identify the text contained in the first image, and then the first image is masked according to the identified text to remove the text in the first image, thereby obtaining a second image containing mathematical symbols. Then, the second image is convolved using a second convolution kernel to identify the mathematical symbols in the second image. In combination with the characteristics of text and mathematical symbols, convolution kernels of different sizes are used. Since the area of ​​text in the image is larger than the area of ​​mathematical symbols in the image, the receptive field of the first convolution kernel used in identifying text is larger than the receptive field of the second convolution kernel used in identifying mathematical symbols, and the text and mathematical symbols in the image can be more accurately identified.

[0067] According to one embodiment of the present invention, step 1100 includes steps 1110 - 1140 .

[0068] Step 1110: Based on the pixel statistics of the original image, determine the pixels with the largest proportion as the background.

[0069] The original image refers to the image taken of the page of an ancient book. For example, for a paper ancient book, the original image is obtained by photographing one of its pages. The color of the writing on the ancient book paper is different from the color of the ancient book paper. For example, the ancient book paper is yellow and the writing is black. Usually the writing only occupies a part of the entire ancient book page, so the pixels with the largest proportion in the original image can be used as the background.

[0070] Step 1120: remove the background of the original image to obtain a first intermediate image.

[0071] After removing the background in the original image, the remaining content includes the text written on the ancient book, mathematical symbols and noise areas, such as the noise area formed by the stains on the pages of the ancient book. The first intermediate image includes the text, mathematical symbols and noise areas.

[0072] Step 1130: Based on pixel statistics of the first intermediate image, determine the pixel with the largest proportion as the text color.

[0073] When writing, the same color is usually used. Therefore, the colors of all the text and mathematical symbols in the first intermediate image are the same color. For example, the color of the text is black. However, for the noise area in the first intermediate image, its color is different from the color of the text. Moreover, the colors of different noise areas may also be different.

[0074] By counting the pixels of the first intermediate image, the text color and the color of the noise area are obtained. Since the noise area is smaller than the text area, the pixel with the largest proportion can be used as the text color.

[0075] Step 1140: Using the text color as the target image area, perform an opening operation on the first intermediate image to obtain a first image.

[0076] After obtaining the text color, an opening operation is performed on the first intermediate image to remove other areas in the first image except the text color, that is, to remove the noise area in the first intermediate image, and obtain a first image containing text and mathematical symbols.

[0077] Based on the characteristics that the text color is different from the color of the noise area, the present invention can accurately extract the text color in the first intermediate image by counting the pixel points of the first intermediate image and taking the pixel with the largest proportion as the text color. Then, an opening operation is performed on the first intermediate image according to the text color to remove the noise area contained in the first intermediate image and retain the text area in the first intermediate image to avoid the loss of the text area.

[0078] According to one embodiment of the present invention, the size of the first convolution kernel is at least 5*5; the size of the second convolution kernel does not exceed 5*5.

[0079] According to an embodiment of the present invention, a first text sequence and positions of the text in the image are obtained based on the first processing result.

[0080] The distribution of the descriptive text in the image is determined based on the position of the first text sequence in the image.

[0081] The second character sequence and the distribution of the corresponding character sequence in the image are obtained based on the second processing result.

[0082] Based on the distribution of the second character sequence and the corresponding character sequence in the image, boundaries of several groups of character sequences contained in the image are determined.

[0083] Several groups of regions containing mathematical symbols are determined based on the boundaries of several groups of text sequences.

[0084] Based on the second text sequence contained in the plurality of groups of regions, a replacement character for replacing the text sequence is determined.

[0085] The first text sequence represents the text content included in the first processing result. The position of the text in the image can be represented by the image coordinates. The distribution of the first text sequence in the image represents the position of each text in the first text sequence in the image. By recognizing the image after the first convolution processing, the position of each text in the first text sequence in the image can be obtained. The text in the first text sequence constitutes a descriptive text.

[0086] The second character sequence represents the mathematical symbols included in the second processing result. Specifically, the second character sequence is a mathematical symbol presented by Chinese characters. The distribution of the second character sequence in the image represents the position of each mathematical symbol in the second character sequence in the image. By recognizing the image after the second convolution processing, the position of each mathematical symbol in the second character sequence in the image can be obtained.

[0087] Based on the distribution of the second text sequence in the image, the boundaries of multiple groups of text sequences are obtained. Since the second text sequence contains multiple mathematical symbols represented by Chinese characters, some of the mathematical symbols exist alone, and these mathematical symbols are surrounded by descriptive texts. For another part of the mathematical symbols, at least one adjacent position is also a mathematical symbol. The boundary of the text sequence represents the boundary of the mathematical symbol. For a mathematical symbol that exists alone, the boundary is the boundary of the mathematical symbol, and the corresponding area only contains one mathematical symbol. For multiple mathematical symbols that exist adjacent to each other, the boundary is formed by connecting the boundaries of the multiple mathematical symbols, and the area will contain these multiple mathematical symbols.

[0088] For the second text sequence contained in each group of regions, the corresponding characters are determined, and the text sequence is replaced by the determined characters. For different second text sequences, different characters can be used for replacement.

[0089] According to an embodiment of the present invention, in response to the dictionary determining that the character semantic attribute of the second character sequence is a function character, the corresponding character is replaced with the function corresponding to the dictionary;

[0090] In response to the character semantics of the second character sequence determined by the dictionary being a variable character, the corresponding character is replaced with a character different from the function.

[0091] From the perspective of character semantic attributes, mathematical symbols include function characters and variable characters. For example, in the mathematical expression "1 + 1 = 2", the mathematical symbols "1" and "2" represent variable characters, while the mathematical symbols "+" and "=" represent function characters. For ancient Chinese mathematical symbols, although the forms of ancient Chinese mathematical symbols are different from modern mathematical symbols, the specific semantics can be the same. Ancient Chinese mathematical symbols are also divided into function characters and variable characters.

[0092] The dictionary contains the meanings of various ancient Chinese mathematical symbols. After obtaining an ancient Chinese mathematical symbol, the character semantic attribute of the ancient Chinese mathematical symbol can be determined by querying the dictionary to determine whether the ancient Chinese mathematical symbol is a variable character or a function character.

[0093] If it is determined that the ancient Chinese mathematical symbol is a function character, then the function character corresponding to the character semantics of the ancient Chinese mathematical symbol in the dictionary is used to replace the ancient Chinese mathematical symbol. For example, if it is determined that the semantics of the ancient Chinese mathematical symbol is "addition", and the function character corresponding to "addition" in the dictionary is "+", then the function character "+" can be used to replace the ancient Chinese mathematical symbol.

[0094] If it is determined that the ancient Chinese mathematical symbol is a variable character, then the variable character corresponding to the character semantics of the ancient Chinese mathematical symbol in the dictionary is used to replace the ancient Chinese mathematical symbol. For example, if it is determined that the semantics of the ancient Chinese mathematical symbol is "one", and the variable character corresponding to "one" in the dictionary is "1", then the variable character "1" can be used to replace the ancient Chinese mathematical symbol.

[0095] The present invention ensures that the meaning expressed by the content after replacement is the same as that before replacement by using the character corresponding to the character semantic attribute of the second character sequence for replacement.

[0096] According to an embodiment of the present invention, based on the first processing result and the image region corresponding to the second processing result, the unrecognized image region in the first image is determined.

[0097] Based on the first image recognition result and the second image recognition result, the recognized regions are removed to obtain a third image.

[0098] The third image is cropped to obtain a number of sub-images to be recognized.

[0099] Process each sub - image to obtain the recognition result of the mathematical symbols within the sub - image.

[0100] The recognition result of the mathematical symbols within the sub - image includes implicit mathematical symbols and their offsets relative to the origin of the first image.

[0101] By processing the second image with a second convolution kernel, the mathematical symbols presented by Chinese characters can be extracted. However, symbols with lengths and widths greater than that of a non - symbol Chinese character cannot be processed. At this time, all the recognized Chinese characters need to be removed from the image and then processed.

[0102] The original image includes text, mathematical symbols, and noise. By performing an opening operation on the original image, the noise area is removed to obtain the first image. The first image includes text and mathematical symbols. Here, the mathematical symbols include those presented by Chinese characters and the mathematical symbols of non - symbol Chinese characters.

[0103] The recognition result of the first image includes text, and the recognition result of the second image includes the mathematical symbols presented by Chinese characters. Removing the recognition results of the first image and the second image from the first image, that is, removing the text and the mathematical symbols presented by Chinese characters from the first image, then the remaining mathematical symbols in the third image are only those of non - symbol Chinese characters.

[0104] After obtaining the third image, crop the third image to get multiple sub - images. Some of the sub - images contain mathematical symbols of non - symbol Chinese characters. By recognizing each sub - image, the mathematical symbols of non - symbol Chinese characters contained in the sub - image are recognized.

[0105] During the process of recognizing the sub - image, the offset of the implicit mathematical symbol relative to the origin of the first image is also recognized. Specifically, the position of the implicit mathematical symbol in the sub - image can be obtained by recognizing the sub - image. Since the sub - image is obtained by cropping the third image, and the third image is obtained by removing the recognition results of the first image and the second image from the first image, the position of the sub - image in the third image is also the position of the sub - image in the first image. Combining the position of the sub - image in the third image and the position of the implicit mathematical symbol in the sub - image, the offset of the implicit mathematical symbol relative to the origin of the first image can be obtained. The implicit mathematical symbol is a mathematical symbol presented by a Chinese character. For example, the Chinese character "讷" is the Chinese character form of the mathematical symbol ln, so the Chinese character "讷" is an implicit mathematical symbol.

[0106] According to one embodiment of the present invention, based on the distribution of the second text sequence and the corresponding text sequence in the image, the boundaries of several groups of text sequences contained in the image are determined; based on the boundaries of the several groups of text sequences, several groups of regions containing mathematical symbols are determined; based on the second text sequence and implicit mathematical symbols contained in the several groups of regions, replacement characters for replacing the text sequences are determined.

[0107] The second character sequence represents the mathematical symbols included in the second processing result. Specifically, the second character sequence is a mathematical symbol presented by Chinese characters. The distribution of the second character sequence in the image represents the position of each mathematical symbol in the second character sequence in the image. By recognizing the image after the second convolution processing, the position of each mathematical symbol in the second character sequence in the image can be obtained.

[0108] Based on the distribution of the second text sequence in the image, the boundaries of multiple groups of text sequences are obtained. Since the second text sequence contains multiple mathematical symbols represented by Chinese characters, some of the mathematical symbols exist alone, and these mathematical symbols are surrounded by descriptive texts. For another part of the mathematical symbols, at least one adjacent position is also a mathematical symbol. The boundary of the text sequence represents the boundary of the mathematical symbol. For a mathematical symbol that exists alone, the boundary is the boundary of the mathematical symbol, and the corresponding area only contains one mathematical symbol. For multiple mathematical symbols that exist adjacent to each other, the boundary is formed by connecting the boundaries of the multiple mathematical symbols, and the area will contain these multiple mathematical symbols.

[0109] For the second text sequence contained in each group of regions, the corresponding characters are determined, and the text sequence is replaced by the determined characters. For different second text sequences, different characters can be used for replacement.

[0110] According to one embodiment of the present invention, in response to determining according to the dictionary that the character semantic attributes of the second text sequence are function characters, function formulas are used to replace corresponding characters; in response to determining according to the dictionary that the character semantics of the second text sequence are variable characters, characters different from the function formula are used to replace corresponding characters.

[0111] When determining the replacement character for replacing the second character sequence, the corresponding character is used for replacement according to the character semantic attribute of the second character sequence. From the perspective of character semantic attribute, the ancient mathematical symbols include function characters and variable characters.

[0112] The dictionary contains the meanings of various ancient mathematical symbols. After obtaining an ancient mathematical symbol, the character semantic attributes of the ancient mathematical symbol can be determined by querying the dictionary to determine whether the ancient mathematical symbol is a variable character or a function character.

[0113] If it is determined that the ancient mathematical symbol is a function character, then the function formula is used to replace the ancient mathematical symbol. If it is determined that the ancient mathematical symbol is a variable character, then a symbol different from the function formula is used to replace the ancient mathematical symbol.

[0114] According to one embodiment of the present invention, function symbols are used instead of implicit mathematical symbols.

[0115] According to one embodiment of the present invention, a device for recognizing mathematical symbols in ancient books includes:

[0116] The first image acquisition unit is used to perform an opening operation on the original image based on pixel values ​​to obtain a first image.

[0117] The first processing unit is used to perform a first convolution process on the first image using a first convolution kernel, and recognize the image after the first convolution process to obtain a first processing result.

[0118] The second image acquisition unit adjusts the first image based on the text recognized by the first processing result, so that the recognition result included in the first processing result is masked to obtain the second image.

[0119] A second processing unit performs a second convolution process on the second image using a second convolution kernel, and recognizes the image after the second convolution process to obtain a second processing result.

[0120] The mathematical symbol recognition unit determines the position and relevance of the mathematical symbol based on the first processing result and the second processing result.

[0121] Those of ordinary skill in the art will appreciate that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0124] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0125] In addition, each functional module in the embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0126] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the energy-saving signal sending / receiving method of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0127] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0128] It should be understood that the size of the sequence number of each step in the content of the invention and the embodiments of the present invention does not absolutely mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. For the purpose of example and description, the foregoing description of the implementation of the present disclosure has been given. The foregoing description is not exhaustive and is not intended to limit the present disclosure to the exact form disclosed. Various deformations and modifications may exist according to the above teachings, or various deformations and modifications may be obtained from the practice of the present disclosure. These embodiments are selected and described to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can use the present disclosure in various embodiments and various modifications suitable for the specific purpose conceived.

Claims

1. A method for recognizing mathematical symbols in ancient books, characterized in that: include: Performing an opening operation on the original image based on pixel values ​​to obtain a first image; Performing a first convolution process on the first image using a first convolution kernel, and recognizing the image after the first convolution process to obtain a first processing result; Adjusting the first image based on the text recognized by the first processing result so that the recognition result included in the first processing result is masked to obtain a second image; Performing a second convolution process on the second image using a second convolution kernel, and recognizing the image after the second convolution process to obtain a second processing result; determining the location and relevance of the mathematical symbol based on the first processing result and the second processing result; The receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel, and the edge weight of the first convolution kernel is larger than the internal weight, and the edge weight of the second convolution kernel is smaller than the internal weight; When the convolution result of the first convolution kernel and the second convolution kernel is lower than the threshold value, the convolution result is set to 0.

2. The method for recognizing mathematical symbols in ancient books as claimed in claim 1, characterized in that: Based on the pixel statistics of the original image, the pixels with the largest proportion are determined as the background; Remove the background of the original image to obtain a first intermediate image; Based on pixel statistics of the first intermediate image, determine the pixel with the largest proportion as the text color; Taking the text color as the target image area, an opening operation is performed on the first intermediate image to obtain a first image.

3. The method for recognizing mathematical symbols in ancient books as claimed in claim 1, characterized in that: Obtaining a first text sequence and a position of the text in the image based on the first processing result; determining a distribution of the descriptive text in the image based on a position of the first text sequence in the image; Obtaining distribution of a second character sequence and a corresponding character sequence in the image based on the second processing result; Determining boundaries of a plurality of groups of character sequences contained in the image based on distribution of the second character sequence and the corresponding character sequence in the image; Determining a plurality of groups of regions containing mathematical symbols based on the boundaries of a plurality of groups of text sequences; Based on the second text sequence contained in the plurality of groups of regions, a replacement character for replacing the text sequence is determined.

4. The method for recognizing mathematical symbols in ancient books as claimed in claim 3, characterized in that: In response to the dictionary determining that the semantic attribute of the characters of the second text sequence is a function character, replacing the corresponding characters with the functions corresponding to the dictionary; In response to the character semantics of the second text sequence determined by the dictionary to be variable characters, characters that are different from functions are used to replace corresponding characters.

5. The method for recognizing mathematical symbols in ancient books as claimed in claim 1, characterized in that: Determining an unrecognized image region within the first image based on an image region corresponding to the first processing result and the second processing result; Based on the first image recognition result and the second image recognition result, the recognized area is removed to obtain a third image; Cropping the third image to obtain a plurality of sub-images to be identified; Processing each sub-image to obtain a recognition result of the mathematical symbols in the sub-image; The recognition result of the mathematical symbol in the sub-image includes the implicit mathematical symbol and its offset relative to the origin of the first image.

6. The method for recognizing mathematical symbols in ancient books as claimed in claim 5, characterized in that: Determining boundaries of a plurality of groups of character sequences contained in the image based on distribution of the second character sequence and the corresponding character sequence in the image; Determining a plurality of groups of regions containing mathematical symbols based on the boundaries of a plurality of groups of text sequences; Based on the second text sequence and the implicit mathematical symbol contained in the plurality of groups of regions, a replacement character is determined for replacing the text sequence.

7. The method for recognizing mathematical symbols in ancient books as claimed in claim 6, characterized in that: In response to determining according to the dictionary that the semantic attribute of the characters of the second text sequence is a function character, replacing the corresponding characters with a function formula; In response to the character semantics of the second text sequence determined according to the dictionary being variable characters, corresponding characters are replaced with characters that are different from the function formula.

8. The method for recognizing mathematical symbols in ancient books as claimed in claim 5, characterized in that: Use function notation instead of implicit math notation.

9. A device for recognizing mathematical symbols in ancient books, characterized in that: include: A first image acquisition unit, configured to perform an opening operation on the original image based on pixel values ​​to obtain a first image; A first processing unit, configured to perform a first convolution process on the first image using a first convolution kernel, and recognize the image after the first convolution process to obtain a first processing result; A second image acquisition unit adjusts the first image based on the text recognized by the first processing result, so that the recognition result included in the first processing result is masked to obtain a second image; Performing a second processing unit and a second convolution process on the second image using a second convolution kernel, and recognizing the image after the second convolution process to obtain a second processing result; a mathematical symbol recognition unit that determines the position and relevance of the mathematical symbol based on the first processing result and the second processing result; The receptive field of the first convolution kernel is larger than the receptive field of the second convolution kernel, and the edge weight of the first convolution kernel is larger than the internal weight, and the edge weight of the second convolution kernel is smaller than the internal weight; When the convolution result of the first convolution kernel and the second convolution kernel is lower than the threshold value, the convolution result is set to 0.

Images