Formula annotation methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a formula annotation method, apparatus, electronic device, and storage medium, and pertains to the field of image recognition technology. The method includes: first, acquiring an original image containing the formula to be annotated; then, performing downsampling processing on the original image using n different multiples through a backbone network to obtain a feature map corresponding to each of the n multiples as a first feature map, thereby obtaining n first feature maps at different feature levels; then, performing representational enhancement processing on the n first feature maps using a cascaded feature enhancement network composed of m feature enhancement networks to obtain m sets of second feature maps, each set including n second feature maps; performing feature fusion processing on the m sets of second feature maps using a feature fusion network; and obtaining a target image based on a third feature map, wherein the target image includes annotation markers for the formula to be annotated, and the location of the formula in the image is more accurately annotated based on the third feature map with strong representational capabilities.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and more specifically, to a formula annotation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology, it has been applied to many fields, including education. Formula detection technology, a key component of computer technology, has significant application value and can be used in AI (Artificial Intelligence) learning machines, intelligent grading, and question bank input / output. However, in practical applications, mathematical and physical formulas may exhibit variations such as large formula area size, irregular formula area shape, or multiple formulas clustered together. These variations can lead to difficulties in accurately identifying the location of formulas during the identification process. Summary of the Invention

[0003] The purpose of this disclosure is to provide a formula annotation method, apparatus, electronic device, and storage medium to overcome the aforementioned technical problems.

[0004] A first aspect of this disclosure provides a formula annotation method, comprising: acquiring an original image, wherein the original image contains a formula to be annotated; performing downsampling processing on the original image by n multiples through a backbone network to obtain feature maps corresponding to each of the n multiples as first feature maps, wherein n is an integer greater than 2, and the n first feature maps have different sizes; performing representational capability enhancement processing on the n first feature maps through a cascaded feature enhancement network to obtain m sets of second feature maps, wherein m is an integer greater than 2, the cascaded feature enhancement network includes m sequentially cascaded feature enhancement networks, and each of the m sets of second feature maps includes n second feature maps; performing feature fusion processing on the m sets of second feature maps through a feature fusion network to obtain a third feature map, wherein the size of the third feature map is the same as the size of the original image; and acquiring a target image based on the third feature map, wherein the target image includes annotation markers for annotating the formula to be annotated.

[0005] Optionally, the step of enhancing the representational ability of the n first feature maps through a cascaded feature enhancement network to obtain m sets of second feature maps includes: inputting the n first feature maps into the first feature enhancement network in the cascaded feature enhancement network for representational ability enhancement processing to obtain a set of feature maps output by the first feature enhancement network, wherein the set of feature maps includes n feature maps; inputting the n feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representational ability enhancement processing to obtain a set of feature maps output by the next feature enhancement network; replacing the first feature enhancement network with the next feature enhancement network, and continuing to execute the above steps of inputting the set of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representational ability enhancement processing to obtain a set of feature maps output by the next feature enhancement network, until all m feature enhancement networks in the cascaded feature enhancement network output a set of feature maps, and using the feature maps output by the m feature enhancement networks in the cascaded feature enhancement network as the m sets of second feature maps.

[0006] Optionally, the feature enhancement network includes an upsampling enhancement module and a downsampling enhancement module. The step of inputting the n first feature maps into the first feature enhancement network of the cascaded feature enhancement network to obtain a set of feature maps output by the first feature enhancement network includes: inputting the n first feature maps into the upsampling enhancement module of the first feature enhancement network for processing to obtain a set of feature maps output by the upsampling enhancement module of the first feature enhancement network; and inputting the set of feature maps output by the upsampling enhancement module into the downsampling enhancement module of the first feature enhancement network for processing to obtain a set of feature maps output by the downsampling enhancement module.

[0007] Optionally, the second feature images within each of the m groups of second feature images have different sizes, and there are second feature images of the same size between each group. The step of performing feature fusion processing on the m groups of second feature images through a feature fusion network to obtain a third feature image includes: adding the second feature images of the same size in the m groups of second feature images to obtain n fused feature images; and upsampling the n fused feature images to obtain a third feature image.

[0008] Optionally, obtaining the target image based on the third feature map includes: obtaining a probability map corresponding to the third feature map and a threshold map corresponding to the third feature map; performing a differentiable operation on the probability map and the threshold map through a differentiable binarization network to obtain a binary map; and determining the binary map as the target image.

[0009] Optionally, obtaining the target image based on the third feature map includes: obtaining a probability map corresponding to the third feature map; and determining the probability map as the target image.

[0010] Optionally, obtaining the target image based on the third feature map includes: obtaining a threshold map corresponding to the third feature map; and determining the threshold map as the target image.

[0011] A second aspect of this disclosure provides a formula annotation apparatus, comprising: an acquisition module for acquiring an original image, wherein the original image contains a formula to be annotated; a downsampling module for performing downsampling processing on the original image at n different multiples through a backbone network to obtain a feature map corresponding to each of the n multiples as a first feature map, wherein n is an integer greater than 2, and the n first feature maps have different sizes; an enhancement module for performing representational capability enhancement processing on the n first feature maps through a cascaded feature enhancement network to obtain m sets of second feature maps, wherein m is an integer greater than 2, the cascaded feature enhancement network includes m sequentially cascaded feature enhancement networks, and each of the m sets of second feature maps includes n second feature maps; a fusion module for performing feature fusion processing on the m sets of second feature maps through a feature fusion network to obtain a third feature map, wherein the size of the third feature map is the same as the size of the original image; and an annotation module for acquiring a target image based on the third feature map, wherein the target image includes annotation markers for annotating the formula to be annotated.

[0012] A third aspect of this disclosure provides an electronic device, comprising: a memory storing a computer program thereon; and a processor for executing the computer program in the memory to implement the steps of the method described above.

[0013] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium that, when executed by a processor, implements the steps of the above-described method.

[0014] The formula annotation method, apparatus, electronic device, and storage medium disclosed herein first acquire an original image containing the formula to be annotated; then, the original image is downsampled by n factors through a backbone network to obtain a feature map corresponding to each of the n factors as a first feature map, thereby obtaining n first feature maps of different feature levels; then, the n first feature maps are enhanced by a cascaded feature enhancement network composed of m feature enhancement networks to obtain m sets of second feature maps. The cascaded feature enhancement network is equivalent to performing an initial feature fusion process on the first feature maps, where n and m are both integers greater than 2, and each of the m sets of second feature maps includes n first feature maps. The m sets of second feature maps are processed by a feature fusion network (i.e., a second feature fusion process) to obtain a third feature map. This third feature map has the advantages of both high-level and low-level features, resulting in stronger representation capabilities. The size of the third feature map is the same as that of the original image. The target image is obtained based on the third feature map. The target image includes annotations for the formula to be annotated. The image features are enhanced by performing two feature fusions through a cascaded feature enhancement network and a feature fusion network, thereby improving image performance. The location of the formula to be annotated in the image is more accurately annotated based on the third feature map with strong representation capabilities.

[0015] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0017] Figure 1 A flowchart illustrating a formula annotation method provided in an embodiment of this disclosure is shown;

[0018] Figure 2 A schematic diagram of the formula annotation model is shown;

[0019] Figure 3 This disclosure illustrates Figure 1 A flowchart illustrating step S130 of the formula annotation method shown;

[0020] Figure 4 This disclosure illustrates Figure 3 A flowchart illustrating step S131 of the formula annotation method shown;

[0021] Figure 5 A schematic diagram of the enhancement process is shown;

[0022] Figure 6A schematic diagram of the upsampling enhancement module processing is shown;

[0023] Figure 7 A schematic diagram of the downsampling enhancement module processing is shown;

[0024] Figure 8 This disclosure illustrates Figure 1 A flowchart illustrating step S140 of the formula annotation method shown;

[0025] Figure 9 A schematic diagram of the fusion process is shown;

[0026] Figure 10 This disclosure illustrates Figure 1 A flowchart illustrating step S150 of the formula annotation method shown;

[0027] Figure 11 This disclosure illustrates Figure 1 Another flowchart illustrating step S150 of the formula annotation method shown;

[0028] Figure 12 This disclosure illustrates Figure 1 Another flowchart illustrating step S150 of the formula annotation method shown;

[0029] Figure 13 A flowchart illustrating a formula annotation method provided in another embodiment of this disclosure is shown;

[0030] Figure 14 (a) shows the original image;

[0031] Figure 14 (b) shows the target image;

[0032] Figure 15 A block diagram of a formula annotation device provided in one embodiment of the present disclosure is shown;

[0033] Figure 16 A block diagram of an electronic device for performing a formula notation method according to an embodiment of the present disclosure is shown;

[0034] Figure 17 A block diagram of an electronic device for performing a formula annotation method according to an embodiment of the present disclosure is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] With the development of computer technology, it has been applied to many fields, such as education. Formula detection technology, in particular, has high application value and can be used in AI learning machines, intelligent grading, and question bank input and output. However, in practical applications, mathematical, physical, and chemical formulas may exhibit characteristics such as large ranges in formula area size, irregular shapes, or multiple formulas clustered together. These variations can lead to difficulties in accurately identifying the location of formulas during the identification process.

[0037] To address the aforementioned issues, image processing methods are currently employed to detect formulas. When the original image containing the formula is obtained, preprocessing operations such as binarization are first performed on the image. Then, by combining connected component methods and empirical thresholds, candidate regions for formulas are obtained on the binarized image. Finally, heuristic rules are used to determine the final formula region from the image.

[0038] However, the inventors discovered that this method heavily relies on image preprocessing, has weak model generalization ability, and the various modules in the model are interconnected. Poor performance of the upstream module will seriously affect the processing performance of the downstream module, thus ultimately leading to poor formula detection results.

[0039] To address the aforementioned issues, deep learning methods can be employed for formula detection. These methods generally employ two strategies: regression-based and segmentation-based approaches. Regression-based methods transform the formula detection problem into a target regression problem, while segmentation-based methods transform it into semantic segmentation or instance segmentation problems.

[0040] However, regression-based formula detection methods cannot handle formula regions of arbitrary shapes. Segmentation-based formula detection methods are currently popular, but existing formula detection methods have complex post-processing and feature extraction processes, which are not conducive to the engineering implementation of formula detection models.

[0041] To address the aforementioned technical problems, the inventors, through long-term research, discovered and proposed a formula annotation method, device, electronic device, and storage medium. This method enhances the representational capability of the original image by performing two feature fusions using a cascaded feature enhancement network and a feature fusion network, thereby improving image performance. Furthermore, the location of the formula to be annotated in the image is more accurately marked based on the third feature map with stronger representational capability. The specific formula annotation method will be described in subsequent embodiments.

[0042] This embodiment provides a formula annotation method. The execution subject of this method can be an electronic device, which can be a terminal device or a server. The terminal device can be a mobile phone, tablet computer, laptop computer, smart wearable device, learning machine, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server. Please refer to [link to relevant documentation]. Figure 1 The formula annotation method may specifically include the following steps:

[0043] Step S110: Obtain the original image, wherein the original image contains the formula to be labeled.

[0044] The original image displays formulas that need to be labeled. This labeling is necessary to facilitate subsequent formula recognition. These formulas can be mathematical, numerical, chemical, physical, or other similar formulas.

[0045] The electronic device acquires the original image. In one embodiment, the electronic device is equipped with a camera, which captures images of books, newspapers, exam papers, displays, blackboards, etc., and uses these images as the original images for subsequent formula annotation.

[0046] In one real-time approach, the electronic device can retrieve the original image from a storage location. One method involves the electronic device pre-storing the original image, which it can then retrieve from its local storage. Another method involves other devices (e.g., mobile phones, servers, etc.) storing the original image, and the electronic device establishing a communication connection with these other devices to retrieve the original image for subsequent formula annotation.

[0047] Step S120: The original image is downsampled by n factors through the backbone network to obtain the feature map corresponding to each of the n factors as the first feature map, where n is an integer greater than 2 and the n first feature maps have different sizes.

[0048] The backbone network is used to extract image features to obtain feature maps. The features extracted by the backbone network are divided into high-level features and low-level features, which have different performance characteristics. High-level features have stronger semantic information but lower resolution and poorer ability to perceive details. Therefore, high-level features can be used to determine whether there is foreground in the original image (as shown by the formula in this scheme). Low-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and they contain more noise. Therefore, low-level features can be used to identify the type of foreground. Optionally, the backbone network can be a pyramid Transformer, SHuffleNetV2, MobileNetV3, etc. In this embodiment, the backbone network extracts image features from the original image through downsampling to obtain the feature map of the original image, and uses the feature map of the original image as the first feature map. Downsampling can be understood as reducing the size of the original image.

[0049] Optionally, the original image is downsampled by n factors through a backbone network to obtain a feature map corresponding to each of the n factors as a first feature map. It can be understood that a first feature map is obtained through each factor, and correspondingly, n first feature maps can be obtained through the n factors in this embodiment.

[0050] Since downsampling at different ratios results in different reductions in the original image, this embodiment employs n different downsampling ratios, resulting in n different sizes for the obtained n first feature maps. It is understood that the sizes of the n first feature maps are all different.

[0051] For example, n is 4, and the four downsampling factors can be 4x, 8x, 16x, and 32x. After downsampling the original image by 4x through the backbone network, the first feature map C3 output by the backbone network is obtained. After downsampling the original image by 8x through the backbone network, the first feature map C4 output by the backbone network is obtained. After downsampling the original image by 16x through the backbone network, the first feature map C5 output by the backbone network is obtained. After downsampling the original image by 32x through the backbone network, the first feature map C6 output by the backbone network is obtained. The larger the downsampling factor, the greater the reduction in size of the original image. Therefore, considering the downsampling factors mentioned above, the sizes of the first feature maps C3-C6 decrease sequentially.

[0052] It should be noted that the backbone network in this embodiment can be a separate algorithm, or it can be as follows: Figure 2 As shown, the backbone network is a network module in the trained formula annotation model 100 (this model is used to annotate the formulas to be annotated on the original image), namely... Figure 2 The backbone network is 110.

[0053] Step S130: Enhance the representational ability of the n first feature maps by using a cascaded feature enhancement network to obtain m sets of second feature maps, where m is an integer greater than 2. The cascaded feature enhancement network includes m sequentially cascaded feature enhancement networks, and each of the m sets of second feature maps includes n second feature maps.

[0054] The Cascaded Feature Enhanced Network (CascadeFEN) consists of m sequentially cascaded Feature Enhanced Networks (FENs). Representational capability refers to both high-level and low-level features.

[0055] The n first feature maps obtained in the above steps have different sizes, and each size has its own advantages and disadvantages. By fusing features of different sizes, the advantages of high-level and low-level features can be combined to improve image performance and facilitate subsequent annotation of formula positions in the image. In this embodiment, the n first feature maps are fused through a cascaded enhancement network to enhance their representation ability. Each of the m feature enhancement networks in the cascaded feature enhancement network needs to undergo representation ability enhancement processing. Each of the m feature enhancement networks outputs a set of second feature maps, and finally m sets of second feature maps are obtained. Each set of second feature maps includes n second feature maps. Therefore, in this embodiment, m*n second feature maps are obtained. The size of the m*n second feature maps is unchanged compared to the n first feature maps.

[0056] It should be noted that the cascaded feature enhancement network in this embodiment can be a standalone algorithm, or it can be as follows: Figure 2 As shown, the backbone network is a network module in the trained formula annotation model 100, namely... Figure 2 Cascaded feature enhancement network 120 in the middle.

[0057] Step S140: Perform feature fusion processing on the m groups of second feature maps through a feature fusion network to obtain a third feature map, wherein the size of the third feature map is the same as the size of the original image.

[0058] In this embodiment, a Feature Fusion Network (FFN) is used to fuse features of the same size. The FFN performs feature fusion processing on m sets of second feature maps to further improve image performance, obtaining a third feature map. In step S120, the image is reduced in size through downsampling. During feature fusion processing in this step, the feature map is restored to the size of the original image; that is, the size of the obtained third feature map is the same as the size of the original image, facilitating subsequent annotation of the formulas to be labeled in the image.

[0059] It should be noted that the feature fusion network in this embodiment can be a standalone algorithm, or it can be as follows: Figure 2 As shown, the backbone network is a network module in the trained formula annotation model 100, namely... Figure 2 Feature fusion network 130.

[0060] Step S150: Obtain the target image based on the third feature map, wherein the target image includes annotation markers for annotating the formula to be annotated, and the annotation markers may include annotation boxes and text annotations.

[0061] The electronic device processes the third feature map to obtain the target image, which has been annotated. This facilitates the segmentation or recognition of formulas within the image.

[0062] Since the size of the third feature image is the same as that of the original image, the position of the formula to be labeled is also the same on both the original image and the third feature image. By labeling the formula on the third feature image, the position of the labeled formula is the same as that on the original image, thus achieving the labeling of the formula to be labeled in the original image.

[0063] This embodiment provides a formula annotation method. First, an original image containing the formula to be annotated is obtained. Then, the original image is downsampled by n factors through a backbone network to obtain a feature map corresponding to each of the n factors as a first feature map, thereby obtaining n first feature maps at different feature levels. Next, a cascaded feature enhancement network composed of m feature enhancement networks is used to enhance the representational ability of the n first feature maps, obtaining m sets of second feature maps. The cascaded feature enhancement network is equivalent to performing an initial feature fusion process on the first feature maps, where n and m are both integers greater than 2, and each of the m sets of second feature maps includes n second feature maps. The m sets of second feature maps are fused using a feature fusion network (i.e., the second feature fusion process) to obtain a third feature map. This third feature map has the advantages of both high-level and low-level features, resulting in stronger representation capabilities. The size of the third feature map is the same as that of the original image. The target image is then obtained based on the third feature map. The target image includes annotations for the formula to be annotated. By performing two feature fusions on the image features through a cascaded feature enhancement network and a feature fusion network, the representation capabilities are enhanced, improving image performance. The location of the formula to be annotated in the image is more accurately annotated based on the third feature map with strong representation capabilities.

[0064] When enhancing the representational ability of n first feature maps using a cascaded feature enhancement network, all feature enhancement networks in the cascaded network need to enhance the representational ability of the feature images, that is, the feature images are enhanced sequentially through the cascaded sequence. Optionally, Figure 3 This disclosure illustrates Figure 1 Please refer to a flowchart illustrating step S130 of the formula notation method shown. Figure 3 The above step S130 may include the following sub-steps:

[0065] Sub-step S131: Input the n first feature maps into the first feature enhancement network in the cascaded feature enhancement network for representation enhancement processing, and obtain a set of feature maps output by the first feature enhancement network, wherein the set of feature maps includes n feature maps.

[0066] The cascaded feature enhancement network consists of m cascaded feature enhancement networks. The first feature enhancement network in the cascaded feature enhancement network is input into the first feature enhancement network. The first feature enhancement network enhances the representation ability of the n first feature maps of different sizes, and the n feature maps output by the first feature enhancement network are obtained as a set of feature maps. It can be understood that the n feature maps in the set of feature maps are all output by the same feature enhancement network.

[0067] As a way, such as Figure 5As shown, the feature enhancement network includes an upsampling enhancement module and a downsampling enhancement module. Please refer to [link to relevant documentation]. Figure 4 When performing representation capability enhancement processing through the upsampling enhancement module and the downsampling enhancement module, sub-step S131 may include the following sub-steps:

[0068] Sub-step S1311: Input the n first feature maps into the upsampling enhancement module of the first feature enhancement network for processing to obtain a set of feature maps output by the upsampling enhancement module of the first feature enhancement network.

[0069] The n first feature maps are input into the upsampling enhancement module of the first feature enhancement network. The upsampling enhancement module processes the n first feature maps, performing operations such as sampling, addition, depthwise separable convolution, batch normalization, and non-linear ReLU activation, to obtain a single feature map output by the upsampling enhancement module. In other words, the n first feature maps are processed to obtain n feature maps.

[0070] Continuing with the examples above, such as Figure 5 As shown, the n first feature maps are C3, C4, C5, and C6, respectively. First feature maps C3-C6 are input into the first feature enhancement network. The first feature enhancement network extracts the smallest first feature map C6 from the four first feature maps, and determines first feature map C6 as the feature map to be operated on, C6'. The upsampling enhancement module processes C6' and the first feature map C5, as follows... Figure 5 As shown, this processing is represented by ⊕, and the meaning of the ⊕ processing performed by the upsampling enhancement module is as follows: Figure 6 As shown, combined with Figure 5 and Figure 6 The upsampling enhancement module's processing is explained as follows: First, feature map C6' is upsampled by a factor of 2 to obtain a result that is twice the size of feature map C6'. Then, this twice-upsampled result is added to the first feature map C5 to obtain the summed result. This summed result is then subjected to depthwise separable convolution, batch normalization, and non-linear ReLU activation operations in sequence to obtain feature map C5', completing one ⊕ processing step. Similarly, please refer to [link to relevant documentation]. Figure 5 The upsampling enhancement module performs ⊕ processing on feature map C5' and the first feature map C4 to obtain feature map C4'; then, it performs ⊕ processing on feature map C4' and the first feature map C3 to obtain feature map C3'. The four feature maps finally obtained by the upsampling enhancement module are C3', C4', C5', and C6'.

[0071] Sub-step S1312: Input the set of feature maps output by the upsampling enhancement module into the downsampling enhancement module of the first feature enhancement network for processing, and obtain the set of feature maps output by the downsampling enhancement module.

[0072] The n feature maps output by the upsampling enhancement module are input into the downsampling enhancement module. The downsampling enhancement module processes the maps, including convolution, batch normalization, ReLU nonlinear activation, kernel depth adjustment, and separable convolution, to obtain a set of feature maps output by the downsampling enhancement module.

[0073] Continuing with the examples above, such as Figure 5 As shown, feature map C3' is used as feature map C3' to be processed. The downsampling enhancement module processes feature map C3' and feature map C4', as follows: Figure 7 As shown, this processing is represented by ⊕, and the meaning of the ⊕ processing performed by the downsampling enhancement module is as follows: Figure 7 As shown, combined with Figure 5 and Figure 7 The processing of the downsampling enhancement module is described below. Specifically, feature map C3” is first subjected to a 1*1 convolution, batch normalization, and non-linear ReLU activation. Then, a depthwise separable convolution with a stride of 2 and a 3*3 kernel is performed. Since upsampling was performed in the upsampling enhancement module, the feature map is enlarged. In this embodiment, the size of the feature map is reduced by the aforementioned depthwise separable convolution, so that the size of the final output feature map P3 after feature enhancement is the same as the size of the first feature map C3. Similarly, feature map P4 is obtained by processing feature map C3” and feature map C5’; feature map P5 is obtained by processing feature map C4” and feature map C5’; and feature map P6 is obtained by processing feature map C5” and feature map C6’. The final set of feature maps output by the downsampling enhancement module includes feature maps P3, P4, P5, and P6.

[0074] Sub-step S132: Input a set of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representational capability enhancement processing, and obtain a set of feature maps output by the next feature enhancement network.

[0075] The cascading order of cascaded feature enhancement networks involves inputting a set of feature maps (n feature maps, e.g., feature maps P3-P6) output by the first feature enhancement network into the next feature enhancement network for representation enhancement processing, thereby obtaining a set of feature maps (n feature maps) output by the next feature enhancement network.

[0076] Similarly, the next feature enhancement network also includes an upsampling enhancement module and a downsampling enhancement module. The processing methods of the upsampling enhancement module and the downsampling enhancement module of the next feature enhancement network are similar to those of the first feature enhancement network, and will not be repeated here.

[0077] Sub-step S133: Replace the first feature enhancement network with the next feature enhancement network, and continue to execute the above steps of inputting a set of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representational capability enhancement processing, and obtaining a set of feature maps output by the next feature enhancement network, until all m feature enhancement networks in the cascaded feature enhancement network output a set of feature maps, and use the feature maps output by the m feature enhancement networks in the cascaded feature enhancement network as the m sets of second feature maps.

[0078] When a feature enhancement network in a cascaded feature enhancement network has not yet undergone representation enhancement processing, it indicates that representation enhancement processing still needs to continue. The first feature enhancement network is replaced by the next feature enhancement network as the first feature enhancement network in the row, and the above step 132 is continued. It can be understood that the set of feature maps output by the new first feature enhancement network is used as the output of the next feature enhancement network of the new first feature enhancement network, so that the next feature enhancement network continues to perform representation enhancement processing, until all feature enhancement networks in the cascaded feature enhancement network have performed feature enhancement processing and output a set of feature maps.

[0079] After all m feature enhancement networks in the cascaded feature enhancement network have performed enhancement processing, the feature maps output by the m feature enhancement networks are used as m sets of second feature maps. For example, for a set of feature maps P3-P6 output by the first feature enhancement network, this set of feature maps is used as the second feature maps.

[0080] Optionally, Figure 8 This disclosure illustrates Figure 1 The illustrated flowchart shows a step S140 of the formula annotation method. The second feature images within each of the m groups of second feature images have different sizes, while there are second feature images of the same size between each group. Please refer to [link / reference]. Figure 8 Step S140 includes the following sub-steps:

[0081] Sub-step S141: Add the second feature maps of the same size in the m groups of second feature maps to obtain n fused feature maps.

[0082] In this embodiment, the size of the second feature map within each of the m groups of second feature images is different. It can be understood that the size of the second feature map within a group corresponds to the size of the first feature map after processing by the backbone network. For example, for a group of second feature maps P3, P4, P5, and P6, the size of second feature map P3 is the same as the size of first feature map C3; the size of second feature map P4 is the same as the size of first feature map C4; the size of second feature map P5 is the same as the size of first feature map C5; and the size of second feature map P6 is the same as the size of first feature map C6. Since the sizes of the n first feature maps are different, the sizes of the n second feature maps within each group are also different, meaning there are n different sizes among the m groups of second feature maps.

[0083] Since the size of each group of second feature maps corresponds to the size of the first feature map after processing by the backbone network, there are second feature maps of the same size in each group. It can be understood that in the m groups of second feature maps, the size of the second feature map corresponding to the first feature map C3 in each group is the same; the size of the second feature map corresponding to the first feature map C4 in each group is the same; the size of the second feature map corresponding to the first feature map C5 in each group is the same; and the size of the second feature map corresponding to the first feature map C6 in each group is the same.

[0084] The feature fusion in this embodiment is performed between second feature maps of the same size. The fusion method can be addition. M sets of second feature maps of the same size are added together. This means that adding m second feature maps of the same size yields n fused feature maps. Please refer to... Figure 9 The four sets of second feature maps with the same size are added together to obtain four fused feature maps.

[0085] Sub-step S142: Upsample the n fused feature maps to obtain a third feature map.

[0086] Since the n fused feature maps have different sizes, in order to facilitate the annotation of the formula to be annotated, the n fused feature maps are upsampled so that the n feature maps are restored to the size of the original image, and a third feature map is obtained. The number of third feature maps is n.

[0087] Please see Figure 9 , Figure 9 In this context, U represents upsampling, which involves upsampling the four fused feature maps to obtain four third feature maps.

[0088] The target image is obtained based on the third feature map. In some implementations, the target image includes a binary image; please refer to [link to relevant documentation]. Figure 10 Step S150 may include the following sub-steps:

[0089] Sub-step S151: Obtain the probability map corresponding to the third feature map and the threshold map corresponding to the third feature map.

[0090] Predict the probability map and threshold map based on the third feature map.

[0091] One approach is to use a preset function to predict the probability map and threshold map based on the third feature map.

[0092] As another approach, probability maps and threshold maps are predicted using a pre-trained formula-annotated model. Alternatively, as... Figure 2 As shown, the formula annotation model 100 also includes a formula annotation network 140. The third feature map output by the feature fusion network 130 is input into the formula annotation network 140 to obtain the probability map and threshold map output by the formula annotation network 140.

[0093] Sub-step S152: Perform a differentiable operation on the probability map and the threshold map through a differentiable binarization network to obtain a binary map.

[0094] In this embodiment, a differentiable binarization network is used to perform differentiable operations on the probability map and the threshold map to obtain a binary map. A binary map refers to a digital image in which each pixel has only two possible values ​​(0 or 1).

[0095] As one approach, the binarization function in a differentiable binarization network defines the correspondence between the probability map, the threshold map, and the binary map. Based on this correspondence, the corresponding binary maps of the probability map and the threshold map are obtained. Optionally, the expression of the binarization function is as follows:

[0096]

[0097] Among them, B i,j The value represents the position in the binary image; e is a mathematical constant; k is the threshold, optionally k can be 50; P i,j T represents the value of the location in the probability map. i,j This represents the value at the threshold map location.

[0098] Combining formula (1), after obtaining the probability map and the threshold map, the value P of the probability map position is obtained. i,j And obtain the value T of the threshold map position. i,j The value P of the probability map position i,j And obtain the value T of the threshold map position. i,j Substituting into formula (1), we obtain the value B of the binary map position. i,j The value B of the binary map position i,j Used to represent a binary image. In a binary image, the foreground (i.e., formula) region is displayed as 1, and the background (non-formula) region is displayed as 0.

[0099] Sub-step S153: Determine the binary image as the target image.

[0100] Using a binary image as the target image, the formula to be annotated is implemented.

[0101] In other implementations, the target image includes a probability map; see [link to relevant documentation]. Figure 11 Step S150 may include the following sub-steps:

[0102] Sub-step S154: Obtain the probability map corresponding to the third feature map.

[0103] Predicting a probability map based on the third feature map can be done in one way: by inferring the probability map using a pre-defined function (e.g., the sigmoid function). Alternatively, the probability map can be predicted using a pre-trained formula-annotated model. Optionally, as... Figure 2 As shown, the third feature map output by the feature fusion network 130 is input into the formula annotation network 140 to obtain the probability map output by the formula annotation network 140.

[0104] Sub-step S155: Determine the probability map as the target image.

[0105] In other implementations, the target image includes a threshold map; see [link to relevant documentation]. Figure 12 Step S150 includes the following sub-steps:

[0106] Sub-step S156: Obtain the threshold map corresponding to the third feature map.

[0107] Predicting the threshold map based on the third feature map can be done in one way: by inferring the threshold map using a pre-defined function. Alternatively, the threshold map can be predicted using a pre-trained formula-annotated model. Optionally, as... Figure 2 As shown, the third feature map output by the feature fusion network 130 is input into the formula annotation network 140 to obtain the threshold map output by the formula annotation network 140.

[0108] Sub-step S157: Determine the threshold image as the target image.

[0109] In other real-time methods, pre-training Figure 2The formula annotation network 140 in the formula annotation model 100 shown is specifically designed as follows: after the third feature map output by the feature fusion network 130 of the formula annotation model 100, a probability map and a threshold map are inferred based on the third feature map, and then a binary map is calculated based on the probability map and the threshold map. The calculated probability map, threshold map and binary map are used as image labels to train the above-mentioned training formula annotation network 140. The loss function is calculated for the image output by the trained formula annotation network 140 and the image sample. If the loss function converges, the trained formula annotation network 140 is obtained, thereby obtaining the trained formula annotation model 100.

[0110] Specifically, when calculating the aforementioned loss functions, it is necessary to calculate the loss function between the probability map and its labels, the loss function between the threshold map and its labels, and the loss function between the binary map and its labels. Taking the calculation of the loss function between the binary map and its labels as an example:

[0111] The differentiable binary function in formula (1) can be simplified to:

[0112]

[0113] In formula (2), f(x) is the value of the binary map position, x = P i,j -T i,j ;

[0114] Optionally, the loss function is chosen to be the binary cross-entropy loss function, which is:

[0115] BCE loss =-ylog(f(x))-(1-y)log(1-f(x)) (3)

[0116] Among them, BCE lpss y is the loss value, and y is the image label of the binary image, which can be 0 or 1.

[0117] For a binary graph, when it is the foreground (i.e., the region where the formula is located), y = 1.

[0118]

[0119] When the background is the area outside the formula region, y = 0.

[0120]

[0121] For l respectively + l - Differentiation yields:

[0122]

[0123]

[0124] When annotating the formulas to be labeled, the original image is input into the trained formula annotation model 100. The formula annotation model 100 outputs at least one of a binary image, a probability image, and a threshold image. The output results can be based on pre-set output rules. For example, if the output probability image is pre-set, the above model will output the probability image.

[0125] The formula annotation method provided in this embodiment enhances image features through cascaded feature enhancement networks and feature fusion networks, facilitating subsequent annotation of the formulas to be annotated. Even when the formulas to be annotated have problems such as large formula region size span, irregular shape, or several formula regions being closely adjacent, they can still be accurately annotated, ensuring high adaptability of the formula annotation model. Furthermore, based on massive data and the completeness of the model architecture, the model has strong expressive power and generalization ability, ensuring high accuracy. The formula detection model supports simultaneous access to services by multiple users and elastic scaling function, ensuring strong robustness of the model.

[0126] Optionally, in scenarios where the cascaded feature enhancement network includes three cascaded feature enhancement networks, such as... Figure 13 As shown, the formula annotation method may include: first, obtaining the original image; then, inputting the original image into the backbone network to obtain the first feature maps C3, C4, C5, and C6 corresponding to the downsampling processing of the backbone network by 4x, 8x, 16x, and 32x respectively; then, inputting the first feature maps C3-C6 into a cascaded feature enhancement network composed of three cascaded feature enhancement networks FEN to obtain three sets of second feature maps output by the cascaded feature enhancement network. Each set of second feature maps includes four images, and the size of each set of second feature maps corresponds to that of the first feature map. It can be understood that each set of second feature maps contains a second feature map with the same size as the first feature map C3. The feature maps are as follows: each group of second feature maps contains a second feature map with the same size as the first feature map C4, each group of second feature maps contains a second feature map with the same size as the first feature map C5, and each group of second feature maps contains a second feature map with the same size as the first feature map C6. These three groups of second feature maps are output into a feature fusion network (FFN). The FFN fuses the three groups of second feature maps to obtain a feature map (this feature map is the third feature map in the above embodiment). A probability map and a threshold map are estimated using the feature map. A differentiable operation is then performed on the probability map and the threshold map to obtain a binary map. The formula to be labeled is then labeled in the obtained binary map. Optionally, in addition to using a binary map, a probability map or a threshold map can also be used as the final labeled image. Figure 14 The image in (a) is the original image, and the image after annotation is shown in 14(b). The annotation of the formula to be annotated is achieved by using annotation boxes (squares) and text annotations (SXWL).

[0127] The specific details of the formula annotation method in this embodiment can be found in the above embodiments, and will not be repeated here.

[0128] To implement the above-described formula annotation method, this embodiment provides a formula annotation device. Figure 15 A block diagram of a formula annotation apparatus provided in an embodiment of this disclosure is shown. Please refer to [link / reference]. Figure 15 The formula annotation device 200 includes: an acquisition module 210, a downsampling module 220, an enhancement module 230, a fusion module 240, and an annotation module 250.

[0129] The acquisition module 210 is used to acquire the original image, wherein the original image contains the formula to be labeled;

[0130] The downsampling module 220 is used to perform downsampling processing on the original image by n multiples through the backbone network, and obtain the feature map corresponding to each multiple as the first feature map, where n is an integer greater than 2, and the n first feature maps have different sizes;

[0131] The enhancement module 230 is used to enhance the representational ability of the n first feature maps through a cascaded feature enhancement network to obtain m sets of second feature maps, where m is an integer greater than 2. The cascaded feature enhancement network includes m sequentially cascaded feature enhancement networks, and each of the m sets of second feature maps includes n second feature maps.

[0132] The fusion module 240 is used to perform feature fusion processing on the m groups of second feature maps through a feature fusion network to obtain a third feature map, wherein the size of the third feature map is the same as the size of the original image;

[0133] The annotation module 250 is used to obtain a target image based on the third feature map, wherein the target image includes annotation markers for annotating the formula to be annotated.

[0134] Optionally, the enhancement module 230 includes: a first enhancement module, a second enhancement module, and a loop module.

[0135] The first enhancement module is used to input the n first feature maps into the first feature enhancement network in the cascaded feature enhancement network for representation enhancement processing, and obtain a set of feature maps output by the first feature enhancement network, wherein the set of feature maps includes n feature maps;

[0136] The second enhancement module is used to input the n feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representation enhancement processing, and obtain a set of feature maps output by the next feature enhancement network.

[0137] The loop module is used to replace the first feature enhancement network with the next feature enhancement network and continue to execute the above steps of inputting a set of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network for representational capability enhancement processing, and obtaining a set of feature maps output by the next feature enhancement network, until all m feature enhancement networks in the cascaded feature enhancement network output a set of feature maps, and the feature maps output by the m feature enhancement networks in the cascaded feature enhancement network are used as the m sets of second feature maps.

[0138] Optionally, the first enhancement module includes: an enhancement input module and an enhancement output module.

[0139] An enhanced input module is used to input the n first feature maps into the upsampling enhancement module of the first feature enhancement network for processing, so as to obtain a set of feature maps output by the upsampling enhancement module of the first feature enhancement network;

[0140] An enhanced output module is used to input a set of feature maps output by the upsampling enhancement module into the downsampling enhancement module of the first feature enhancement network for processing, so as to obtain a set of feature maps output by the downsampling enhancement module.

[0141] Optionally, the second feature images within each of the m groups of second feature images have different sizes, and there are second feature images of the same size between each group. The fusion module 240 includes an addition module and a recovery module.

[0142] The addition module is used to add the second feature images of the same size in the m groups of second feature images to obtain n fused feature images;

[0143] The recovery module is used to upsample the n fused feature maps to obtain a third feature map.

[0144] Optionally, the annotation module 250 includes: a first acquisition module, a binary image acquisition module, and a first annotation module.

[0145] The first acquisition module is used to acquire the probability map corresponding to the third feature map and the threshold map corresponding to the third feature map;

[0146] The binary image acquisition module is used to perform differentiable operations on the probability image and the threshold image through a differentiable binarization network to obtain a binary image.

[0147] The first annotation module is used to identify the binary image as the target image.

[0148] Optionally, the annotation module 250 includes a second acquisition module and a second annotation module.

[0149] The second acquisition module is used to acquire the probability map corresponding to the third feature map;

[0150] The second annotation module is used to identify the probability map as the target image.

[0151] Optionally, the annotation module 250 includes a third acquisition module and a third annotation module.

[0152] The third acquisition module is used to acquire the threshold map corresponding to the third feature map;

[0153] The third annotation module is used to determine the threshold map as the target image.

[0154] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0155] Figure 16 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 16 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0156] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the formula annotation method described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0157] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described formula annotation method.

[0158] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the formula annotation method described above. For example, the computer-readable storage medium may be the memory 702 including the program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the formula annotation method described above.

[0159] Figure 17 This is a block diagram illustrating an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. (Refer to...) Figure 17 The electronic device 1900 includes a processor 1922, which may be one or more, and a memory 1932 for storing computer programs executable by the processor 1922. The computer program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1922 may be configured to execute the computer program to perform the formula notation method described above.

[0160] Additionally, the electronic device 1900 may also include a power supply component 1926 and a communication component 1950. The power supply component 1926 can be configured to perform power management of the electronic device 1900, and the communication component 1950 can be configured to enable communication of the electronic device 1900, such as wired or wireless communication. Furthermore, the electronic device 1900 may also include an input / output (I / O) interface 1958. The electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM etc.

[0161] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the formula annotation method described above. For example, the non-transitory computer-readable storage medium may be the memory 1932 including the program instructions described above, which may be executed by the processor 1922 of the electronic device 1900 to complete the formula annotation method described above.

[0162] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described formula annotation method when executed by the programmable device.

[0163] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0164] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0165] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A formula annotation method characterized by comprising: The method comprises: acquiring an original picture, wherein the original picture contains a formula to be labeled; performing n times of down-sampling processing on the original picture through a backbone network to obtain a feature map corresponding to each of the n times of down-sampling as a first feature map, wherein n is an integer greater than 2, and the sizes of the n first feature maps are different; performing representation ability enhancement processing on the n first feature maps through a cascaded feature enhancement network to obtain m groups of second feature maps, wherein m is an integer greater than 2, the cascaded feature enhancement network comprises m feature enhancement networks connected in sequence, each of the m groups of second feature maps comprises n second feature maps, and the representation ability enhancement processing on the n first feature maps through the cascaded feature enhancement network to obtain the m groups of second feature maps comprises: inputting the n first feature maps into a first feature enhancement network in the cascaded feature enhancement network to perform representation ability enhancement processing to obtain a group of feature maps output by the first feature enhancement network, wherein the group of feature maps comprises n feature maps; inputting the group of feature maps output by the first feature enhancement network into a next feature enhancement network of the first feature enhancement network to perform representation ability enhancement processing to obtain a group of feature maps output by the next feature enhancement network; replacing the first feature enhancement network with the next feature enhancement network, and continuing to perform the step of inputting the group of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network to perform representation ability enhancement processing to obtain a group of feature maps output by the next feature enhancement network until the m feature enhancement networks in the cascaded feature enhancement network all output a group of feature maps, and taking the feature maps output by the m feature enhancement networks in the cascaded feature enhancement network as the m groups of second feature maps; performing feature fusion processing on the m groups of second feature maps through a feature fusion network to obtain a third feature map, wherein the size of the third feature map is the same as that of the original picture; acquiring a target image according to the third feature map, wherein the target image comprises a labeling mark for labeling the formula to be labeled.

2. The method of claim 1, wherein, The feature enhancement network comprises an up-sampling enhancement module and a down-sampling enhancement module, and the inputting of the n first feature maps into the first feature enhancement network in the cascaded feature enhancement network to obtain a group of feature maps output by the first feature enhancement network comprises: inputting the n first feature maps into the up-sampling enhancement module of the first feature enhancement network to perform processing to obtain a group of feature maps output by the up-sampling enhancement module of the first feature enhancement network; inputting the group of feature maps output by the up-sampling enhancement module into the down-sampling enhancement module of the first feature enhancement network to perform processing to obtain a group of feature maps output by the down-sampling enhancement module.

3. The method of claim 1, wherein, The sizes of the second feature images in each of the m groups of second feature images are different, and there are second feature images of the same size between the groups, the feature fusion network is used to perform feature fusion processing on the m groups of second feature images, and a third feature map is obtained, including: adding the second feature images of the same size in the m groups of second feature images to obtain n fused feature maps; performing up-sampling processing on the n fused feature maps to obtain a third feature map.

4. The method of claim 1, wherein, The third feature map is used to obtain a target image, including: obtaining a probability map corresponding to the third feature map and a threshold value map corresponding to the third feature map; performing a differentiable operation on the probability map and the threshold value map through a differentiable binarization network to obtain a binary map; determining the binary map as the target image.

5. The method of claim 1, wherein, The third feature map is used to obtain a target image, including: obtaining a probability map corresponding to the third feature map; determining the probability map as the target image.

6. The method of claim 1, wherein, The third feature map is used to obtain a target image, including: obtaining a threshold value map corresponding to the third feature map; determining the threshold value map as the target image.

7. A formula annotation device characterized by comprising: The device includes: an acquisition module configured to acquire an original picture, wherein the original picture contains a formula to be labeled; a down-sampling module configured to perform n-multiple down-sampling processing on the original picture through a backbone network to obtain feature maps corresponding to each multiple as first feature maps, wherein n is an integer greater than 2, and the sizes of the n first feature maps are different; an enhancement module configured to perform representation ability enhancement processing on the n first feature maps through a cascaded feature enhancement network to obtain m groups of second feature maps, wherein m is an integer greater than 2, the cascaded feature enhancement network includes m feature enhancement networks connected in sequence, each of the m groups of second feature maps includes n second feature maps, and the enhancement module includes a first enhancement module, a second enhancement module, and a circulation module; the first enhancement module is configured to input the n first feature maps into a first feature enhancement network in the cascaded feature enhancement network to perform representation ability enhancement processing and obtain a group of feature maps output by the first feature enhancement network, wherein the group of feature maps includes n feature maps; the second enhancement module is configured to input the n feature maps output by the first feature enhancement network into a next feature enhancement network of the first feature enhancement network to perform representation ability enhancement processing and obtain a group of feature maps output by the next feature enhancement network; the circulation module is configured to replace the first feature enhancement network with the next feature enhancement network and continue to perform the above step of inputting the group of feature maps output by the first feature enhancement network into the next feature enhancement network of the first feature enhancement network to perform representation ability enhancement processing until the m feature enhancement networks in the cascaded feature enhancement network all output a group of feature maps, and the feature maps output by the m feature enhancement networks in the cascaded feature enhancement network are taken as the m groups of second feature maps. A fusion module is configured to perform feature fusion processing on the m groups of second feature maps by a feature fusion network to obtain a third feature map, wherein a size of the third feature map is the same as a size of the original picture. A labeling module is configured to obtain a target image according to the third feature map, wherein the target image comprises a labeling mark for labeling the to-be-labeled formula.

8. An electronic device, comprising: The computer program comprises: a memory having stored thereon a computer program; a processor configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images