Image information extraction method and device, equipment and storage medium

Abstract

The invention discloses an image information extraction method and device, equipment and a storage medium, and the method is applied to first equipment, and comprises the steps: carrying out the image enhancement of a first image under the condition that the first image comprises a target object, and obtaining an enhanced second image; performing image information extraction on the target object by using the extraction model of the target object to obtain attribute information of a plurality of elements of the target object and association information among the elements; and converting the attribute information of the plurality of elements of the target object and the association information among the elements into a target file format to obtain a target file of the target object. Therefore, the mobile terminal can quickly and accurately extract the attribute information and the associated information of the elements in each object from the image, and convert the attribute information and the associated information into a standardized target file format, so that the readability and universality of the file are improved. In addition, secondary editing of the target object can be realized based on the target file.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image information extraction method, device, equipment and storage medium. Background Art

[0002] Images can intuitively display data and information, making data easier to understand and analyze, and play an important role in various fields. Due to image format limitations, it is impossible to directly obtain the displayed data and information from images. In related technologies, multi-segment processing methods are usually used to extract information from images. First, key elements are detected from the image, and then the detected key elements are further processed and analyzed based on a series of post-processing rules to obtain the relevant information displayed in the image.

[0003] However, the existing image information extraction methods have low extraction accuracy and poor versatility, and are unable to cope with complex and changeable real environments. Summary of the invention

[0004] The embodiments of the present application are intended to provide an image information extraction method, apparatus, device and storage medium to improve the accuracy and versatility of image information extraction.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, a method for extracting image information is provided, comprising:

[0007] In the case where the first image to be processed includes the target object, performing image enhancement on the first image to obtain an enhanced second image;

[0008] determining one or more target objects and their object types included in the second image;

[0009] Determining an extraction model of the target object according to the object type of the target object;

[0010] Extracting image information of the target object using the extraction model of the target object to obtain attribute information of multiple elements of the target object and association information between the elements;

[0011] The attribute information of the multiple elements of the target object and the association information between the elements are converted into a target file format to obtain a target file of the target object.

[0012] In a second aspect, an image information extraction device is provided, comprising:

[0013] an image enhancement unit, configured to, when a first image to be processed includes a target object, perform image enhancement on the first image to obtain an enhanced second image;

[0014] a determination unit, configured to determine one or more target objects and their object types included in the second image; and determine an extraction model of the target object according to the object type of the target object;

[0015] An extraction unit, configured to extract image information of the target object using an extraction model of the target object, and obtain attribute information of multiple elements of the target object and association information between the elements;

[0016] The extraction unit is further used to convert the attribute information of the multiple elements of the target object and the association information between the elements into a target file format to obtain a target file of the target object.

[0017] According to a third aspect, a first device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0018] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0019] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0020] According to a fifth aspect, a computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0021] The embodiments of the present application provide an image information extraction method, device, equipment and storage medium, which can enable a mobile terminal to quickly and accurately extract attribute information and associated information of elements in each object from an image, and convert it into a standardized target file format, thereby improving the readability and versatility of the file. Furthermore, secondary editing of the target object can also be achieved based on the target file. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the first process of the image information extraction method in the embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of the single image information extraction interface in the embodiment of this application Figure 1 ;

[0024] Figure 3 This is a schematic diagram of the single image information extraction interface in the embodiment of this application Figure 2 ;

[0025] Figure 4 Schematic diagram of the multi-image information extraction interface in the embodiment of this application Figure 1 ;

[0026] Figure 5 Schematic diagram of the multi-image information extraction interface in the embodiment of this application Figure 2 ;

[0027] Figure 6 This is a schematic diagram of the automatic chart information extraction interface in the embodiment of the present application. Figure 1 ;

[0028] Figure 7 This is a schematic diagram of the automatic chart information extraction interface in the embodiment of the present application. Figure 2 ;

[0029] Figure 8 This is a schematic diagram of an interface for copying and sharing the results of chart information extraction in an embodiment of the present application;

[0030] Fig. 9 This is a schematic diagram of the editing interface of the chart data source in the embodiment of the present application;

[0031] Fig.10 A schematic diagram of the editing interface of chart types and chart elements in an embodiment of the present application;

[0032] Fig.11 This is a schematic diagram of the mind map editing interface in the embodiment of this application;

[0033] Fig.12 This is a schematic diagram of an interface for extracting image information in the scan process according to an embodiment of the present application;

[0034] Fig.13 This is a second flow chart of the image information extraction method in the embodiment of the present application;

[0035] Fig.14 This is a schematic diagram of an image sample in an embodiment of the present application;

[0036] Fig.15 Schematic diagram of the training process of the image pre-classification module in the embodiment of the present application;

[0037] Fig.16 This is a flow chart of the fine-tuning process of the image pre-classification module in an embodiment of the present application;

[0038] Fig.17 This is a schematic diagram of the image quality enhancement process in an embodiment of the present application;

[0039] Fig.18 A schematic diagram showing the comparison before and after trapezoidal correction in an embodiment of the present application;

[0040] Fig.19 This is a schematic diagram of the trapezoidal correction process in the embodiment of the present application;

[0041] Fig. 20 This is a schematic diagram of the process of extracting chart information in an embodiment of the present application;

[0042] Fig.21 This is a schematic diagram of the mind map annotation in the embodiment of this application;

[0043] Fig. 22 A schematic diagram for notating the flow chart in the embodiments of the present application;

[0044] Fig.23 A schematic diagram showing the detection frames of the center point and the intersection of the pie chart in the embodiment of the present application;

[0045] Fig.24 This is a schematic diagram of the composition structure of the image information extraction device in the embodiment of the present application;

[0046] Fig.25 This is a schematic diagram of the composition structure of the first device in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0048] Figure 1 FIG. 1 is a schematic diagram of a first process of the image information extraction method in an embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0049] Step 101: When a first image to be processed includes a target object, image enhancement is performed on the first image to obtain an enhanced second image;

[0050] The first image is an image to be processed, and the first image may specifically be an image captured by a camera, a currently displayed image, an image determined through an image selection operation, and the like.

[0051] Image enhancement is used to improve image quality, including clarity, color reproduction, and detail performance. By enhancing the edge information, contour information, and contrast of the image, the differences between the features of different objects in the image can be made more significant, thereby improving the image's resolution and detail performance. This helps users more accurately identify and distinguish different objects and features in the image. The enhanced image can more effectively extract and identify feature information, improving the accuracy and efficiency of subsequent image information extraction.

[0052] In some embodiments, performing image enhancement on a first image to obtain an enhanced second image includes: performing image quality analysis on the first image to determine one or more image enhancement algorithms corresponding to the first image; and performing image enhancement processing on the first image based on the one or more image enhancement algorithms to obtain an enhanced second image.

[0053] Image quality analysis is used to analyze image quality and select appropriate image enhancement algorithms to enhance the image according to the actual quality of the image, thereby improving the image quality and the accuracy of subsequent image information extraction. Among them, one or more image enhancement algorithms include but are not limited to at least one of the following: trapezoidal correction, surface correction, moiré removal, deblurring, etc.

[0054] Exemplarily, the input image is subjected to trapezoidal correction by a trapezoidal correction model, the input image is subjected to surface correction by a surface correction model, the input image is subjected to demoiré by a demoiré model, and the input image is deblurred by a deblurring model.

[0055] In some embodiments, the enhancement model performs image enhancement processing based on the corresponding image enhancement algorithm. Exemplarily, the trapezoidal correction model is configured to implement the steps of the following method: obtaining the original document image to be corrected; predicting the document edge probability to obtain the document edge; performing straight line fitting on the document edge to obtain a straight line set; locating the document vertices according to the straight line set to obtain the four vertex coordinates of the document; obtaining the coordinates of the four target vertices of the document in the target plane according to the four vertex coordinates of the document; calculating the perspective transformation matrix M according to the four vertex coordinates and the coordinates of the four target vertices; performing padding operation on the image to be corrected according to the four vertex coordinates; and then performing perspective transformation on the padded image to be corrected according to M to complete the document correction.

[0056] In some embodiments, the enhancement model is a deep neural network model that implements an image enhancement process or implements multiple image enhancements.

[0057] When the enhancement model is a deep neural network model, the method also includes: obtaining a third training sample set, the third training sample including a third image sample and an image true value; constructing an initial enhancement model; using the third training sample set to train the initial enhancement model of the target object to obtain a trained enhancement model.

[0058] Exemplarily, the deblurring model is a deep neural network model, and the training process of the deblurring model can be as follows:

[0059] S1: Collect clear image data and build dataset 1;

[0060] S2: Add random noise to the clear image to obtain data set 2.

[0061] S3: add motion blur to the noisy image to obtain data set 3;

[0062] S4: Add tadpole-like deformation to the blurred image to obtain dataset 4;

[0063] S5: Use data set 1 and data set 4 to train a deblurring model to obtain a trained deblurring model.

[0064] In some embodiments, the method further includes: if the first image does not include the target object, determining that the first image is an unextractable image and skipping the steps of the image information extraction method. By identifying unextractable images, the scope of application of the image information extraction method provided in the embodiment of the present application is expanded and the processing efficiency is improved.

[0065] In some embodiments, the method further includes: when the first image does not include the target object, outputting first prompt information to prompt the user that the first image is an unextractable image. By outputting the first prompt information, the user is promptly informed that the current image is a rejected image, thereby improving the user experience.

[0066] In some embodiments, the method further includes: pre-classifying the first image to determine whether the first image includes the target object. Exemplarily, the first image is pre-classified using an image pre-classification model to obtain a classification result, and the classification result is used to indicate whether the first image includes the target object. The image recognition model includes a feature extraction network and a classification head, and the feature extraction network is configured to extract features from the first image to obtain a feature embedding representation, and the classification head is configured to perform image classification based on the feature embedding representation to determine whether the first image is the target object.

[0067] Pre-classification is used to determine whether the image type of the first image is an image type that supports image information extraction operations, that is, whether the image type includes an extractable target object; if it does, then the subsequent image information extraction steps are continued; if it does not, then the first image can be determined to be a rejected image and ignored to avoid wasting computing power.

[0068] In some embodiments, the method also includes: obtaining an original training sample set, the original training sample set including original image samples and their category labels, the category labels being used to indicate whether the image samples contain a target object; performing data enhancement on the original image samples in the original training sample set to obtain an enhanced first training sample set; and using the first training sample set to train a first image pre-classification model to obtain a trained image pre-classification model.

[0069] Exemplarily, data enhancement may include: performing random data enhancement on image samples based on one or more random data enhancement strategies to obtain enhanced image samples; performing data mixing operations on the enhanced image samples to obtain mixed image samples; and constructing a first training sample set using the enhanced image samples and the mixed image samples.

[0070] Exemplarily, training a first image pre-classification model using a first training sample set to obtain a trained image pre-classification model may include: an enhanced image sample is input into the first image pre-classification model to obtain an enhanced image feature embedding representation output by a feature extraction network of the first image pre-classification model and a classification result output by a classification head; a mixed image sample is input into the first image pre-classification model to obtain a mixed image feature embedding representation output by a feature extraction network of the first image pre-classification model; a first loss function is calculated using the classification result and the category label; based on the first loss function, a gradient perturbation is added to the enhanced image feature embedding representation to obtain a perturbed image feature embedding representation; a second loss function is calculated based on the perturbed image feature embedding representation, the enhanced image feature embedding representation and the mixed image feature embedding representation; and the weight parameters of the image pre-classification model are optimized based on the second loss function to obtain a second image pre-classification model.

[0071] Feature Embedding Representation usually refers to the process of converting high-dimensional discrete features (such as pixel values, colors, textures, etc. of an image) into low-dimensional continuous feature vectors. Feature embedding representation can be understood as image feature data. This representation method aims to capture key information in the image while reducing the dimensionality of the data to facilitate subsequent machine learning, deep learning, or information retrieval tasks.

[0072] In some embodiments, the second loss function includes: Info NCE (Information Noise-Contrastive Estimation) loss function and SupCon (Supervised Contrastive Learning) loss function. Info NCE loss function is a loss function commonly used in contrastive learning. The core idea is to learn meaningful representations by maximizing the similarity of similar sample pairs while minimizing the similarity of dissimilar sample pairs. SupCon loss function bridges the gap between self-supervised learning and fully supervised learning, and enables contrastive learning to be applied in supervised environments.

[0073] In some embodiments, the method also includes: freezing the weight parameters of the feature extraction network of the second image preclassification model, optimizing the weight parameters of the classification head of the second image preclassification model, and obtaining a trained image preclassification model; or, replacing the classification head of the second image preclassification model with a new classification head; freezing the weight parameters of the feature extraction network of the second image preclassification model, optimizing the weight parameters of the new classification head of the second image preclassification model, and obtaining a trained image preclassification model.

[0074] Step 102: determining one or more target objects and their object types included in the second image;

[0075] When the second image includes multiple target objects, each target object may be extracted separately, or multiple target objects may be extracted simultaneously.

[0076] In some embodiments, one or more target objects and their object types in the second image are identified by an image recognition model.

[0077] In some embodiments, the target object and its object type are determined by the second device; accordingly, the method further includes: uploading the second image to the second device so that the second device performs image detection on the second image, and determines the location information and object type of one or more target objects in the second image; receiving the location information and object type of the target object sent by the second device; and determining the one or more target objects in the second image based on the location information of the one or more target objects. Exemplarily, the second device can be a network device, and the image classification task can be implemented by the network side, sharing the processing burden of the mobile terminal, while improving processing efficiency and performance.

[0078] Step 103: determining an extraction model of the target object according to the object type of the target object;

[0079] In some embodiments, one object type corresponds to one extraction model for information extraction, or multiple object types correspond to one extraction model for information extraction.

[0080] In some embodiments, the method also includes: obtaining a second training sample set, the second training sample includes a second image sample and its label information, the second image sample contains a target object, and the label information includes the position information of the first detection box of each element of the target object, the element type, the element identifier, and the associated element identifier; constructing an initial extraction model of the target object; using the second training sample set to train the initial extraction model of the target object to obtain a trained extraction model of the target object.

[0081] In some embodiments, the extraction model is an end-to-end neural network model.

[0082] When the target object is a mind map, the extraction model of the mind map includes: a backbone network, a feature pyramid network and a detection head; the mind map is input into the backbone network to output a first feature map, the first feature map is input into the feature pyramid network for multi-scale feature extraction to output a second feature map, and the first feature map and the second feature map are input into the detection head to output the element type of each element of the mind map, the position information of the first detection box and the position information of the second detection box.

[0083] When the target object is a flowchart, the extraction model of the flowchart includes an encoder and a decoder; the encoder includes a backbone network and a self-attention module, the flowchart is input into the backbone network for multi-scale feature extraction to output a multi-scale feature map, the feature map of the first scale is input into the self-attention module to output a global feature map, the global feature map and the feature maps of other scales are fused to obtain multiple first feature vectors output by the encoder; the multiple first feature vectors are output from the decoder to the type information of each element of the flowchart, the position information of the first detection box, and the position information of the second detection box.

[0084] Step 104: extracting image information of the target object using the extraction model of the target object to obtain attribute information of multiple elements of the target object and association information between the elements;

[0085] The attribute information and the association information are used to at least represent the positions and association relationships of multiple elements constituting the target object. In some embodiments, the attribute information includes the element type of the current element and the position information of the first detection box, and the association information includes the position association information of the first detection box and the second detection box of the current element, where the first detection box is the bounding rectangle of the current element, and the second detection box is the bounding rectangle of the associated element of the current element.

[0086] Exemplarily, the position information of the first detection frame includes the coordinates of the upper left corner and the lower right corner of the first detection frame, and the position association information includes the coordinates of the upper left corner of the first detection frame and the coordinates of the upper left corner of the second detection frame.

[0087] Exemplarily, the position information of the first detection frame includes the first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame. Exemplarily, the second center coordinates may be relative coordinates of the center of the second detection frame relative to the center of the first detection frame.

[0088] The detection box (also called the bounding box bbox) is a rectangular box that describes the location and range of the target object in the image. It is usually determined by four key parameters: x, y, width and height. Among them, x and y represent the coordinate values ​​of the upper left corner (or center point) of the rectangular box in the image coordinate system, and the width and height determine the size of the rectangular box. Through the bbox, we can accurately know the specific location and size of the target object in the image.

[0089] It should be noted that the attribute information and the associated information can be expressed separately or in combination. For example, the center coordinates of the first detection frame are (x1, y1) and the size is (w, h), and the center coordinates of the second detection frame are (x2, y2); the attribute information can be expressed by a four-tuple [x1, y1, w, h], and the associated information can be expressed by a four-tuple [x1, y1, x2, y2]. The attribute information and the associated information can also be expressed by a six-element [x1, y1, w, h, x2, y2].

[0090] The target objects include at least one of the following charts: mind map, flow chart, pie chart, bar chart;

[0091] The element types of the mind map include at least one of the following: root node, intermediate node, leaf node, thick node;

[0092] The element types of a flowchart include at least one of the following: process, decision, subprocess, start / end, document, data, condition, flow direction;

[0093] The element types of a pie chart include at least one of the following: icon title, legend, center point, sector point, text area;

[0094] The element types of the bar chart include at least one of the following: icon title, bar shape, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

[0095] In some embodiments, the attribute information includes at least element text information;

[0096] The extraction model is configured to perform the following steps: detect multiple elements of the target object, and obtain the first coordinates and sizes of the detection boxes of the multiple elements as attribute information of the multiple elements; determine the number of detection boxes of the target object; when the number of detection boxes is greater than or equal to a first threshold, extract text information based on the first coordinates and size of the third detection box to obtain text information of the third detection box, and the third detection box is part or all of the detection box in the target object; when the number of detection boxes is less than the first threshold, extract text information on the target object to obtain text information of the target object; determine the text information of the fourth detection box based on the text information of the target object and the first coordinates and size of the fourth detection box.

[0097] When the number of coordinate frames is small (less than 50), you can directly perform optical character recognition (OCR) text recognition on the entire image, and then assemble the content of OCR text recognition based on the mind map node detection frame to obtain the text information of the corresponding node. When the number of coordinate frames is large (greater than or equal to 50), perform text recognition on each node separately to ensure the accuracy of text recognition.

[0098] In some embodiments, the attribute information includes at least: element identification, element type, style, padding, and border.

[0099] Step 105: Convert the attribute information of the multiple elements of the target object and the association information between the elements into a target file format to obtain a target file of the target object.

[0100] The target file format can be a unified and standardized format, which can solve the problems of non-interoperability faced by various applications, fields, systems, cross-terminals, and ecosystems based on universality, readability, progressiveness, modularity, extensibility, and unboundedness.

[0101] The target file format may be a custom file format or a standard file format. For example, the standard file format may be XMind, JSON, or XML.

[0102] In some embodiments, the method further includes: acquiring a first image to be processed.

[0103] Exemplarily, the content of the first file is displayed on the user interface of the first application; the first image is obtained from the first file. The first application is used to display the content of the first file, and the first application can also be used to edit the content of the first file, by extracting image information from the target object in the picture format in the first file, obtaining an editable extraction result, and then the extraction result can be edited again.

[0104] The format of the first file may be a text format, for example, a text format such as docx, pdf, PPT, etc. The format of the first file may also be a picture format such as jpg, jpeg, gif, etc.

[0105] More specifically, obtaining a first image from a first file includes: in response to a first image selection operation, obtaining the first image from the currently displayed content of the first file; or, determining one or more extractable images from the first file; using the one or more extractable images as the first image to be processed; or, determining one or more extractable images from the first file; and in response to a second image selection operation, obtaining the first image from the one or more extractable images.

[0106] That is, the first image can be obtained from the currently displayed content of the first file according to the first image selection operation. Exemplarily, the first image selection operation can be a user click operation, a screenshot operation, etc., or the first image can be obtained from all extractable images in the first file according to the second image selection operation. All extractable images in the first file can also be used as the first image.

[0107] The extractable images in the first file may specifically be all images included in the first file, or images that meet a preset condition. Exemplarily, the method further includes: determining a first object type to be processed; and determining one or more extractable images from the first file based on the first object type. The first object type is an extractable object type, and an image containing the first object type is determined to be an extractable image.

[0108] In some embodiments, obtaining the first image from the first file includes: starting a camera in response to a third image selection operation; and obtaining the first image captured by the camera.

[0109] That is to say, the first image can also be obtained through the camera, and the third image selection operation can be a camera startup operation. For example, the third application starts the camera to shoot the target object, obtains the first image, and extracts information from the target object based on the image information extraction method provided in the embodiment of the present application. The third application can specifically be an application including a "scan" function, and the "scan" function is started to shoot the target object.

[0110] In some embodiments, the method further includes: in response to the image information extraction operation, displaying one or more object types that can be extracted; in response to the selection operation of the first object type, executing the steps of the image information extraction method, and the object type of the target object belongs to the first object type. In this way, by displaying one or more object types that can be extracted to remind the user, the user is prevented from trying to extract unsupported object types, which affects the user experience.

[0111] The first object type is an extractable object type, and the first object type includes the object type of the target object. In some embodiments, the object type of the target object is a subtype of the first object type. For example, when the first object type is a chart, the chart can be further subdivided into: mind map, flow chart, pie chart, bar chart, line chart, scatter chart, etc. The first object type is a view, and the view can be further subdivided into a computer view or an engineering view.

[0112] In some embodiments, the object type of the target object is a first object type, for example, the first object type is a chart type and the first object type is a formula.

[0113] In some embodiments, the method further includes: displaying the extraction result of the target object in the first display area based on the target file of the target object. Displaying the extraction result to the user helps to perform relevant processing on the extraction result, such as editing, saving, inserting, copying, sharing, etc.

[0114] In some embodiments, the method further includes: displaying the target object in the second display area, the first display area and the second display area being different display areas of the same user interface. Comparing and displaying the extraction result of the target object and the target object (which can also be understood as the original image of the target object), the extraction effect is intuitively displayed to the user.

[0115] In some embodiments, the method further comprises: in response to an insert operation of the extraction result of the target object, inserting the target file of the target object into the first file to which the target object belongs to obtain a second file; based on the second file, displaying the target object and its extraction result in the user interface of the first application; or,

[0116] The method also includes: in response to a replacement operation of the target object extraction result, replacing the target object in the first file with the target file of the target object to obtain a third file; based on the third file, displaying the target object extraction result on the user interface of the first application.

[0117] In some embodiments, when the first image includes a plurality of target objects, the method further includes: determining a selected first target object in response to a selection operation of the target object; and obtaining an insertion operation or a replacement operation of an extraction result of the first target object.

[0118] In some embodiments, the method further includes: editing the extraction result of the target object to obtain a new target file.

[0119] Exemplarily, the extraction result is edited in the first application. The first application has an editing function, and after obtaining the extraction result of the target object, the extraction result is added to the first file, and then the extraction result is displayed by the first application, and the extraction result is edited using the editing tool of the first application.

[0120] In some embodiments, the extraction result of the target object is edited to obtain a new target file, including: in response to an edit operation on the target object, a first edit menu is displayed, the first edit menu includes one or more edit objects; in response to a selection operation on the first edit object, a second edit menu of the first edit object is displayed, the second edit menu includes one or more editable elements of the first edit object; based on the second edit menu, editing information is obtained to edit the extraction result of the target object to obtain a new target file.

[0121] When the target object is a chart, the one or more editing objects include at least one of the following: a data source, a chart type, and a chart element; when the first editing object is a data source, the second editing menu includes a data table, and insert operations and delete operations of the data table; when the first editing object is a chart type, the second editing menu includes the original chart type and one or more replaceable chart types; when the first editing object is a chart element, the second editing menu includes the original style and element setting items of the target object.

[0122] In some embodiments, when the target object is a pie chart or a bar chart, the element setting items include at least one of the following: an element display control item, an element position setting item, and an element content setting item; when the target object is a mind map, the element setting items include at least one of the following: a new branch item, a deleted branch item, a branch element insertion item, a branch element deletion item, an element style setting item, and an element content setting item; when the target object is a flowchart, the element setting items include at least one of the following: an element insertion item, an element deletion item, an element type conversion item, an element content setting item, and an element connection relationship setting item.

[0123] Element style settings can include at least one of the following: color, font, shape, etc.

[0124] In some embodiments, the method further includes: in response to a first format conversion operation, converting the target file into a first file format to obtain a fourth file; the fourth file is used to display the target object in the user interface of the second application, and the first file format is a file format supported by the second application; in response to a second format conversion operation, converting the second file or the third file into the second file format to obtain a fifth file.

[0125] The first format conversion operation can be understood as a save-as operation for the extraction result, and the second format conversion operation can be understood as a save-as operation for the second file or the third file. The save-as operation facilitates file transfer and adapts to different applications.

[0126] The first file format may be a picture format such as jpg, jpeg, gif, etc., and the second file format may be a standard file format, for example, a text format such as docx, pdf, PPT, etc., or a picture format such as jpg, jpeg, gif, etc.

[0127] In order to better reflect the purpose of this application, the target object further illustrates the embodiments of this application using diagrams as examples.

[0128] Figure 2 and Figure 3This is a schematic diagram of the single image information extraction interface in the embodiment of the present application; after the user opens the file using an application with image information extraction function, he can select the image by long pressing the image area in the document, or select the image to be processed by taking a screenshot. Figure 2 The "AI Document Assistant" in the lower left corner of the middle left picture calls up the relevant functions corresponding to the image information extraction operation. Figure 2 In the middle-right picture, the "AI Document Assistant" menu bar displays one or more object types that can be extracted, including "Extract Chart" and "Extract Formula". You can extract a single chart information by selecting the "Extract Chart" function.

[0129] When the user selects "Extract Chart", the AI ​​system will read the image selected by the user, and then run the image information extraction algorithm provided in the embodiment of the present application to analyze and reconstruct the image content. Figure 3 As shown in the middle left picture, it prompts "Chart is being produced".

[0130] When the AI ​​system finishes running the image information extraction algorithm, it will present the extracted chart in the lower area of ​​the screen, such as Figure 3 As shown in the middle right picture. Users can further use the "Replace original image" button to replace the original image with the extracted chart in the original document, so as to facilitate secondary editing of the chart in the original document. Users can also use the "Insert below" button to insert the extracted chart below the corresponding chart in the original document, so as to facilitate comparison of the extraction results.

[0131] Figure 4 and Figure 5 This is a schematic diagram of the multi-image information extraction interface in the embodiment of this application. When the image selected by the user contains multiple chart areas, the user can still use the "Extract Chart" function through the "AI Document Assistant" page. At this time, the AI ​​system will automatically identify and segment the multiple chart areas in the original image, and extract information from the charts in each area separately.

[0132] After the AI ​​system runs the information extraction algorithm, it will present the reconstructed results in the form of pages, with each page presenting a chart information extraction result, such as Figure 5 As shown. Users need to check the extracted results in the page before they can continue to use the function buttons of "Replace original image" or "Insert below".

[0133] Figure 6The following is a schematic diagram of the automatic chart information extraction interface in the embodiment of the present application. As shown in the left figure of Figure 6, the user can also use the chart information extraction function without manually selecting the chart. At this time, when the user opens the file using the application, he can directly click "AI Document Assistant" and select the object type to be extracted in the pop-up menu bar, such as selecting the "Extract Chart" function. Figure 6 When the user selects "Extract Charts", the system will automatically identify all charts in the original document in picture format and display an "Extract" button under each chart. The user can further select the charts for which the chart information is to be extracted, such as Figure 7 It should be noted that only one "Extract" button can be displayed, and all charts can be batch extracted by clicking the "Extract" button.

[0134] When the chart information is successfully extracted, Figure 8 As shown in the left picture, users can use the "Copy" function to convert the chart into a picture format and copy it to other editing tools that support picture formats. Figure 8 As shown in the right picture, users can also use the "Share" function to convert the extracted chart into a specific format: picture, Word, PDF, PPT.

[0135] The client-side editor supports the editing of pie charts, column charts, and mind maps. For pie charts and column charts, because their underlying data formats are in table format, editing them actually translates into editing the underlying table data. For example, Fig. 9 and Fig.10 shown.

[0136] Editing support for pie charts and column charts: data sources support free deletion, addition of rows and columns, and content modification; the title can modify the display position, title content, and whether to display; the legend can adjust the display position and whether to display; pie charts and column charts can be converted into each other.

[0137] The diagram of editing the mind map is as follows Fig.11 shown.

[0138] Supports the editing functions of mind maps: adding subjects and sub-topics at the same level; modifying the text content in topics at all levels; deleting topics at all levels; and modifying topic styles.

[0139] Fig.12 This is a schematic diagram of the image information extraction interface implemented in the scan method of the present application embodiment. The function entry of chart information extraction can also be added in the scan method. When the user opens the scan method, if the AI ​​algorithm recognizes that the current interface is a document and contains image categories supported by graphic information extraction, the scan method can call the chart information extraction capability to extract information and present the results and result export function to the user.

[0140] In order to better reflect the purpose of this application, further examples are given based on the above embodiments of this application. Fig.13 As shown, the technical solution provided in the embodiment of the present application can be divided into the following core functional modules according to the data processing flow:

[0141] Step 1301: Image pre-classification module: Use the terminal-side pre-classification algorithm for the image in the user's current document (or the image actively selected by the user) to determine whether the current image contains an object type that the system can support reconstruction.

[0142] Step 1302: Image quality enhancement module. For supported object types, image enhancement algorithms are used to enhance the quality of the image according to the actual quality of the image. Possible enhancement strategies include: keystone correction, surface correction, moiré removal, and deblurring.

[0143] Step 1303: Information extraction module: According to the chart type, corresponding extraction algorithms can be used for different object types to extract attribute information of multiple elements and association information between elements, and convert them into a standardized file format to obtain a target file.

[0144] Step 1304: A rendering module renders the extracted object information on a display interface based on the target file.

[0145] Specifically, the object types that can be extracted in the embodiments of the present application are exemplified by flowcharts, mind maps, bar charts, and pie charts, and further examples are given for the data processing flow of each module of a model.

[0146] 1. Image pre-classification module

[0147] The image pre-classification module is used to determine whether the currently input image is an image type that the system can support reconstruction (flowchart, mind map, bar chart, pie chart). The main technical difficulty faced by the end-side pre-classification module is how to accurately identify rejected samples in open scenes. In the embodiment of the present application, all images that are not flowcharts, mind maps, bar charts, and pie charts are considered rejected samples. Fig.14 As shown, the three images at the top are extractable samples, and the three images at the bottom are rejected samples.

[0148] In general, the embodiment of the present application provides a pre-classification module that can distinguish various types of images and reject images outside the specification. It can be used in common office scenarios and can also be migrated to other similar scenarios. The general operation is: for a chart image in format A opened by user software or presented on a display device, a screenshot or photo of the content of the chart part is taken, and the model then outputs the corresponding category, which is then assigned to the subsequent parsing algorithm to obtain editable structured data, greatly improving the accuracy and rationality of the assignment and the user experience.

[0149] Fig.15 The following is a schematic diagram of the training process of the image pre-classification module in the embodiment of the present application. The specific steps of the image pre-classification module are as follows:

[0150] S1: Obtain training sample set images.

[0151] Collect relevant image data from public channels or take photos with mobile phones, including image files of mind maps, bar charts, pie charts, and flow charts.

[0152] S2: Use the random data augmentation strategy (RandAugment) to obtain two different views.

[0153] A set of image augmentation operations are randomly selected from a predefined pool of operations, and each selected augmentation operation is randomly applied with different strengths. For each training sample, the corresponding augmentation operations are applied in a random order according to the selected augmentation strategy. We apply the random data augmentation strategy twice to each training sample, and obtain the tensor representations (images) of two different augmented image samples.

[0154] A tensor is a multidimensional array that can be represented as a vector, matrix, or higher-dimensional data structure. The tensor representation of an image can be viewed as a two-dimensional array (or matrix) where each element represents the pixel value at the corresponding position in the image. The tensor representation of an image usually includes a two-dimensional array (or matrix) of multiple channels. The tensor representation of an image can be understood as image data.

[0155] The main idea of ​​RandAugment is to randomly select transformations and adjust their sizes to increase the diversity of the data and thus improve the generalization ability of the model.

[0156] S3: Use the resizemix method to construct more mixed views and obtain the tensor representation of the mixed image samples (aug_images), that is, the mixed image samples.

[0157] This application adopts a resize mixing scheme to further expand the training data to simulate samples of unknown categories. Resizemix is ​​a data mixing method combined with salient information detection. Compared with other data mixing methods mixup, it has stronger interpretability and more reasonable label allocation strategy. At the same time, it can also more reasonably generate some mixed views as a supplement to the training sample set.

[0158] The core idea of ​​the Resizemix method is to directly scale an image and paste it on another image to mix the data. This method avoids the problems of misassignment of labels and loss of object information that may be caused by traditional cropping-based data mixing methods.

[0159] S4: performing a deep copy operation on the tensor representation of the enhanced image sample to obtain the tensor representation of the enhanced image sample (orig_images), that is, the enhanced image sample;

[0160] S5: Input the tensor representation of the enhanced image samples (orig_images) and the tensor representation of the mixed image samples (aug_images) into the feature extraction network (Feature Extractor) to obtain the auxiliary classification results (aux_output) output by the auxiliary classification head, the enhanced image feature embedding representation (Feature Embeddings Orig) and the mixed image feature embedding representation (Feature Embeddings Mix).

[0161] In some embodiments, the MobileNet V2 network is used as a feature extraction network, and the fully connected layer of its original classification head is replaced with a linear layer with a spectral norm as a feature projection layer, so that the gradient is more stable when the weight is updated. In addition, the embodiment of the present application also uses a small multilayer perceptron (Multilayer Perceptron, MLP) as an auxiliary classification head (which can also be understood as a classification network or classification layer) to generate an auxiliary classification result aux_output. The feature projection layer receives enhanced image samples and mixed image samples as input, and obtains enhanced image feature embedding representation (Feature Embeddings Orig) and mixed image feature embedding representation (FeatureEmbeddings Mix) through the forward calculation of the network.

[0162] The deep copy operation creates a new image object and copies all the data in the original image object to the new object. In this way, the original image object and the new image object have independent memory addresses and data. Therefore, modifications to the original image object will not affect the new image object obtained through the deep copy.

[0163] S6: Calculate the first loss function of the auxiliary classification result and perform gradient perturbation.

[0164] For the auxiliary classification result aux_output generated in S5, the cross entropy loss function (i.e., the first loss function) is calculated, and the gradient of the first loss function is used as the perturbation of the original feature. The specific formula is as follows:

[0165]

[0166] Where g represents the gradient of the cross entropy loss with respect to the feature embedding representation; ε is a hyperparameter that controls the size of the perturbation and needs to be set by yourself. After obtaining the gradient perturbation, the original feature embedding Feature Embeddings Orig is added to the perturbation to obtain the perturbed feature embedding Feature Embeddings Att.

[0167] S7: Calculation of the second loss function.

[0168] First calculate Figure 4 The calculation of the InfoNCE loss function in . The expression of the InfoNCE loss function is as follows:

[0169]

[0170] Among them, q represents the current sample, k + represents its only positive sample. Then calculate the main loss function, which is Figure 4 The SupCon loss function in is expressed as follows:

[0171]

[0172] Where y represents the label of the current data, and s(·) represents the similarity operation between labels.

[0173] S8: Fine-tune the classification task based on the pre-trained feature extraction network.

[0174] After the pre-training phase is completed, the weight parameters of the current model are fine-tuned to improve the model performance. When fine-tuning, the present application embodiment provides two methods that can be selected by the user. The first method is to freeze the feature extraction network parameters, add a fully connected layer with SpectralNorm as the classification head, and perform classification task training on the training sample set, such as Fig.16 As shown in Figure 1, the original auxiliary classification head is replaced with a new classification head. The second method is to directly load the original network weights without introducing a new structure, but only update the parameters of the auxiliary head.

[0175] 2. Image quality enhancement module

[0176] The image quality enhancement module first uses the image quality pre-analysis algorithm to determine the image enhancement method (which can be one or more) that needs to be used for the input image. Fig.17 From left to right priority, the selected enhancement methods are applied to the image in turn, and the enhanced output image is finally obtained.

[0177] 2.1 Image quality pre-analysis algorithm

[0178] The image quality pre-analysis algorithm is used to determine whether the current input image needs specific image enhancement processing (keystone correction, surface correction, de-moiré, de-blurring). The types of images collected in this section include: images with tilted shooting positions, images with curved surfaces, images with moiré, and images with blur; the image quality pre-analysis model is trained using the collected image samples.

[0179] 2.2 Keystone Correction

[0180] Keystone correction corrects the distorted image caused by tilted shooting to a normal shape, such as Fig.18 The complete process of keystone correction is as follows Fig.19 The complete steps of keystone correction are as follows:

[0181] S1: Obtain the original document image to be corrected

[0182] S2: Document edge probability prediction

[0183] S2.1: Input the document image to be corrected into the document edge detection model to obtain a document edge probability map, which is recorded as the first document edge. In the document edge probability map, the value of each pixel point represents the probability that the pixel point at the corresponding position in the image to be corrected belongs to the document edge, and the value range is [0.0, 1.0]. The larger the value, the higher the probability that the pixel point belongs to the document edge.

[0184] S2.2: Binarize the edge of the first document to obtain the edge of the second document. Use a threshold binarization method to binarize the edge of the first document to obtain the edge of the second document.

[0185] The specific process is to traverse the edge probability value of each pixel, and if the probability value is greater than the set threshold (for example, 0.85), then reassign the edge probability of the pixel to 1, otherwise it is assigned to 0. Equivalently, traverse the visualized edge probability map, if the value of the pixel is greater than, then assign the pixel to 255, otherwise it is assigned to 0, thereby obtaining a document edge binary map, which is recorded as the second document edge.

[0186] S2.3: Filter the edge of the second document, filter out the wrong edge, and obtain the edge of the third document. Since there may be some "edge blocks" in the background or inside the document area in the edge of the second document, they need to be filtered out. The filtering method is: use the connected domain algorithm to find all the connected edges in the edge of the second document, calculate the area of ​​each connected edge, and if the area is less than the set threshold, filter it out, and only retain the edge with an area greater than the threshold. The filtered result is recorded as the edge of the third document.

[0187] S2.4: Thinning the edge of the third document to obtain the fourth document edge. In the edge of the third document, the thickness of the edge is usually greater than one pixel. If the straight line detection is performed directly at this time, many invalid straight lines will be generated. Therefore, it is necessary to thin the edge. In the embodiment of the present application, we use the ZhangSuen thinning algorithm to extract the edge skeleton, which is recorded as the fourth document edge.

[0188] S3: Document boundary straight line fitting

[0189] S3.1: Using the progressive Hough transform algorithm, perform straight line detection on the edge of the fourth document to obtain a first straight line set, and convert the straight lines from the rectangular coordinate representation into the following parameterized form:

[0190]

[0191] Where (x1, y1) and (x2, y2) represent two different points on the line.

[0192] S3.2: Sort and cluster the straight lines in the first straight line set, and take the cluster center of each class to form a second straight line set, so as to realize straight line merging. Merge the straight lines in the first straight line set.

[0193] The lines in the first line set are merged as follows:

[0194] 1) Sort the lines in the first line set from small to large according to the ρ value;

[0195] 2) Define all the line clusters as C, the first type of line cluster as c1, and initialize them as c1 = {l1,}, C = {c1,}, where l1 is the first line in the sorted first line set;

[0196] 3) Select the next straight line li (2≤i≤n) and calculate c -1 Center c -1 represents the last type of straight line cluster in the current C. If l i and If the difference of ρ and θ is less than the set threshold, then l iJoin c -1 Otherwise, l i Based on this, a new class of straight line cluster c is created. j = {l i ,}, and c j Add to C;

[0197] 4) Repeat step 3) until all the lines are traversed and finally C = {c1, c2, ..., c m}, each c i There is at least one straight line in the equation, and if there are multiple straight lines, they are approximately the same straight line;

[0198] 5) After completing the above straight line clustering, calculate each type of straight line cluster c i Center All The set of This is the result of merging the straight lines, recorded as the second straight line set.

[0199] S3.3: Filter out the lines in the second straight line set that are almost horizontal or vertical and close to the image boundary, and then use the union-find algorithm to cluster the filtered straight line set. If the number of cluster categories is less than 4, cancel the previous filtering; the filtered straight line set is recorded as the third straight line set, and the number of straight lines in the third straight line set is less than or equal to the number in the second straight line set.

[0200] S4: Document vertex positioning

[0201] S4.1: Calculate the angles between two straight lines in the third straight line set. If the angles meet a threshold range (such as (50°, 90°]), calculate the intersection of the two straight lines. All such intersections constitute a candidate intersection set.

[0202] S4.2: Using the K-Means algorithm, the intersection points in the candidate intersection point set are sawn into four types of point clusters, denoted as C1, C2, C3, and C4.

[0203] S4.3: According to the set filtering rules, the four types of point clusters are filtered respectively. The centers of the four types of point clusters after filtering are the four vertices of the document.

[0204] The four types of point clusters are filtered separately, and the filtering process is as follows:

[0205] 1) According to whether the intersection point is in the image, point cluster C i The intersection points in (1≤i≤4) are divided into “point clusters within the graph” C i,inner and “cluster of points outside the graph” C i,outer , that is, C i =C i,inner +C i,outer ;

[0206] 2) If C i,outer is empty, that is, point cluster C i There is no external point in the graph, then for C i,inner Filtering is performed by: if C i,inner If there is only one intersection point in C, it is retained directly; i,inner If there are multiple points in C, only the points whose coordinates are on the edge of the third document are retained; i,inner If multiple points in are not on the edge of the third document, then only the point closest to the center of the image is retained.

[0207] 3) If C i,outer If there is an intersection in (this corresponds to the "document corner missing" problem caused by incomplete shooting), clear C i,inner , and for C i,outer Filtering is performed by: if C i,outer If there is only one intersection point in C, it is retained directly; i,outer If there are multiple intersection points in , only the intersection point closest to the center of the image is retained.

[0208] 4) After obtaining the four types of point clusters after filtering, calculate the center of each type of point cluster. The coordinates of the four centers are the coordinates of the four vertices of the document, denoted as P = {(x i ,y i ),1≤i≤4}, so far the document vertex positioning is completed.

[0209] S5: Perspective Transformation

[0210] The perspective transformation process is as follows:

[0211] 1) Determine whether the quadrilateral formed by the four document vertices P is reasonable. The judgment conditions are: whether the area ratio of the quadrilateral to the entire image is greater than 10%, and whether the four internal angles of the quadrilateral are within the range of [50°, 130°]. If the quadrilateral meets the above two conditions, continue to perform subsequent perspective transformation; otherwise, do not perform subsequent steps and directly return to the original image.

[0212] 2) According to the coordinates of the four document vertices, determine whether to perform padding operation. If padding operation is performed, calculate the padding width of the image to be corrected in the four directions of up, down, left, and right, and use the padding width to fill the original image, and update the coordinates of the four vertices accordingly;

[0213] 3) Calculate the four target vertices of the document in the target plane according to the coordinates of the four vertices;

[0214] 4) According to the four vertices and the four target vertices, the perspective transformation matrix M is calculated, and then the padded image to be corrected is perspective transformed according to M to complete the document correction.

[0215] 2.2 Image Deblurring

[0216] Image deblurring can make blurry images caused by long-distance shooting clearer, so that the subsequent chart information extraction algorithm can extract chart information more accurately. The detailed steps of image deblurring are as follows.

[0217] S1: Collecting clear image data

[0218] A dataset of 5,000 clear images is collected, called dataset 1.

[0219] S2: Add random noise

[0220] Add random noise to data set 1. Process each image in data set 1 in turn, and select one of the four strategies of Gaussian noise, salt and pepper noise, Poisson noise, and no noise with the probability distribution [0.3, 0.3, 0.3, 0.1]. The random selection range of the Gaussian noise degree parameter used in the embodiment of the present application is [0, 32], the random selection range of the Poisson noise degree parameter is [0, 32], and the random selection range of the salt and pepper noise degree parameter is [0, 0.04], and data set 2 is obtained.

[0221] S3: Adding motion blur

[0222] The problem of lens shaking when shooting with a real mobile phone is simulated by adding motion blur of different degrees and angles to Dataset 2. The proportion of images with motion blur added in the embodiment of the present application is 20%, and the ranges of the adjustable motion blur angle and degree parameters are [10, 20] and [1, 180] respectively. Dataset 3 is obtained.

[0223] S4: Adding tadpole-like deformation

[0224] Data set 3 is simulated by selecting a method based on average pooling and Gaussian blur to simulate the tadpole-shaped deformation effect of the two-dimensional code at different long distances in the real environment. The embodiment of the present application uses three methods: multi-degree tadpole-shaped degradation, multi-degree Gaussian blur degradation and non-degradation, with a ratio of 0.7:0.2:0.1. Tadpole-shaped degradation and Gaussian blur degradation use two degrees of degradation, lighter and heavier, respectively, with a ratio of 0.8:0.2. During the degradation process, for diversified two-dimensional code images, the degradation step calculates the size of the average pooling kernel according to the size of the positioning code of the two-dimensional code, and two-dimensional codes of different densities use pooling kernels of different sizes. In the embodiment of the present application, the pooling window size and the Gaussian blur kernel size are defined as 1 / 5 and 1 / 4 of the positioning code size in the lighter and heavier degrees, respectively, and the pooling step size is fixed to 1 / 4 of the pooling window size. Data set 4 is obtained.

[0225] S5: Training the deblurring model

[0226] Exemplarily, the deblurring model can be a deep learning model DeblurNet, and the deep learning model DeblurNet is trained using data set 1 and data set 4 to realize image deblurring. DeblurNet adopts an encoder-decoder structure, which mainly includes three parts: a contracting path (CP), an expanding path (EP) and a jump connection, and the structure is symmetrically distributed. The contracting path process is to use the maximum pooling layer (ML) + convolution to continuously perform four downsamplings. As the spatial resolution decreases and the number of channels increases, the context information in the picture can be fully captured; the expansion path process is to use the bilinear interpolation method (BI) + convolution to continuously perform four upsamplings. As the number of channels decreases and the spatial resolution increases, the information of interest in the picture can be accurately located. In addition, the jump connection process is implemented by splicing, which can retain more dimensions or position information, so that the deep network layer of image reconstruction can freely choose between shallow features and deep features, which is particularly suitable for restoring local and global information in blurred images.

[0227] 3. Information extraction module

[0228] like Fig. 20 As shown in the figure, for an input image, the information extraction module first uses the layout analysis algorithm to identify the chart area from the image and determines the chart type of each chart area. Then, the chart is sent to the corresponding information extraction module according to the chart type. The following focuses on the end-to-end solution for extracting information from four types of charts.

[0229] 3.1: Mind Map Information Extraction Algorithm

[0230] The mind map information extraction algorithm extracts structured data from mind maps (such as mind maps in JPG, PNG, and BMP formats) and converts these data into mind map files in a specific format (such as XMind and JSON formats).

[0231] The mind map extraction model is used to implement the flowchart information extraction algorithm. The training method of the mind map extraction model is as follows:

[0232] S1: Obtain mind map image samples.

[0233] Collect and annotate mind map images. Fig.21 As shown, the types of image annotations include: root node, intermediate node, leaf node, and cluster node.

[0234] The rectangular annotation box has the following attributes: label (label), point (detection box coordinates), where point represents the coordinates of the upper left corner and lower right corner of the rectangular box. In addition, each node is also labeled with the node ID and the ID of its parent node. The complete annotation of a node can be expressed as:

[0235] {"parent_ID":1,"label":"node","node_ID":2,"points":[[1482,364],[1694,462]]}.

[0236] S2: Build a model, use S1 to get image samples to train the model, and use the model to get the position and size of the detection box of each node (i.e., the first detection box), and the position of its parent node detection box (i.e., the second detection box).

[0237] The embodiment of the present application designs a new end-to-end mind map reconstruction model. This model is based on the idea of ​​target detection, combines node position recognition and node relationship recognition into one, and outputs the target node position and the center point coordinates of its parent node at one time in the target detection head. In terms of model design, the following model optimizations and modifications are mainly carried out:

[0238] 1) In the regression stage of the object detection bbox, two new regression targets are added to represent the center coordinates of the parent node.

[0239] 2) Use the relative position of the parent node for regression. Since the distance between the target relative value parent node and the current node is very close, the relative value is small, so using the relative position is easier to train.

[0240] 3) Generalized Focal Loss (GFL) is used for weighted calculation.

[0241] 4) Give higher loss weight to samples whose parent nodes cannot be accurately predicted.

[0242] Based on the optimized model, the mind map image input by the user is input into the mind map structure reconstruction model, and multiple detection boxes and parent node positions can be obtained at the same time to obtain the key structural information of the mind map.

[0243] S3: Perform optical character recognition on mind maps.

[0244] Through S2, the detection box coordinates {x1, y1, x2, y2} of each node in the mind map are obtained, and the text information is extracted.

[0245] When the number of coordinate frames is small (less than 50), the entire image is directly OCR-recognized, and then the OCR-recognized content is assembled based on the mind map node detection frame to obtain the text information of the corresponding node. When the number of coordinate frames is large (greater than or equal to 50), each node is separately recognized to ensure the accuracy of text recognition.

[0246] S4: Generate mind map structured data

[0247] The results obtained from S2 and S3 are assembled to obtain a standardized file format.

[0248] 3.2: Flowchart Information Extraction Algorithm

[0249] The core of the flowchart information extraction algorithm is to extract structured data from the flowchart (such as JPG, PNG, BMP format) and convert this data into a flowchart file in a specific format (such as XMind, JSON format).

[0250] The flowchart extraction model is used to implement the flowchart information extraction algorithm. The training method of the flowchart extraction model is as follows:

[0251] S1: Obtain a flow chart image.

[0252] Collect and annotate flowchart images. Fig. 22 As shown, the types of image annotations include: process, decision, sub-process, start / end, document, data, condition, flow direction, etc.

[0253] The rectangular annotation box has the following attributes: label label, coordinate point, where point represents the coordinates of the upper left corner and lower right corner of the rectangular box. In addition, each node is also labeled with the node ID and the ID of its parent node. The complete annotation of a node can be expressed as:

[0254] {"parent_ID":[1,13,14],"label":"'Process'","node_ID":2,point[(38,266),(198,385)]}.

[0255] S2: Build a model and use it to obtain the bounding box and relationship recognition results of each node.

[0256] The embodiment of the present application models the extraction of relationships in the flowchart as the prediction of relationship pairs, where a relationship pair is represented by (coordinates of the parent node, coordinates of the child node). The model adopts an encoder-decoder architecture, and the model includes two decoders, which are used for extracting node relationships in the flowchart and detecting node targets.

[0257] In terms of the encoder, ResNet is used as the backbone network to output feature maps of three different scales. The smallest feature map is input into the self-attention module for calculation to capture the features at the full image level, and then the features are fused with the feature maps of the other two scales to obtain the final output of the encoder.

[0258] In terms of the decoder, a deformable attention mechanism is used to reduce computational complexity while maintaining the ability of long-distance modeling. After obtaining multiple feature vectors output by the encoder, the model needs to select 300 vectors from them as query vectors for the decoder. The selection criteria are as follows: the multiple feature vectors output by the encoder are passed into the feature evaluation module, which can parse the information in the feature vectors to obtain the estimated relationship pairs and the corresponding confidence. Then, according to the size of the confidence, a selection is made from the feature vectors output by the encoder. Based on the selected feature vectors, the decoder is used to output the location and relationship information of the target text box.

[0259] S3: Perform optical character recognition on the flowchart.

[0260] Through S2, the detection box coordinates {x1, y1, x2, y2} of each node in the flowchart are obtained, and the text information is extracted.

[0261] When the number of coordinate frames is small (less than 50), the entire image is directly OCR-recognized, and then the OCR-recognized content is assembled based on the mind map node detection frame to obtain the text information of the corresponding node. When the number of coordinate frames is large (greater than or equal to 50), each node is separately recognized to ensure the accuracy of text recognition.

[0262] S4: Generate flowchart structured data

[0263] The results obtained from S2 and S3 are assembled to obtain a standardized file format.

[0264] 3.3: Histogram and pie chart information extraction algorithm

[0265] The bar chart and pie chart adopt the same technical solution, using an end-to-end neural network to extract the key elements and the relationship between the elements in the chart.

[0266] The extraction model is used to implement the bar chart information extraction algorithm and the pie chart information extraction algorithm. The training method of the extraction model is as follows:

[0267] S1: Collecting Datasets

[0268] Collect and annotate pie chart and bar chart images. For pie charts, the information that needs to be annotated includes: chart title, legend, sectors, and other explanatory text. For bar charts, the information that needs to be annotated includes: chart title, bar type, x-axis name, x-axis value, y-axis name, y-axis value.

[0269] S2: Object representation

[0270] like Fig.23 As shown in the figure, for a pie chart, the objects to be recognized include: chart title, legend, sector and other explanatory text. The title, legend and other text description are represented as a rectangular box, represented by four values ​​[x, y, w, h], where x, y represent the coordinates of the center point of the rectangular box; w represents the width of the rectangular box; h represents the height of the rectangular box. For sectors, an indirect representation is used. A sector chart is represented by the point ( Fig.23 The center detection box in the figure represents the intersection of each sector on the arc ( Fig.23 For these points, they are enlarged into a 5*5 rectangular box, and then represented using the rectangular box representation method. In this representation method, a sector can be represented by the midpoint of the sector and the intersection of two arcs.

[0271] For a bar chart, the objects that need to be recognized include: chart title, bar, x-axis name, x-axis value, y-axis name, y-axis value, and other text areas. All these objects are represented by a rectangular box [x, y, w, h].

[0272] S3: Relational representation method

[0273] In a pie chart, the relationships between objects include:

[0274] 1) The relationship between the legend and each sector

[0275] 2) The relationship between the additional text box and each sector

[0276] In a bar chart, the relationships between objects include:

[0277] 1) The relationship between each column and the legend;

[0278] 2) The relationship between each column and the text box.

[0279] For any relationship, the embodiment of the present application uses a four-tuple to represent it: [x1, y1, x2, y2], where x1, y1 represent the center point coordinates of a column or sector in the current relationship; x2, y2 represent the center point coordinates of another object in the current relationship.

[0280] In terms of representation, each relationship and each key element is represented by four values.

[0281] S4: Building the extraction model

[0282] The extraction model adopts an encoder-decoder architecture. It contains two decoders, one for node recognition (key element) and the other for node relationship recognition. The encoder is responsible for extracting image features and outputting relationship pairs and confidence levels. The decoder is responsible for filtering and optimizing the results output by the encoder to obtain the final full-graph node relationship. At the same time, the image features extracted by the encoder will pass through the node recognition decoder to output the node recognition results.

[0283] In terms of encoder, a CNN model, such as ResNet, is used as the backbone network, which can output feature maps of multiple scales. After the backbone network is calculated, the feature map with the smallest size is input into the self-attention module for calculation to capture the features at the full image level, and then the features are fused with the feature maps of the other two scales to obtain the final output of the encoder.

[0284] After obtaining multiple feature vectors output by the encoder, the model needs to select 300 vectors (the maximum number of nodes that can be detected and the number of relationships that can be extracted) as queries for the decoder. The selection criteria are as follows: Multiple feature vectors output by the encoder are passed to the feature evaluation module, which can parse the information in the feature vectors to obtain the estimated relationship pairs and the corresponding confidence. Then, according to the size of the confidence, the feature vectors output by the encoder are selected.

[0285] In the decoder, a deformable attention mechanism is used to reduce computational complexity while maintaining the ability of long-distance modeling. Deformable Transformer has excellent performance in computer vision tasks, especially in object detection tasks. By introducing a deformable attention mechanism, the model can effectively handle spatial changes in images and improve its ability to understand image content. In the standard Transformer model, the attention mechanism considers the interactions between all elements in the input sequence, which leads to huge computational costs when processing images because images are usually converted into a large number of pixel points or feature point sequences. The deformable Transformer allows the model to focus only on a part of the key area of ​​the image instead of the entire image through the deformable attention mechanism. This method not only reduces the amount of computation, but also improves the efficiency and effect of image processing. Deformable attention allows the model to dynamically select a small number of key sampling positions in each attention head for attention calculation, and these positions are dynamically determined based on the input data, which allows the model to focus more on important information in the image and improves the performance and generalization ability of the model.

[0286] S5: Using the model to extract graph information

[0287] In practical applications, for an input pie chart or bar chart, the model obtained by S4 can directly obtain the attribute information of all nodes in the graph and the association information between nodes.

[0288] The image information extraction method provided by the embodiment of the present application has the following beneficial effects:

[0289] Users can use this function to quickly and conveniently extract relevant information from image charts and export it to third-party editors (such as Word, Excel) for editing, which can greatly improve the user's editing efficiency of image charts on mobile devices.

[0290] In terms of chart information extraction, the embodiment of the present application designs and implements an end-to-end deep neural network model for four types of charts. The end-to-end solution can make the model more robust in real environments with complex backgrounds and noise. And in terms of actual model and product performance, the end-to-end solution is significantly better than other existing solutions. This function supports intelligent recognition and information extraction of flowcharts, mind maps, bar charts, and pie charts in documents, and can be shared or copied to third-party editing tools for secondary editing.

[0291] In terms of data flow, standardized file formats can form a complete link from data collection to use. From collection to transfer to sharing and use, the standardized file format opens up a complete data link on the device in a low-cost, easy-to-transmit and easy-to-understand form. The stored data has strong identifiability, and the entire device will provide content support to third-party applications and service layers in a complete data form, shortening the path for content to reach users and reducing the cost of using complex functions and scenarios.

[0292] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image information extraction device, such as Fig.24 As shown, the image information extraction device 240 includes:

[0293] An image enhancement unit 2401 is used to perform image enhancement on the first image to be processed to obtain an enhanced second image when the first image to be processed includes a target object;

[0294] The determining unit 2402 is used to determine one or more target objects and their object types included in the second image; and determine an extraction model of the target object according to the object type of the target object;

[0295] An extraction unit 2403 is used to extract image information of a target object using an extraction model of the target object, and obtain attribute information of multiple elements of the target object and association information between the elements;

[0296] The extraction unit 2403 is further configured to convert the attribute information of the multiple elements of the target object and the association information between the elements into a target file format to obtain a target file of the target object.

[0297] In some embodiments, the image information extraction device 240 further includes: a display unit, used to display the content of the first file on the user interface of the first application; and a processing unit, used to obtain the first image from the first file.

[0298] In some embodiments, the processing unit is used to obtain the first image from the currently displayed content of the first file in response to a first image selection operation; or, determine one or more extractable images from the first file; use the one or more extractable images as the first image to be processed; or, determine one or more extractable images from the first file; and obtain the first image from the one or more extractable images in response to a second image selection operation.

[0299] In some embodiments, the processing unit is further configured to determine a first object type to be processed; and based on the first object type, determine one or more extractable images from the first file.

[0300] In some embodiments, the processing unit is further configured to start the camera in response to the third image selection operation; and obtain the first image captured by the camera.

[0301] In some embodiments, the display unit is also used to display one or more extractable object types in response to an image information extraction operation; in response to a selection operation of a first object type, trigger the step of executing the image information extraction method, and the object type of the target object belongs to the first object type.

[0302] In some embodiments, the display unit is further configured to display an extraction result of the target object in the first display area based on the target file of the target object.

[0303] In some embodiments, the display unit is further used to display the target object in the second display area, and the first display area and the second display area are different display areas of the same user interface.

[0304] The processing unit is further used to, in response to an insertion operation of the extraction result of the target object, insert the target file of the target object into the first file to which the target object belongs to obtain a second file; the display unit is further used to display the target object and its extraction result on the user interface of the first application based on the second file; or,

[0305] The processing unit is also used to respond to the replacement operation of the extraction result of the target object, replace the target object in the first file with the target file of the target object to obtain a third file; the display unit is also used to display the extraction result of the target object in the user interface of the first application based on the third file.

[0306] In some embodiments, when the first image includes multiple target objects, the processing unit is further used to determine the selected first target object in response to a selection operation of the target object; and obtain an insertion operation or a replacement operation of the extraction result of the first target object.

[0307] In some embodiments, the processing unit is used to edit the extraction result of the target object to obtain a new target file.

[0308] In some embodiments, the processing unit is used to display a first editing menu in response to an editing operation on the target object, the first editing menu including one or more editing objects; in response to a selection operation on the first editing object, display a second editing menu for the first editing object, the second editing menu including one or more editable elements of the first editing object; based on the second editing menu, obtain editing information to edit the extraction result of the target object to obtain a new target file.

[0309] In some embodiments, when the target object is a chart, the one or more editing objects include at least one of the following: a data source, a chart type, and a chart element;

[0310] When the first editing object is a data source, the second editing menu includes a data table, an insert operation and a delete operation of the data table;

[0311] When the first editing object is a chart type, the second editing menu includes the original chart type and one or more replaceable chart types;

[0312] When the first editing object is a chart element, the second editing menu includes the original style and element setting items of the target object.

[0313] In some embodiments, when the target object is a pie chart or a bar chart, the element setting item includes at least one of the following: an element display control item, an element position setting item, and an element content setting item;

[0314] When the target object is a mind map, the element setting item includes at least one of the following: adding a branch item, deleting a branch item, inserting a branch element, deleting a branch element, an element style setting item, and an element content setting item;

[0315] When the target object is a flowchart, the element setting item includes at least one of the following: an element insertion item, an element deletion item, an element type conversion item, an element content setting item, and an element connection relationship setting item.

[0316] In some embodiments, the processing unit is used to convert the target file into the first file format in response to a first format conversion operation to obtain a fourth file; the fourth file is used to display the target object in the user interface of the second application, and the first file format is a file format supported by the second application; in response to the second format conversion operation, the second file or the third file is converted into the second file format to obtain a fifth file.

[0317] In some embodiments, the processing unit is further configured to, when the first image does not include the target object, determine that the first image is a non-extractable image and skip the steps of the image information extraction method.

[0318] In some embodiments, the processing unit is further configured to output first prompt information to prompt a user that the first image is an unextractable image when the first image does not include the target object.

[0319] In some embodiments, the image information extraction device 240 also includes: an image pre-classification unit, which is used to pre-classify the first image using an image pre-classification model to obtain a classification result, and the classification result is used to indicate whether the first image includes a target object. The image pre-classification model includes a feature extraction network and a classification head. The feature extraction network is configured to extract features from the first image to obtain a feature embedding representation, and the classification head is configured to perform image classification based on the feature embedding representation to determine whether the first image is a target object.

[0320] In some embodiments, the image pre-classification model is trained by the following method: obtaining an original training sample set, the original training sample set including original image samples and their category labels, the category labels being used to indicate whether the image samples contain a target object; performing data enhancement on the original image samples in the original training sample set to obtain an enhanced first training sample set; and using the first training sample set to train a first image pre-classification model to obtain a trained image pre-classification model.

[0321] In some embodiments, data enhancement includes: performing random data enhancement on image samples based on one or more random data enhancement strategies to obtain enhanced image samples; performing data mixing operations on the enhanced image samples to obtain mixed image samples; and constructing a first training sample set using the enhanced image samples and the mixed image samples.

[0322] In some embodiments, the training process of the image pre-classification model includes: the enhanced image samples are input into the first image pre-classification model to obtain the enhanced image feature embedding representation output by the feature extraction network of the first image pre-classification model and the classification result output by the classification head; the mixed image samples are input into the first image pre-classification model to obtain the mixed image feature embedding representation output by the feature extraction network of the first image pre-classification model; the first loss function is calculated using the classification result and the category label; based on the first loss function, the gradient perturbation is added to the enhanced image feature embedding representation to obtain the perturbed image feature embedding representation; based on the perturbed image feature embedding representation, the enhanced image feature embedding representation and the mixed image feature embedding representation, the second loss function is calculated; based on the second loss function, the weight parameters of the image pre-classification model are optimized to obtain the second image pre-classification model.

[0323] In some embodiments, the weight parameters of the feature extraction network of the second image preclassification model are frozen, and the weight parameters of the classification head of the second image preclassification model are optimized to obtain a trained image preclassification model; or, the classification head of the second image preclassification model is replaced with a new classification head; the weight parameters of the feature extraction network of the second image preclassification model are frozen, and the weight parameters of the new classification head of the second image preclassification model are optimized to obtain a trained image preclassification model.

[0324] In some embodiments, the image enhancement unit 2501 is used to perform image quality analysis on the first image to determine one or more image enhancement algorithms corresponding to the first image; based on the one or more image enhancement algorithms, perform image enhancement processing on the first image to obtain an enhanced second image.

[0325] In some embodiments, the one or more image enhancement algorithms include at least one of the following: keystone correction, surface correction, de-moiré, and deblurring.

[0326] In some embodiments, the one or more target objects and their object types are determined by a second device;

[0327] The image information extraction device 250 also includes: a communication unit, which is used to upload the second image to the second device so that the second device can perform image detection on the second image and determine the location information and object type of the target object in the second image; receive the location information and object type of the target object sent by the second device; and determine the target object in the second image based on the location information of the target object.

[0328] In some embodiments, the attribute information includes the element type of the current element and the position information of the first detection box, and the association information includes the position association information of the first detection box and the second detection box of the current element. The first detection box is the circumscribed rectangular box of the current element, and the second detection box is the circumscribed rectangular box of the associated element of the current element.

[0329] In some embodiments, the position information of the first detection frame includes first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame.

[0330] In some embodiments, the target object is a chart, and the target object includes at least one of the following: a mind map, a flow chart, a pie chart, and a bar chart;

[0331] The element types of the mind map include at least one of the following: root node, intermediate node, leaf node, thick node;

[0332] The element types of a flowchart include at least one of the following: process, decision, subprocess, start / end, document, data, condition, flow direction;

[0333] The element types of a pie chart include at least one of the following: icon title, legend, center point, sector point, text area;

[0334] The element types of the bar chart include at least one of the following: icon title, bar shape, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

[0335] In some embodiments, the attribute information includes at least element text information;

[0336] The extraction model is configured to perform the following steps: detect multiple elements of the target object, and obtain the first coordinates and sizes of the detection boxes of the multiple elements as attribute information of the multiple elements; determine the number of detection boxes of the target object; when the number of detection boxes is greater than or equal to a first threshold, extract text information based on the first coordinates and size of the third detection box to obtain text information of the third detection box, and the third detection box is part or all of the detection box in the target object; when the number of detection boxes is less than the first threshold, extract text information on the target object to obtain text information of the target object; determine the text information of the fourth detection box based on the text information of the target object and the first coordinates and size of the fourth detection box.

[0337] In some embodiments, the attribute information includes at least one of the following: element identification, element type, content, style, padding, and border.

[0338] In some embodiments, the extraction model is trained by the following method: obtaining a second training sample set, the second training sample includes a second image sample and its label information, the second image sample contains a target object, and the label information includes the position information of the first detection box of each element of the target object, the element type, the element identifier, and the associated element identifier; constructing an initial extraction model of the target object; using the second training sample set to train the initial extraction model of the target object to obtain a trained extraction model of the target object.

[0339] In some embodiments, when the target object is a mind map, the extraction model of the mind map includes: a backbone network, a feature pyramid network and a detection head; the mind map is input into the backbone network to output a first feature map, the first feature map is input into the feature pyramid network for multi-scale feature extraction to output a second feature map, the first feature map and the second feature map are input into the detection head to output the element type of each element of the mind map, the position information of the first detection box and the position information of the second detection box.

[0340] In some embodiments, when the target object is a flowchart, the extraction model of the flowchart includes an encoder and a decoder;

[0341] The encoder includes a backbone network and a self-attention module. The flowchart is input into the backbone network for multi-scale feature extraction to output a multi-scale feature map. The feature map of the first scale is input into the self-attention module to output a global feature map. The global feature map and the feature maps of other scales are fused to obtain multiple first feature vectors output by the encoder; the type information of each element of the flowchart, the position information of the first detection box, and the position information of the second detection box are output in the multiple first feature vector decoder.

[0342] In practical applications, the above-mentioned device can be a first device or a chip applied to the first device. In the present application, the device can realize the functions of multiple units by means of software, hardware, or a combination of software and hardware, so that the device can execute the image information extraction method provided in any of the above-mentioned embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the image information extraction method, and the present application will not elaborate on this one by one.

[0343] The first device may include, for example, a mobile phone, a tablet computer, a laptop computer, a PDA, a portable media player (PMP), a navigation device, a wearable device, a smart bracelet, a camera, etc.

[0344] Based on the hardware implementation of each unit in the above-mentioned image information extraction device, the embodiment of the present application also provides a first device, such as Fig.25 As shown, the first device 250 includes: a processor 2501 and a memory 2502 configured to store a computer program that can be run on the processor;

[0345] The processor 2501 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0346] Of course, in practical applications, Fig.25 As shown, the various components in the first device 250 are coupled together through a bus system 2503. It can be understood that the bus system 2503 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 2503 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 2503 in the figure.

[0347] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0348] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0349] An embodiment of the present application further provides a chip, the chip including a processor, and the processor can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0350] Optionally, the chip may further include a memory, wherein the processor may call and run a computer program from the memory to implement the method in the embodiment of the present application.

[0351] The memory may be a separate device independent of the processor, or may be integrated into the processor.

[0352] Optionally, the chip may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0353] Optionally, the chip may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0354] Optionally, the chip can be applied to the first device in the embodiment of the present application, and the chip can implement the corresponding processes implemented by the first device in each method of the embodiment of the present application. For the sake of brevity, it will not be repeated here.

[0355] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of the first device to complete the steps of the aforementioned method.

[0356] The embodiments of the present application also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the embodiments of the present application.

[0357] Optionally, the computer program product can be applied to the first device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the first device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0358] The embodiment of the present application also provides a computer program.

[0359] Optionally, the computer program can be applied to the first device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the first device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0360] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0361] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "have", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0362] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0363] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0364] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0365] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0366] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0367] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for extracting image information, applied to a first device, characterized in that: The method comprises: In the case where the first image to be processed includes the target object, performing image enhancement on the first image to obtain an enhanced second image; determining one or more target objects and their object types included in the second image; Determining an extraction model of the target object according to the object type of the target object; Extracting image information of the target object using the extraction model of the target object to obtain attribute information of multiple elements of the target object and association information between the elements; The attribute information of the multiple elements of the target object and the association information between the elements are converted into a target file format to obtain a target file of the target object.

2. The method according to claim 1, characterized in that The method further comprises: Displaying the content of the first file in the user interface of the first application; The first image is obtained from the first file.

3. The method according to claim 2, characterized in that The acquiring the first image from the first file comprises: In response to the first image selection operation, acquiring the first image from the currently displayed content of the first file; or, Determine one or more extractable images from the first file; use the one or more extractable images as the first image to be processed; or, One or more extractable images are determined from the first file; and in response to a second image selection operation, the first image is acquired from the one or more extractable images.

4. The method according to claim 3, characterized in that The method further comprises: determining a first object type to be processed; Based on the first object type, one or more extractable images are determined from the first file.

5. The method according to claim 1, characterized in that The method further comprises: In response to the third image selection operation, starting the camera; Acquire the first image captured by the camera.

6. The method according to claim 1, characterized in that The method further comprises: In response to the image information extraction operation, displaying one or more object types that can be extracted; In response to a selection operation of a first object type, the steps of the image information extraction method are performed, and the object type of the target object belongs to the first object type.

7. The method according to claim 1, characterized in that The method further comprises: Based on the target file of the target object, an extraction result of the target object is displayed in a first display area.

8. The method according to claim 7, characterized in that The method further comprises: The target object is displayed in a second display area, and the first display area and the second display area are different display areas of the same user interface.

9. The method according to claim 7, characterized in that: The method further comprises: In response to an insert operation of the extraction result of the target object, inserting the target file of the target object into the first file to which the target object belongs, to obtain a second file; Based on the second file, displaying the target object and the extraction result thereof on the user interface of the first application; or, The method further comprises: In response to a replacement operation of the extraction result of the target object, replacing the target object in the first file with the target file of the target object to obtain a third file; Based on the third file, the extraction result of the target object is displayed on the user interface of the first application.

10. The method according to claim 9, characterized in that When the first image includes a plurality of target objects, the method further includes: In response to a selection operation of a target object, determining a selected first target object; An insert operation or a replace operation is performed on the extraction result of the first target object.

11. The method according to claim 7 or 9, characterized in that: The method further comprises: The extraction result of the target object is edited to obtain a new target file.

12. The method according to claim 11, characterized in that The step of editing the extraction result of the target object to obtain a new target file includes: In response to the editing operation of the target object, displaying a first editing menu, wherein the first editing menu includes one or more editing objects; In response to a selection operation of a first editing object, displaying a second editing menu of the first editing object, the second editing menu including one or more editable elements of the first editing object; Based on the second editing menu, editing information is acquired to edit the extraction result of the target object to obtain a new target file.

13. The method according to claim 12, characterized in that When the target object is a chart, the one or more editing objects include at least one of the following: a data source, a chart type, and a chart element; When the first editing object is a data source, the second editing menu includes a data table, an insert operation and a delete operation of the data table; When the first editing object is a chart type, the second editing menu includes an original chart type and one or more replaceable chart types; When the first editing object is a chart element, the second editing menu includes the original style and element setting items of the target object.

14. The method according to claim 13, characterized in that When the target object is a pie chart or a bar chart, the element setting item includes at least one of the following: an element display control item, an element position setting item, and an element content setting item; When the target object is a mind map, the element setting item includes at least one of the following: a new branch item, a deleted branch item, a branch element insertion item, a branch element deletion item, an element style setting item, and an element content setting item; When the target object is a flowchart, the element setting item includes at least one of the following: an element insertion item, an element deletion item, an element type conversion item, an element content setting item, and an element connection relationship setting item.

15. The method according to claim 9, characterized in that The method further comprises: In response to the first format conversion operation, the target file is converted into the first file format to obtain a fourth file; the fourth file is used to display the target object in the user interface of the second application, and the first file format is a file format supported by the second application; In response to the second format conversion operation, the second file or the third file is converted into a second file format to obtain a fifth file.

16. The method according to claim 1, characterized in that The method further comprises: In the case that the first image does not include the target object, the first image is determined to be an unextractable image, and the steps of the image information extraction method are skipped.

17. The method according to claim 16, characterized in that The method further comprises: In the case that the first image does not include the target object, first prompt information is output to prompt the user that the first image is an unextractable image.

18. The method according to claim 1, characterized in that The method further comprises: A first image is pre-classified using an image pre-classification model to obtain a classification result, wherein the classification result is used to indicate whether the first image includes the target object. The image pre-classification model includes a feature extraction network and a classification head, wherein the feature extraction network is configured to extract features from the first image to obtain a feature embedding representation, and the classification head is configured to perform image classification based on the feature embedding representation to determine whether the first image is the target object.

19. The method according to claim 18, characterized in that The method further comprises: Acquire an original training sample set, wherein the original training sample set includes original image samples and their category labels, wherein the category labels are used to indicate whether the image samples contain a target object; Performing data enhancement on the original image samples in the original training sample set to obtain an enhanced first training sample set; The first training sample set is used to train the first image pre-classification model to obtain a trained image pre-classification model.

20. The method according to claim 19, characterized in that The data enhancement includes: Based on one or more random data enhancement strategies, performing random data enhancement on the image sample to obtain an enhanced image sample; Performing a data mixing operation on the enhanced image samples to obtain mixed image samples; The first training sample set is constructed using the enhanced image samples and the mixed image samples.

21. The method according to claim 20, characterized in that The step of training the first image pre-classification model using the first training sample set to obtain a trained image pre-classification model includes: The enhanced image sample is input into a first image pre-classification model to obtain an enhanced image feature embedding representation output by a feature extraction network of the first image pre-classification model and a classification result output by a classification head; The mixed image sample is input into a first image pre-classification model to obtain a mixed image feature embedding representation output by a feature extraction network of the first image pre-classification model; Calculating a first loss function using the classification result and the category label; Based on the first loss function, adding gradient perturbation to the enhanced image feature embedding representation to obtain a perturbed image feature embedding representation; Calculate a second loss function based on the perturbed image feature embedding representation, the enhanced image feature embedding representation and the mixed image feature embedding representation; The weight parameters of the image pre-classification model are optimized based on the second loss function to obtain a second image pre-classification model.

22. The method according to claim 21, characterized in that The method further comprises: Freeze the weight parameters of the feature extraction network of the second image pre-classification model, optimize the weight parameters of the classification head of the second image pre-classification model, and obtain a trained image pre-classification model; or, Replacing the classification head of the second image pre-classification model with a new classification head; The weight parameters of the feature extraction network of the second image pre-classification model are frozen, and the weight parameters of the new classification head of the second image pre-classification model are optimized to obtain a trained image pre-classification model.

23. The method according to claim 1, characterized in that The performing image enhancement on the first image to obtain an enhanced second image includes: Performing image quality analysis on the first image to determine one or more image enhancement algorithms corresponding to the first image; Based on the one or more image enhancement algorithms, image enhancement processing is performed on the first image to obtain the enhanced second image.

24. The method according to claim 23, characterized in that The one or more image enhancement algorithms include at least one of the following: keystone correction, surface correction, moiré removal, and deblurring.

25. The method according to claim 1, characterized in that The one or more target objects and their object types are determined by a second device; The method further includes: uploading the second image to the second device so that the second device performs image detection on the second image to determine the location information and object type of the one or more target objects in the second image; receiving location information and object types of the one or more target objects sent by the second device; The one or more target objects in the second image are determined based on the position information of the one or more target objects.

26. The method according to claim 1, characterized in that The attribute information includes the element type of the current element and the position information of the first detection box, and the association information includes the position association information of the first detection box and the second detection box of the current element. The first detection box is the circumscribed rectangular box of the current element, and the second detection box is the circumscribed rectangular box of the associated element of the current element.

27. The method according to claim 26, characterized in that The position information of the first detection frame includes first center coordinates and size; the position association information includes the first center coordinates of the first detection frame and the second center coordinates of the second detection frame.

28. The method according to claim 26, characterized in that The target object is a chart, and the target object includes at least one of the following: a mind map, a flow chart, a pie chart, and a bar chart; The element types of the mind map include at least one of the following: root node, intermediate node, leaf node, thick node; The element types of the flowchart include at least one of the following: process, decision, sub-process, start / end, document, data, condition, flow direction; The element types of the pie chart include at least one of the following: icon title, legend, center point, sector point, text area; The element types of the bar chart include at least one of the following: icon title, column shape, x-axis name, x-axis value, y-axis name, y-axis value, and text area.

29. The method according to claim 26, characterized in that The attribute information also includes text information; The extraction model is configured to perform the following steps: Detecting multiple elements of the target object to obtain first coordinates and sizes of detection boxes of the multiple elements as attribute information of the multiple elements; Determining the number of detection frames of the target object; When the number of the detection frames is greater than or equal to the first threshold, extracting text information based on the first coordinate and size of the third detection frame to obtain text information of the third detection frame, where the third detection frame is a part or all of the detection frame in the target object; When the number of the detection frames is less than the first threshold, extracting text information of the target object to obtain the text information of the target object; Based on the text information of the target object, and the first coordinates and the size of the fourth detection frame, the text information of the fourth detection frame is determined.

30. The method according to claim 1, characterized in that The method further comprises: Acquire a second training sample set, wherein the second training sample includes a second image sample and label information thereof, wherein the second image sample includes the target object, and the label information includes position information of a first detection box of each element of the target object, an element type, an element identifier, and an associated element identifier; Constructing an initial extraction model of the target object; The initial extraction model of the target object is trained using the second training sample set to obtain a trained extraction model of the target object.

31. The method according to claim 1 or 30, characterized in that When the target object is a mind map, the extraction model of the mind map includes: a backbone network, a feature pyramid network and a detection head; The mind map is input into the backbone network to output a first feature map, the first feature map is input into the feature pyramid network for multi-scale feature extraction to output a second feature map, the first feature map and the second feature map are input into the detection head to output the element type of each element of the mind map, the position information of the first detection box and the position information of the second detection box.

32. The method according to claim 1 or 30, characterized in that When the target object is a flowchart, the extraction model of the flowchart includes an encoder and a decoder; The encoder includes a backbone network and a self-attention module, the flowchart is input into the backbone network to perform multi-scale feature extraction and output a multi-scale feature map, the feature map of the first scale is input into the self-attention module to output a global feature map, and the global feature map and the feature maps of other scales are fused to obtain a plurality of first feature vectors output by the encoder; The multiple first feature vector decoders output type information of each element of the flowchart, position information of the first detection box, and position information of the second detection box.

33. An image information extraction device, characterized in that: The device comprises: an image enhancement unit, configured to, when a first image to be processed includes a target object, perform image enhancement on the first image to obtain an enhanced second image; a determination unit, configured to determine one or more target objects and their object types included in the second image; and determine an extraction model of the target object according to the object type of the target object; An extraction unit, configured to extract image information of the target object using an extraction model of the target object, and obtain attribute information of multiple elements of the target object and association information between the elements; The extraction unit is further used to convert the attribute information of the multiple elements of the target object and the association information between the elements into a target file format to obtain a target file of the target object.

34. A first device, characterized in that: The first device comprises: a processor and a memory configured to store a computer program that can be run on the processor, Wherein, the processor is configured to execute the steps of the method described in any one of claims 1 to 32 when running the computer program.

35. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 32 are implemented.

Images