Product manual element extraction methods, devices, equipment, media, and procedures.

Abstract

This disclosure provides a method for extracting elements from a product manual, applicable to the field of artificial intelligence technology. The method includes: acquiring a product manual image; performing text recognition on the product manual image based on an object detection model and a text recognition model; performing a first extraction on the text recognition result using an information extraction model to obtain product element information; performing a second extraction on the text recognition result using fuzzy extraction rules to obtain supplementary element information; and obtaining extracted elements from the manual based on the product element information and the supplementary element information, wherein the extracted elements include entity element fields and entity element information, and the entity element information corresponds to the entity element fields. This disclosure also provides a product manual element extraction device, equipment, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence technology or finance, specifically to a method, apparatus, device, medium, and program product for extracting elements from a product manual. Background Technology

[0002] In the financial sector, summarizing and extracting key elements from product manuals is a common business scenario. Currently, most solutions for product manual extraction rely on manual data entry, which is cumbersome, costly in terms of labor and time, and has low accuracy. How to quickly, intelligently, and accurately extract key elements from product manuals is a pressing issue that needs to be addressed. Summary of the Invention

[0003] In view of the above-mentioned problems existing in the prior art, embodiments of this disclosure provide a method, apparatus, device, medium, and program product for extracting product manual elements that improves the intelligence and accuracy of product manual element recognition and reduces labor costs. Furthermore, embodiments of this disclosure also provide a method, apparatus, device, medium, and program product for processing product manual elements.

[0004] According to a first aspect of this disclosure, a method for extracting elements from a product manual is provided, comprising: acquiring a product manual image; performing text recognition on the product manual image based on an object detection model and a text recognition model; performing a first extraction on the text recognition result using an information extraction model to obtain product element information; performing a second extraction on the text recognition result using fuzzy extraction rules to obtain supplementary element information; and obtaining extracted elements of the manual based on the product element information and the supplementary element information, wherein the extracted elements of the manual include entity element fields and entity element information, and the entity element information corresponds to the entity element fields.

[0005] According to embodiments of this disclosure, the text recognition of the product manual image based on the object detection model and the text recognition model includes: obtaining line text position information in the product manual image based on the object detection model; cropping the product manual image based on the line text position information to obtain a product manual block image; and inputting the product manual block image into the text recognition model to obtain product manual text information.

[0006] According to an embodiment of this disclosure, extracting product element information by using an information extraction model to extract the text recognition result includes: inputting the text recognition result into a pre-trained information extraction model; and using the pre-trained information extraction model to extract information from the text recognition result and output the product element information, wherein the pre-trained information extraction model is trained based on a product manual sample of less than a preset number combined with a general extraction framework.

[0007] According to an embodiment of this disclosure, the method for pre-training an information extraction model includes: obtaining a product manual sample; annotating the product manual sample with element information; and inputting the product manual sample with element information annotation into the general extraction framework for training until a preset cutoff condition is reached, thereby obtaining the information extraction model, wherein the general extraction framework includes a knowledge-enhanced pre-trained model.

[0008] According to embodiments of this disclosure, the secondary extraction of the text recognition result using fuzzy extraction rules to obtain supplementary element information includes: matching the product element information with a preset element dictionary to determine whether there are any unrecognized entity elements; and performing secondary recognition of the unrecognized entity elements based on regular expression matching rules to obtain the supplementary element information.

[0009] According to embodiments of this disclosure, the step of matching the product element information with a preset element dictionary to determine whether there are any unidentified entity elements includes: obtaining a product element information extraction dictionary, the product element information extraction dictionary including entity element information obtained through a single extraction; storing the entity element information obtained through a single extraction in the preset element dictionary, wherein the preset element dictionary includes entity element fields, and after storage, there is a mapping relationship between the entity element information and the entity element fields; and when there is an entity element field in the preset element dictionary that lacks entity element information with a mapping relationship, determining that the entity element field is an unidentified entity element.

[0010] According to embodiments of this disclosure, the secondary identification of the unidentified entity elements based on regular expression matching rules to obtain the supplementary element information includes: obtaining a keyword list of unidentified entity elements, wherein the keyword list includes keywords associated with the unidentified entity elements, and the keywords associated with the unidentified entity elements include the unidentified entity elements and their synonyms and / or near-synonyms; sorting the keywords associated with the unidentified entity elements from highest to lowest frequency; and using regular expressions to locate the keywords associated with the unidentified entity elements according to the frequency sorting result, and obtaining entity element information that has a mapping relationship with the keywords associated with the unidentified entity elements as the supplementary element information.

[0011] According to an embodiment of this disclosure, the step of using regular expressions to locate the associated keywords of the unidentified entity element according to the frequency sorting result includes: using regular expressions to locate the associated keyword of the unidentified entity element that ranks first in frequency sorting.

[0012] According to an embodiment of this disclosure, after obtaining entity element information that has a mapping relationship with the keywords associated with the unidentified entity element, the method further includes: verifying the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element using auxiliary rules to obtain the supplementary element information, including: determining whether the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to a preset element representation form based on a preset element form specification; and determining that the entity element information is supplementary element information when the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to the preset element representation form.

[0013] The second aspect of this disclosure provides a method for processing product manual elements, including: extracting the product manual elements; encapsulating the extracted manual elements into an interface message; and when the interface is called, feeding back the extracted manual elements and element extraction association information to the front-end page, wherein the product manual elements are extracted based on the product manual element extraction method of the first aspect of this disclosure.

[0014] A third aspect of this disclosure provides a product manual element extraction device, comprising: an acquisition module configured to acquire a product manual image; a recognition module configured to perform text recognition on the product manual image based on a target detection model and a text recognition model; a first extraction module configured to perform a first extraction on the text recognition result using an information extraction model to acquire product element information; a second extraction module configured to perform a second extraction on the text recognition result using fuzzy extraction rules to acquire supplementary element information; and an element extraction module configured to acquire manual extraction elements based on the product element information and the supplementary element information, wherein the manual extraction elements include entity element fields and entity element information, and the entity element information corresponds to the entity element fields.

[0015] According to embodiments of this disclosure, the recognition module includes a localization submodule, a cropping submodule, and an extraction submodule. The localization submodule is configured to obtain the position information of line text in the product manual image based on a target detection model. The cropping submodule is configured to crop the product manual image based on the line text position information to obtain a product manual block image. The extraction submodule is configured to input the product manual block image into the text recognition model to obtain the product manual text information.

[0016] According to embodiments of this disclosure, the first extraction module includes an input submodule and a first acquisition submodule. The input submodule is configured to input the text recognition result into a pre-trained information extraction model. The pre-trained information extraction model is trained based on a smaller than preset number of product manual samples combined with a general extraction framework. The first acquisition submodule is configured to use the pre-trained information extraction model to extract information from the text recognition result and output the product element information.

[0017] According to embodiments of this disclosure, the second extraction module includes a comparison submodule and a first acquisition submodule. The comparison submodule is configured to match the product element information with a preset element dictionary to determine whether any unidentified entity elements exist. The first acquisition submodule is configured to perform secondary identification of the unidentified entity elements based on regular expression matching rules to acquire the supplementary element information.

[0018] According to embodiments of this disclosure, the comparison submodule includes a first acquisition unit, a storage unit, and a judgment unit. The first acquisition unit is configured to acquire a product element information extraction dictionary, which includes entity element information obtained through a single extraction. The storage unit is configured to store the entity element information obtained through a single extraction in a preset element dictionary, wherein the preset element dictionary includes entity element fields, and after storage, a mapping relationship exists between the entity element information and the entity element fields. The judgment unit is configured to determine that an entity element field is an unidentified entity element when an entity element field in the preset element dictionary lacks entity element information with a corresponding mapping relationship.

[0019] According to embodiments of this disclosure, the first acquisition submodule includes a second acquisition unit, a sorting unit, and a third acquisition unit. The second acquisition unit is configured to acquire a keyword list of unidentified entity elements, wherein the keyword list includes keywords associated with the unidentified entity elements, and the associated keywords include the unidentified entity elements and their synonyms and / or near-synonyms. The sorting unit is configured to sort the associated keywords of the unidentified entity elements from highest to lowest frequency. The third acquisition unit is configured to use regular expressions to locate the associated keywords of the unidentified entity elements according to the frequency sorting result, and acquire entity element information that has a mapping relationship with the associated keywords of the unidentified entity elements. The acquired entity element information that has a mapping relationship with the associated keywords of the unidentified entity elements can be used as supplementary element information.

[0020] According to embodiments of this disclosure, the first acquisition submodule may further include a verification unit. The verification unit is configured to use auxiliary rules to verify the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element, thereby acquiring the supplementary element information. This includes: determining whether the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to a preset element representation format based on a preset element form specification; and determining that the entity element information is supplementary element information when the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to the preset element representation format.

[0021] A third aspect of this disclosure provides an information extraction model training apparatus, including a sample collection module, an annotation module, and a training module. The sample collection module is configured to acquire product manual samples. The annotation module is configured to annotate the product manual samples with element information. The training module is configured to input the annotated product manual samples into the general extraction framework for training until a preset cutoff condition is reached, at which point the information extraction model is obtained. The general extraction framework includes a knowledge-enhanced pre-trained model.

[0022] The fourth aspect of this disclosure provides a product manual element processing apparatus, characterized in that it includes: an acquisition module configured to acquire a product manual image; a recognition module configured to perform text recognition on the product manual image based on a target detection model and a text recognition model; a first extraction module configured to perform a first extraction on the text recognition result using an information extraction model to acquire product element information; a second extraction module configured to perform a second extraction on the text recognition result using fuzzy extraction rules to acquire supplementary element information; and an element extraction module configured to acquire manual extraction elements based on the product element information and the supplementary element information, wherein the manual extraction elements include entity element fields and entity element information, and the entity element information corresponds to the entity element fields; a result encapsulation module configured to encapsulate the manual extraction elements into an interface message; and a feedback module configured to feed back the manual extraction elements to a front-end page when the interface is called.

[0023] The fifth aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described product specification element extraction method.

[0024] A sixth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described method for extracting product specification elements.

[0025] The seventh aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for extracting product specification elements.

[0026] The method provided in the embodiments of this disclosure performs text recognition on the product manual image based on a target detection model and a text recognition model, and performs element information extraction twice using an information extraction model and fuzzy extraction rules. This method can accurately, quickly, intelligently and easily extract the entity information of the product manual, greatly reducing the labor costs for business personnel and the threshold for use. Attached Figure Description

[0027] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 The illustration schematically depicts application scenarios of a product specification element extraction method, apparatus, device, medium, and program product according to embodiments of this disclosure;

[0029] Figure 2 A flowchart illustrating a method for extracting product specification elements according to an embodiment of this disclosure is shown.

[0030] Figure 3 The flowchart illustrates a method for performing text recognition on a product instruction manual image using an object detection model and a text recognition model according to some embodiments of the present disclosure.

[0031] Figure 4 The flowchart illustrates a method for performing text recognition on a product instruction manual image using an object detection model and a text recognition model, according to an example of this disclosure.

[0032] Figure 5 The flowchart illustrates a method for extracting product element information by using an information extraction model to extract text recognition results according to an embodiment of the present disclosure.

[0033] Figure 6 A flowchart illustrating a method for obtaining an information extraction model through pre-training according to an embodiment of the present disclosure is shown.

[0034] Figure 7 The diagram illustrates a schematic representation of element entity annotations according to an embodiment of the present disclosure.

[0035] Figure 8 The illustration shows a schematic diagram of the model return results according to an embodiment of the present disclosure.

[0036] Figure 9 The schematic illustration shows the training and prediction architecture of the PaddlePaddle general information extraction model, which is an example of the present disclosure.

[0037] Figure 10 The flowchart illustrates an example of a method for extracting product element information from a product manual using a PaddlePaddle general information extraction model, according to an example of this disclosure.

[0038] Figure 11 The flowchart illustrates a method for obtaining supplementary element information by performing secondary extraction of the text recognition result using fuzzy extraction rules according to an embodiment of the present disclosure.

[0039] Figure 12 The flowchart illustrates a method for matching product element information with a preset element dictionary to determine whether there are any unrecognized entity elements according to an embodiment of the present disclosure.

[0040] Figure 13 The flowchart illustrates a method for obtaining supplementary element information by performing secondary identification of the unidentified entity elements based on regular expression matching rules according to an embodiment of the present disclosure.

[0041] Figure 14 The flowchart illustrates a method for obtaining supplementary element information by verifying entity element information that has a mapping relationship with the keywords associated with the unidentified entity element using auxiliary rules, according to other embodiments of this disclosure.

[0042] Figure 15 A flowchart illustrating a method for processing product specification elements according to an embodiment of this disclosure is shown schematically.

[0043] Figure 16 The flowchart illustrating a method for extracting product specification elements according to a specific example of this disclosure is shown schematically.

[0044] Figure 17 A schematic block diagram of a product specification element extraction device according to an embodiment of the present disclosure is shown.

[0045] Figure 18 A schematic block diagram of an identification module according to an embodiment of the present disclosure is shown.

[0046] Figure 19 A schematic block diagram of a first extraction module according to an embodiment of the present disclosure is shown.

[0047] Figure 20 A schematic block diagram of a second extraction module according to an embodiment of the present disclosure is shown.

[0048] Figure 21 A schematic diagram illustrating the structure of the comparison submodule according to an embodiment of the present disclosure is shown.

[0049] Figure 22 A schematic diagram illustrating the structure of a first acquisition submodule according to an embodiment of the present disclosure is shown.

[0050] Figure 23 A schematic diagram illustrating the structure of a first acquisition submodule according to other embodiments of the present disclosure is shown.

[0051] Figure 24 A schematic block diagram of an information extraction model training apparatus according to an embodiment of the present disclosure is shown.

[0052] Figure 25 A schematic block diagram of a product specification element processing apparatus according to an embodiment of the present disclosure is shown.

[0053] Figure 26 A block diagram schematically illustrates an electronic device suitable for implementing a method for extracting product specification elements and / or a method for processing product specification elements according to embodiments of the present disclosure. Detailed Implementation

[0054] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0055] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0056] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0057] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0058] In the financial sector, summarizing and extracting key elements from product manuals is a common business scenario. Currently, most solutions for product manual extraction rely on manual data entry. Specifically, element extraction in product manuals primarily depends on business personnel manually identifying and extracting elements, including reading the product manual content, locating the desired elements, manually filling in relevant information, and uploading it to the business system. On the one hand, the entire process of manual element extraction is cumbersome and increases labor costs; on the other hand, the accuracy of manual element identification is relatively low. When extracting large amounts of data, this increases the workload of business personnel, reduces efficiency, and accuracy decreases with low data volumes. How to quickly, intelligently, and accurately extract elements from product manuals is a problem that urgently needs to be solved.

[0059] To address the aforementioned problems in the prior art, embodiments of this disclosure provide a method for extracting elements from a product manual, comprising: acquiring a product manual image; performing text recognition on the product manual image based on an object detection model and a text recognition model; performing a first extraction on the text recognition result using an information extraction model to obtain product element information; performing a second extraction on the text recognition result using fuzzy extraction rules to obtain supplementary element information; and obtaining extracted elements of the manual based on the product element information and the supplementary element information, wherein the extracted elements of the manual include entity element fields and entity element information, and the entity element information corresponds to the entity element fields.

[0060] The method provided in the embodiments of this disclosure performs text recognition on the product manual image based on a target detection model and a text recognition model, and performs two-stage element information extraction using an information extraction model and fuzzy extraction rules. This method can automate the detection, recognition and extraction of product manual elements, while occupying relatively few system resources, having a high running speed and a high element extraction accuracy, thus reducing the labor costs for business personnel and the threshold for use.

[0061] It should be noted that the product manual element extraction method, apparatus, device, medium, and program products provided in this disclosure can be used in information extraction related to artificial intelligence technology, and can also be used in various fields other than artificial intelligence technology, such as the financial field. The application fields of the product manual element extraction method, apparatus, device, medium, and program products provided in this disclosure are not limited.

[0062] The above-described operations for achieving at least one objective of this disclosure will be described below in conjunction with the accompanying drawings and their descriptions.

[0063] Figure 1 The illustration shows an application scenario of a product specification element extraction method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0064] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0065] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. For example, users can use terminal devices 101, 102, and 103 to send product manuals to server 105 via network 104. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0066] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0067] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. For example, server 105 can send extracted instruction manual elements to terminal devices 101, 102, and 103 via network 104.

[0068] It should be noted that the product manual element extraction method provided in this embodiment can generally be executed by server 105. Correspondingly, the product manual element extraction device provided in this embodiment can generally be located in server 105. The product manual element extraction method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the product manual element extraction device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0069] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0070] The following will be based on Figure 1 The described scene, through Figures 2 to 16 The method for extracting product specification elements according to the disclosed embodiments is described in detail.

[0071] Figure 2 A flowchart illustrating a method for extracting product specification elements according to an embodiment of this disclosure is shown.

[0072] like Figure 2 As shown, the product manual element extraction method of this embodiment includes operations S210 to S250. This product manual element extraction method can be executed by a processor or by any electronic device including a processor.

[0073] Using S210, obtain images of the product manual.

[0074] According to embodiments of this disclosure, besides images, the sample formats of product manuals mainly include Word and PDF document formats. When using the method of embodiments of this disclosure to extract elements from product manuals, the Word or PDF document files need to be converted into image formats. For Word format product manuals, the Word file can be processed first, removing all annotations, reviews, bolding, etc., from the content before saving. Before operation S210, the Word and PDF files can be converted into images. The resolution of the converted images can be set based on expert experience combined with the requirements of the object detection model, text recognition model, and information extraction model; for example, it can be 700×900 (pixels) or higher. Furthermore, based on business needs and expert experience, the element names and corresponding keywords of the product manual to be extracted can be clearly defined. Taking banking business as an example, Table 1 exemplarily shows the element names and corresponding example element information of the product manual.

[0075] Table 1

[0076]

[0077] In operation S220, text recognition is performed on the product manual image based on the object detection model and the text recognition model.

[0078] According to embodiments of this disclosure, after obtaining a product manual in image format, text recognition can be performed on the image of the product manual based on an object detection model and a text recognition model to further process and obtain element association information. Specifically, the object detection model is used to locate target text information in the image of the product manual. The text recognition model is used to recognize the located target text information. It should be understood that both the object detection model and the text recognition model can be pre-trained.

[0079] When operating S230, the text recognition results are extracted using an information extraction model to obtain product element information.

[0080] In operation S240, the text recognition result is extracted a second time using fuzzy extraction rules to obtain supplementary element information.

[0081] In operation S250, the instruction manual extracts elements based on the product element information and the supplementary element information.

[0082] According to embodiments of this disclosure, a dual extraction method is used to achieve highly accurate automated extraction of product instruction manual elements. First, product element information is extracted based on an information extraction model. To further improve the accuracy and comprehensiveness of recognition, considering the diversity of product element presentation formats, fuzzy extraction rules are used for secondary extraction of the text recognition results. The fuzzy extraction rules, as a supplement to the primary extraction, can be used to identify synonyms and / or near-synonyms associated with entity element names, as well as the element entity information corresponding to these near-synonyms and near-synonyms. Furthermore, summarizing the product element information and supplementary element information yields the extracted instruction manual elements. The extracted instruction manual elements include entity element fields and entity element information, wherein the entity element fields can be preset based on business requirements and / or expert experience. It should be understood that the entity element information corresponds to the entity element fields. For example, the entity element field can be the element name in Table 1, and correspondingly, the entity element information can be the element information of the example corresponding to the element name in Table 1.

[0083] Figure 3 The flowchart illustrates a method for performing text recognition on a product instruction manual image using an object detection model and a text recognition model according to some embodiments of the present disclosure.

[0084] like Figure 3 As shown, the product specification element extraction method of these embodiments includes operations S3 10 to S3 30.

[0085] In operation S310, the position information of the line text in the product manual image is obtained based on the target detection model.

[0086] In operation S320, the product manual image is cropped based on the line text position information to obtain a product manual block image. In the embodiments of this disclosure, the object detection model can locate the position of each line of text in the product manual, and after locating it, it can be converted into an image block containing the line text using image cropping.

[0087] In operation S330, the product manual block image is input into the text recognition model to obtain the text information of the product manual. The text recognition model used in the embodiments of this disclosure is based on image recognition technology for text recognition of the product manual.

[0088] In some specific embodiments, the target detection model can be a YOLO series model. Preferably, it can be a YOLOv3 model to balance detection accuracy and computational efficiency.

[0089] In some specific embodiments, the text recognition model includes a convolutional recurrent neural network (CNN) model. This model can transform text recognition into a time-dependent sequence learning problem based on image sequence recognition. The CNN model consists of convolutional layers, recurrent layers, and transcription layers. The convolutional layers use a deep CNN to extract features from the input image, obtaining a feature map. The recurrent layers use a bidirectional RNN (BLSTM) network to predict the feature sequence, learning each feature vector in the sequence and outputting a predicted label (true value) distribution. The transcription layer uses CTC loss to transform the series of label distributions obtained from the recurrent layers into a final label sequence, exhibiting characteristics of fast recognition speed and high performance.

[0090] Figure 4 The flowchart illustrates a method for performing text recognition on a product instruction manual image using an object detection model and a text recognition model, according to an example of this disclosure.

[0091] like Figure 4 As shown in the example, a trained Yolov3 and CRNN model is used to perform text recognition on an image of a product manual. The model output is the text characters for each line of text in the image, facilitating subsequent semantic element extraction. Yolov3 is an object detection model that can locate the position of detected objects; the CRNN model is a deep learning model for text recognition, capable of recognizing text sequences of arbitrary length.

[0092] In the model pre-training phase, various product manual image samples can be prepared first, and the sample set can be expanded using data processing operations such as scaling and flipping. Text coordinates are marked on the images row by row, and the positions of these marks are recorded and labeled as 1 for training the Yolov3 model. The model training parameters can be set as follows: 64 samples per training iteration, initial learning rate 0.001, weight decay 0.0005, and 2000 iterations. Then, based on the coordinates of the marked text, line text images are generated by cropping the sample data, serving as samples for training the CRNN model. The CRNN model training parameters are as follows: 32 samples per training batch, sample split ratio 0.75, 1000 iterations, and learning rate 0.0001. After training the object detection model and the text recognition model, the product manual image to be predicted can be input into the object detection model to obtain the predicted text position coordinates. After data cropping, the text is further input into the text recognition model for text recognition processing, and finally, the text characters are output.

[0093] Figure 5 The flowchart illustrates a method for extracting product element information by using an information extraction model to extract text recognition results according to an embodiment of the present disclosure.

[0094] like Figure 5 As shown, the method for obtaining product element information in this embodiment includes operations S510 to S520.

[0095] In operation S510, the text recognition result is input into the pre-trained information extraction model.

[0096] In operation S520, the information extraction model obtained from the pre-training is used to extract information from the text recognition results and output the product element information.

[0097] In the embodiments of this disclosure, the pre-trained information extraction model is trained based on a product manual sample size less than a preset number, combined with a general extraction framework. The general extraction framework can be trained on an existing training set. Furthermore, to obtain a model suitable for extracting product manual information according to this disclosure, the general extraction framework can be trained on a small sample size to fine-tune the model parameters. For example, a preset sample size threshold can be set, and a smaller than preset number of product manual samples can be used for fine-tuning during actual training. Preferably, the preset small sample size can be 30, 50, 80, etc.

[0098] Figure 6 A flowchart illustrating a method for obtaining an information extraction model through pre-training according to an embodiment of the present disclosure is shown.

[0099] like Figure 6As shown, the method for obtaining product element information in this embodiment includes operations S610 to S630.

[0100] When operating S610, obtain a sample product manual.

[0101] In operation S620, the product instruction manual sample is annotated with element information.

[0102] In operation S630, the product manual sample after element information annotation is input into the general extraction framework for training until the preset cutoff condition is reached, and then the information extraction model is obtained.

[0103] In embodiments of this disclosure, the general extraction framework includes a knowledge-enhanced pre-trained model. Preferably, the general extraction framework can be a pre-trained model that incorporates large-scale knowledge. Typical general extraction frameworks may include the ERINIE series of models, such as the ERINIE3.0 pre-trained model.

[0104] Specifically, when training the information extraction model of the embodiments disclosed herein, the first step is to annotate the entire product manual content obtained using object detection combined with text recognition. For example, the same element in the product manual is usually a fixed value, but it may exist in different places throughout the document and be presented in different textual statements. For example, the text description might state: "Investment type is fixed income" or "Product return type is fixed income." The entity element name in both cases is "rate type," and the content is "fixed income," but they are presented in different ways. Therefore, to enable the model to find the entity information to be extracted, its samples are annotated, such as... Figure 7 As shown. In Figure 7 In the schematic diagram illustrating the element entity annotation according to an embodiment of the present disclosure, each color indicates the extracted entity element information, and the underline corresponds to the entity element name.

[0105] After fully annotating a small number of samples, the data is fed into the model for iterative training. The training continues until a preset cutoff condition is met, at which point the information extraction model is obtained. This preset cutoff condition can be either a preset number of training iterations or a preset model training accuracy. Once the model is trained, it can extract information from the product manual's key elements. Input the text to be predicted (the text can be a sentence or the full text of the product manual), for example: "XX Fund was established on XX / XX / XX and will mature on XX / XX / XX. The investment type of this product is generally a fixed-income investment, with a management fee rate of XX%, a sales service fee rate of XXX%, and no outsourcing fees. The custodian of this fund is XX Bank." The model will extract the relevant elements from the input text. The model will extract all elements that need to be extracted from the text. The model returns the following results: Figure 8 As shown. In Figure 8 In the schematic diagram illustrating the model return results according to an embodiment of the present disclosure, text represents the element content, start\end represents the starting position, and probability represents the confidence level.

[0106] In a specific example disclosed herein, where the sample size of product manuals is small and the recognition speed requirement is high, the information extraction model can use the PaddlePaddle General Information Extraction Model (Paddle UIE). Typically, in the scenario of financial product manuals, the total number of samples accumulated from the start of a project is usually small, and there are many element aliases. Specifically, the total number of samples accumulated may not exceed 150, and the element names differ between different banks and product manuals. Traditional entity extraction algorithms struggle to achieve good results with a small sample size and a large number of aliased elements. In this case, a general information extraction model can be used. Since it has undergone pre-training on a large-scale general training sample, in the scenario of this disclosure example, by providing a small number of scenario samples, the learned "general knowledge" can be generalized into "specific knowledge," achieving good results with a small sample size of financial product manuals. Furthermore, traditional entity extraction models have complex structures and numerous program files, consuming significant memory and generally requiring substantial GPU resources for online operation. When GPU resources are limited, the lightweight PaddlePaddle general-purpose information extraction model, which supports both CPU and GPU resources, can effectively address this resource shortage. The amount of GPU resources and the complexity of the model significantly influence runtime. When the product manual is a financial product manual, it needs to extract elements from 5-10 or more pages within 30 seconds, requiring a fast processing speed. In this case, the PaddlePaddle general-purpose information extraction model can achieve high accuracy while saving computation time. In addition, the PaddlePaddle general-purpose information extraction model also boasts advantages such as low development costs, simple data annotation, low iteration costs, and high speed. It has significant advantages when applied to scenarios with short development cycles and high requirements for model iteration speed.

[0107] The following combination Figures 9-10 An illustrative diagram illustrating the method of training and extracting general information using the PaddlePaddle model.

[0108] Figure 9 The schematic illustration shows the training and prediction architecture of the PaddlePaddle general information extraction model, which is an example of the present disclosure.

[0109] like Figure 9As shown, the PaddlePaddle general information extraction model is used to extract product element information from the outputs of the object detection model YOLOv3 and the text recognition model CRNN. ​​The PaddlePaddle general information extraction model is a general information extraction task framework that supports extraction tasks such as entities, relationships, events, and sentiment, and features low resource consumption and high performance. For the general information extraction model, product manual text samples are added for model training and fine-tuning. Using the previously determined entity element names, element annotations are extracted from the product manual text output after text recognition, and these sample examples are input into the general information extraction model for training. In the example disclosed in this publication, the preferred training parameters are as follows: learning rate 0.00001, maximum sentence length 512, 8 training samples per session, and 10 iterations.

[0110] Based on the preceding description, Figure 10 The flowchart schematically illustrates a method for extracting product element information from a product manual using a PaddlePaddle generic information extraction model, according to an example of this disclosure. Figure 10 In this process, the model can first be trained using a sample product manual. During training, the samples need to be labeled, with each color representing the extracted entity element information, and the underline corresponding to the entity element name. After obtaining the extraction model through training, the text to be recognized is input into the extraction model. Finally, the extracted manual element information is output and displayed externally.

[0111] In embodiments of this disclosure, fuzzy extraction rules can be used to perform secondary extraction on the text recognition results to obtain supplementary element information. The fuzzy extraction rules may include: locating entity element fields and their associated synonyms, and matching entity information corresponding to the entity element field or its associated synonyms based on regular expression matching rules. According to embodiments of this disclosure, considering the accuracy loss in product manual recognition and the diversity of product element presentation formats, elements not yet extracted are extracted a second time using fuzzy extraction rules to improve the accuracy of automated recognition. Specifically, a database table can be established for possible synonyms and related terms associated with entity element fields, and populated with extraction element rules that can be constructed for each entity element name. Thus, during the secondary recognition process, entity information corresponding to entity element names or their associated synonyms and related terms can be matched based on regular expression matching rules.

[0112] In some specific embodiments, the product manual may include a financial product manual.

[0113] In some specific embodiments, the entity element name includes at least one of the following: fee type, fee value, accrual basis, product initiation date, product maturity date, custodian, or investment type. In some specific embodiments, taking a bank-related custody product prospectus as an example, Table 2 schematically illustrates the fuzzy extraction rules.

[0114] Table 2

[0115]

[0116] Figure 11 The flowchart illustrates a method for obtaining supplementary element information by performing secondary extraction of the text recognition result using fuzzy extraction rules according to an embodiment of the present disclosure.

[0117] like Figure 11 As shown, the method for extracting supplementary element information by performing secondary extraction of the text recognition result using fuzzy extraction rules in this embodiment includes operations S1110 to S1120.

[0118] In operation S1110, the product element information is matched with a preset element dictionary to determine whether there are any unrecognized entity elements.

[0119] Figure 12 The flowchart illustrates a method for matching product element information with a preset element dictionary to determine whether there are any unrecognized entity elements according to an embodiment of the present disclosure.

[0120] like Figure 12 As shown, the method of matching product element information with a preset element dictionary to determine whether there are unidentified entity elements in this embodiment includes operations S1210 to S1230.

[0121] In operation S1210, a product element information extraction dictionary is obtained, which includes entity element information obtained through a single extraction.

[0122] In operation S1220, the entity element information obtained through one extraction is stored in the preset element dictionary, wherein the preset element dictionary includes an entity element field, and after storage, there is a mapping relationship between the entity element information and the entity element field.

[0123] In operation S1230, when the preset element dictionary contains an entity element field that lacks entity element information with a mapping relationship, the entity element field is determined to be an unidentified entity element.

[0124] According to embodiments of this disclosure, when extracting elements from the output specification, the output can be in the form of dictionary characters. Each time the information extraction model extracts elements from the specification, it returns an extraction information dictionary. The highest-scoring content corresponding to each extracted element can be extracted and placed into a preset element dictionary. The preset element dictionary contains all predefined elements that need to be extracted. If some elements are not extracted by the information extraction model, the return value of that element field in the preset element dictionary is displayed as an empty value. During secondary extraction, the preset element dictionary can be checked first. If a certain element field is empty, it is determined to be an unrecognized entity element.

[0125] After obtaining the unrecognized entity elements, return to operation S1120.

[0126] In operation S1120, the unidentified entity elements are re-identified based on regular expression matching rules to obtain the supplementary element information.

[0127] Figure 13 The flowchart illustrates a method for obtaining supplementary element information by performing secondary identification of the unidentified entity elements based on regular expression matching rules according to an embodiment of the present disclosure.

[0128] like Figure 13 As shown, the method for obtaining supplementary element information by performing secondary identification of the unidentified entity elements based on regular expression matching rules in this embodiment includes operations S1310 to S1330.

[0129] In operation S1310, a keyword list of unidentified entity elements is obtained, wherein the keyword list includes keywords associated with unidentified entity elements, and the keywords associated with unidentified entity elements include unidentified entity elements and their synonyms and / or near-synonyms.

[0130] In operation S1320, the keywords associated with the unidentified entity elements are sorted from highest to lowest according to their frequency of occurrence.

[0131] In operation S1330, regular expressions are used to locate the keywords associated with the unidentified entity elements according to the frequency sorting results, and entity element information that has a mapping relationship with the keywords associated with the unidentified entity elements is obtained.

[0132] According to embodiments of this disclosure, the keyword list for unidentified entity elements can be pre-set. For example, for the element field "Establishment Date," the keyword list can include "Establishment Date," "Establishment Day," "Start and End Dates," and "Interest Date." To save computational resources, the keywords associated with unidentified entity elements can be sorted from highest to lowest frequency before performing regular expression matching. The frequency of occurrence of keywords associated with entity elements can be manually identified. When the number of samples to be identified is large, a small number of samples can be selected and the keyword frequency can be statistically analyzed by computer to achieve sorting. When using regular expressions to locate keywords associated with unidentified entity elements, keywords with higher frequencies can be located first. If keywords with higher frequencies cannot be located, keywords with lower frequencies can be located. Entity element information that has a mapping relationship with the keywords associated with unidentified entity elements can be used as supplementary element information.

[0133] In some preferred embodiments, locating the associated keywords of the unidentified entity element according to the frequency ranking using regular expressions includes: locating the associated keyword of the unidentified entity element that ranks first in frequency using regular expressions. Keywords with lower frequencies can be ignored, thereby further reducing data redundancy and improving data processing efficiency.

[0134] According to other embodiments of this disclosure, auxiliary rules can also be used to improve the fault tolerance of secondary identification and extraction. This can be executed after obtaining entity element information that has a mapping relationship with the keywords associated with the unidentified entity element. Specifically, auxiliary rules can be used to verify the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element to obtain the supplementary element information.

[0135] Figure 14 The flowchart illustrates a method for obtaining supplementary element information by verifying entity element information that has a mapping relationship with the keywords associated with the unidentified entity element using auxiliary rules, according to other embodiments of this disclosure.

[0136] like Figure 14 As shown, in some other embodiments, the method for verifying the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element using auxiliary rules to obtain the supplementary element information includes operations S1410 to S1420.

[0137] In operation S1410, based on the preset element form specification, it is determined whether the entity element information that has a mapping relationship with the keyword associated with the unidentified entity element conforms to the preset element representation form.

[0138] In operation S1420, when the entity element information that has a mapping relationship with the keyword associated with the unidentified entity element conforms to the preset element representation form, the entity element information is determined to be supplementary element information.

[0139] The preset element format specifications can be pre-defined based on expert experience. For example, for date-type instruction manual elements, they should be in numerical form. Furthermore, there is a time difference between the entity element information associated with the keyword "establishment date" and the entity element information associated with the keyword "expiration date." Therefore, the extracted entity element information of these two types of elements may have certain numerical relationships and intervals. For another example, the entity element information associated with the keyword "custodian" should generally be a bank or company, so the extracted information should contain words such as "bank" or "company." The extracted entity element information can be further verified by judging whether the entity element information that has a mapping relationship with the unidentified entity element's associated keyword conforms to the preset element representation format. Element information that does not conform to the preset element representation format can be screened to improve the recognition accuracy.

[0140] In the embodiments of this disclosure, the elements of the product manual can also be processed. Specifically, the extracted elements of the manual can be encapsulated and displayed on the front end.

[0141] Figure 15 A flowchart illustrating a method for processing product specification elements according to an embodiment of this disclosure is shown schematically.

[0142] like Figure 15 As shown, the product specification element processing method of this embodiment includes operations S1510 to S1530.

[0143] In operation S1510, the elements of the product manual are extracted.

[0144] In operation S1520, the extracted elements from the instruction manual are encapsulated into interface messages.

[0145] In operation S1530, when the interface is called, the extracted elements of the instruction manual and the related information of the extracted elements are fed back to the front-end page.

[0146] According to embodiments of this disclosure, the product manual elements can be extracted based on the foregoing product manual element extraction method. For example, it can be based on... Figures 2 to 14The product manual element extraction method extracts the product manual elements. When the front-end accesses the output results, elements containing information can be automatically entered into the system. To save resources, null values ​​can be set to not be entered in the display results. Specifically, after obtaining the display results, the results can be encapsulated into an interface message so that the system application can obtain relevant information after calling the interface in the form of HTTP. The relevant information can include whether the call was successful and the call information display. The call information display can include the display of information for each element and the call time, etc. The relevant information can exist in the form of a dictionary.

[0147] The following combination Figure 16 A flowchart detailing a specific example of a method for extracting product specification elements according to this disclosure is provided.

[0148] like Figure 16 As shown, the flowchart of the method for extracting product specification elements in this example includes steps S1 to S6.

[0149] S1. Preparation of Product Manual Sample: Preprocess the product manuals to be extracted, convert the doc format to plain text, convert the PDF to images, and determine the names of the entity elements to be extracted.

[0150] S2. Text Detection and Recognition Model Training: Prepare samples according to the sample format required by the model, train the target detection model and the text recognition model, and fine-tune the parameters.

[0151] S3. Information Extraction Model Training: Train the PaddlePaddle general information extraction model and perform parameter tuning.

[0152] S4. Model Combination: Combine the target detection model and text recognition model with the PaddlePaddle general information extraction model to form an interface.

[0153] S5. Rule Formulation: Define codes for product elements to construct fuzzy extraction rules, which serve as supplementary extraction of element information following the model interface.

[0154] S6. Extraction of Product Manual Elements: The sample input interface and rule combination code prepared in S1 are used to automatically extract product manual elements, and the extraction results are automatically output on the front end and entered into the system.

[0155] Based on the above-described method for extracting elements from product manuals, this disclosure also provides a device for extracting elements from product manuals. The following will be combined with... Figure 17 The device is described in detail.

[0156] Figure 17 A schematic block diagram of a product specification element extraction device according to an embodiment of the present disclosure is shown.

[0157] like Figure 17 As shown, the product manual element extraction device 1000 of this embodiment includes an acquisition module 1010, an identification module 1020, a first extraction module 1030, a second extraction module 1040, and an element extraction module 1050.

[0158] The acquisition module 1010 is configured to acquire product manual images.

[0159] The recognition module 1020 is configured to perform text recognition on the product manual image based on the object detection model and the text recognition model.

[0160] The first extraction module 1030 is configured to extract product element information by using the text recognition result through an information extraction model.

[0161] The second extraction module 1040 is configured to perform secondary extraction on the text recognition result using fuzzy extraction rules to obtain supplementary element information.

[0162] The element extraction module 1050 is configured to obtain instruction manual extraction elements based on the product element information and the supplementary element information. The instruction manual extraction elements include entity element fields and entity element information, wherein the entity element information corresponds to the entity element fields.

[0163] Figure 18 A schematic block diagram of an identification module according to an embodiment of the present disclosure is shown.

[0164] like Figure 18 As shown, the identification module 1020 of this embodiment includes a positioning submodule 10201, a cutting submodule 10202, and an extraction submodule 10203.

[0165] The positioning submodule 10201 is configured to obtain the position information of the line text in the product manual image based on the target detection model.

[0166] The cropping submodule 10202 is configured to crop the product manual image based on the line text position information to obtain a product manual block image.

[0167] The extraction submodule 10203 is configured to input the product manual block image into the text recognition model to obtain the product manual text information.

[0168] According to embodiments of this disclosure, the first extraction module may further include an input submodule and a first acquisition submodule.

[0169] Figure 19 A schematic block diagram of a first extraction module according to an embodiment of the present disclosure is shown.

[0170] like Figure 19 As shown, the first extraction module 1030 of this embodiment includes an input submodule 10301 and a first acquisition submodule 10302.

[0171] The input submodule 10301 is configured to input the text recognition result into a pre-trained information extraction model. The pre-trained information extraction model is trained based on a smaller than preset number of product manual samples combined with a general extraction framework.

[0172] The first acquisition submodule 10302 is configured to use the pre-trained information extraction model to extract information from the text recognition result and output the product element information.

[0173] According to embodiments of this disclosure, the second extraction module may further include a comparison submodule and a second acquisition submodule.

[0174] Figure 20 A schematic block diagram of a second extraction module according to an embodiment of the present disclosure is shown.

[0175] like Figure 20 As shown, the second extraction module 1040 of this embodiment includes a comparison submodule 10401 and a second acquisition submodule 10402.

[0176] The comparison submodule 10401 is configured to match the product element information with a preset element dictionary to determine whether there are any unidentified entity elements.

[0177] The first acquisition submodule 10402 is configured to perform secondary identification on the unidentified entity elements based on regular expression matching rules to obtain the supplementary element information.

[0178] According to embodiments of this disclosure, the comparison submodule may further include a first acquisition unit, a storage unit, and a judgment unit.

[0179] Figure 21 A schematic diagram illustrating the structure of the comparison submodule according to an embodiment of the present disclosure is shown.

[0180] like Figure 21 As shown, the comparison submodule 10401 of this embodiment includes a first acquisition unit 104011, a storage unit 104012 and a judgment unit 104013.

[0181] The first acquisition unit 104011 is configured to acquire a product element information extraction dictionary, which includes entity element information obtained through a single extraction.

[0182] Storage unit 104012 is configured to store the entity element information obtained by one extraction in the preset element dictionary, wherein the preset element dictionary includes an entity element field, and after storage, there is a mapping relationship between the entity element information and the entity element field.

[0183] The judgment unit 104013 is configured to determine that the entity element field is an unidentified entity element when the entity element field in the preset element dictionary lacks entity element information with a mapping relationship.

[0184] According to embodiments of this disclosure, the first acquisition submodule may further include a second acquisition unit, a sorting unit, and a third acquisition unit.

[0185] Figure 22 A schematic diagram illustrating the structure of a first acquisition submodule according to an embodiment of the present disclosure is shown.

[0186] like Figure 22 As shown, the first acquisition submodule 10402 of this embodiment includes a second acquisition unit 104021, a sorting unit 104022 and a third acquisition unit 104023.

[0187] The second acquisition unit 104021 is configured to acquire a keyword list of unidentified entity elements, wherein the keyword list includes keywords associated with unidentified entity elements, and the keywords associated with unidentified entity elements include unidentified entity elements and their synonyms and / or near-synonyms.

[0188] The sorting unit 104022 is configured to sort the keywords associated with the unidentified entity elements from highest to lowest frequency.

[0189] The third acquisition unit 104023 is configured to use regular expressions to locate the unidentified entity element associated keywords according to the frequency sorting result, and to acquire entity element information that has a mapping relationship with the unidentified entity element associated keywords.

[0190] In embodiments of this disclosure, the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element can be used as the supplementary element information.

[0191] According to some other embodiments of this disclosure, the first acquisition submodule may include a verification unit in addition to the second acquisition unit, the sorting unit and the third acquisition unit.

[0192] Figure 23 A schematic diagram illustrating the structure of a first acquisition submodule according to other embodiments of the present disclosure is shown.

[0193] like Figure 23As shown, in addition to the second acquisition unit 104021, the sorting unit 104022 and the third acquisition unit 104023, the first acquisition submodule 10402 of this embodiment may also include a verification unit 104024.

[0194] The functions of the second acquisition unit 104021, the sorting unit 104022, and the third acquisition unit 104023 can be the same as those of the other units. Figure 23 The structure is the same as that in the previous section, so it will not be repeated here.

[0195] The verification unit 104024 is configured to use auxiliary rules to verify the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element, and to obtain the supplementary element information. This includes: determining whether the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to a preset element representation format based on a preset element format specification; and determining that the entity element information is supplementary element information when the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to the preset element representation format.

[0196] This disclosure also provides an information extraction model training apparatus. The following will be combined with... Figure 24 The device is described in detail.

[0197] Figure 24 A schematic block diagram of an information extraction model training apparatus according to an embodiment of the present disclosure is shown.

[0198] like Figure 24 As shown, the information extraction model training device 1100 of this embodiment includes a sample collection module 1101, a labeling module 1102, and a training module 1103.

[0199] The sample collection module 1101 is configured to obtain product instruction manual samples.

[0200] The annotation module 1102 is configured to annotate the element information of the product instruction manual sample.

[0201] The training module 1103 is configured to input a product manual sample with annotated element information into the general extraction framework for training until a preset cutoff condition is reached, at which point the information extraction model is obtained. The general extraction framework includes a knowledge-enhanced pre-trained model.

[0202] This disclosure also provides a device for processing elements of a product instruction manual. The following will be combined with... Figure 25 The device is described in detail.

[0203] Figure 25A schematic block diagram of a product specification element processing apparatus according to an embodiment of the present disclosure is shown.

[0204] like Figure 25 As shown, the product manual element processing device 1200 of this embodiment can cover Figures 17-24 The product manual element extraction device 1000 functions to extract the product manual elements. Specifically, the product manual element processing device 1200 may include an acquisition module 1010, an identification module 1020, a first extraction module 1030, a second extraction module 1040, and an element extraction module 1050; their specific functions are not detailed here. In embodiments of this disclosure, the product manual element processing device 1200 may further include a result encapsulation module 1060 and a feedback module 1070.

[0205] The result encapsulation module 1060 is configured to encapsulate the extracted elements of the instruction manual into an interface message.

[0206] The feedback module 1070 is configured to send the extracted elements of the instruction manual and the related information of the extracted elements back to the front-end page when the interface is called.

[0207] According to embodiments of this disclosure, in the product manual element extraction device, any and multiple modules among the following can be combined into one module: acquisition module 1010, identification module 1020, first extraction module 1030, second extraction module 1040, element extraction module 1050, positioning submodule 10201, cropping submodule 10202, extraction submodule 10203, input submodule 10301, first acquisition submodule 10302, comparison submodule 10401, second acquisition submodule 10402, first acquisition unit 104011, storage unit 104012, judgment unit 104013, second acquisition unit 104021, sorting unit 104022, third acquisition unit 104023, verification unit 104024, sample collection module 1101, annotation module 1102, and training module 1103, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module. Similarly, in the product manual element processing device, any multiple modules among the following can be combined into one module: acquisition module 1010, identification module 1020, first extraction module 1030, second extraction module 1040, element extraction module 1050, positioning submodule 10201, cropping submodule 10202, extraction submodule 10203, input submodule 10301, first acquisition submodule 10302, comparison submodule 10401, second acquisition submodule 10402, first acquisition unit 104011, storage unit 104012, judgment unit 104013, second acquisition unit 104021, sorting unit 104022, third acquisition unit 104023, verification unit 104024, sample collection module 1101, annotation module 1102, training module 1103, result encapsulation module 1060, and feedback module 1070, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module.According to embodiments of this disclosure, the product manual element extraction device includes an acquisition module 1010, an identification module 1020, a first extraction module 1030, a second extraction module 1040, an element extraction module 1050, a positioning submodule 10201, a cropping submodule 10202, an extraction submodule 10203, an input submodule 10301, a first acquisition submodule 10302, a comparison submodule 10401, a second acquisition submodule 10402, a first acquisition unit 104011, a storage unit 104012, a judgment unit 104013, a second acquisition unit 104021, and a sorting unit 1040. 22. At least one of the third acquisition unit 104023, verification unit 104024, sample collection module 1101, annotation module 1102 and training module 1103 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware and firmware, or in any one of the three implementation methods or any appropriate combination of any of them. Similarly, in the product manual element processing device, there are: acquisition module 1010, identification module 1020, first extraction module 1030, second extraction module 1040, element extraction module 1050, positioning submodule 10201, trimming submodule 10202, extraction submodule 10203, input submodule 10301, first acquisition submodule 10302, comparison submodule 10401, second acquisition submodule 10402, first acquisition unit 104011, storage unit 104012, judgment unit 104013, second acquisition unit 104021, sorting unit 104022, and third acquisition unit 10402. 3. At least one of the following components—verification unit 104024, sample collection module 1101, annotation module 1102, training module 1103, result encapsulation module 1060, and feedback module 1070—can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or encapsulating the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three methods.Alternatively, in the product manual element extraction device, at least one of the following components—acquisition module 1010, identification module 1020, first extraction module 1030, second extraction module 1040, element extraction module 1050, positioning submodule 10201, cropping submodule 10202, extraction submodule 10203, input submodule 10301, first acquisition submodule 10302, comparison submodule 10401, second acquisition submodule 10402, first acquisition unit 104011, storage unit 104012, judgment unit 104013, second acquisition unit 104021, sorting unit 104022, third acquisition unit 104023, verification unit 104024, sample collection module 1101, annotation module 1102, and training module 1103—can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run. Similarly, in the product manual element processing device, at least one of the following modules can be implemented, at least partially, as a computer program module: acquisition module 1010, identification module 1020, first extraction module 1030, second extraction module 1040, element extraction module 1050, positioning submodule 10201, cropping submodule 10202, extraction submodule 10203, input submodule 10301, first acquisition submodule 10302, comparison submodule 10401, second acquisition submodule 10402, first acquisition unit 104011, storage unit 104012, judgment unit 104013, second acquisition unit 104021, sorting unit 104022, third acquisition unit 104023, verification unit 104024, sample collection module 1101, annotation module 1102, training module 1103, result encapsulation module 1060, and feedback module 1070. When the computer program module is run, it can perform corresponding functions.

[0208] Figure 26 A block diagram schematically illustrates an electronic device suitable for implementing a method for extracting product specification elements and / or a method for processing product specification elements according to embodiments of the present disclosure.

[0209] like Figure 26As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0210] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0211] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0212] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0213] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0214] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0215] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0216] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0217] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0218] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0219] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0220] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0221] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for extracting elements from a product instruction manual, characterized in that, include: Obtain product instruction manual images; The product manual image is used to perform text recognition based on an object detection model and a text recognition model, wherein the object detection model is used to recognize the position information of the text in the product manual image. The text recognition results are extracted once using an information extraction model to obtain product element information. The information extraction model is fine-tuned and trained based on a sample of product manuals with a general extraction framework, which is less than a preset number. The text recognition results are then subjected to secondary extraction using fuzzy extraction rules to obtain supplementary element information; and Based on the product element information and the supplementary element information, extract the elements from the instruction manual. The extractable elements in the specification include entity element fields and entity element information, wherein the entity element information corresponds to the entity element fields; The step of using fuzzy extraction rules to perform secondary extraction on the text recognition results to obtain supplementary element information includes: The product element information is matched with a preset element dictionary to determine whether there are any unrecognized entity elements; and The unidentified entity elements are re-identified based on regular expression matching rules to obtain the supplementary element information.

2. The method according to claim 1, wherein, The text recognition of the product manual image based on the object detection model and the text recognition model includes: The position information of the line text in the product instruction manual image is obtained based on the object detection model; Based on the line text position information, the product manual image is cropped to obtain a block image of the product manual; and The product manual block image is input into the text recognition model to obtain the product manual text information.

3. The method according to claim 1, wherein, The text recognition results are used to extract product element information using an information extraction model, including: The text recognition results are input into a pre-trained information extraction model; and The pre-trained information extraction model is used to extract information from the text recognition results, and the product element information is output. The pre-trained information extraction model is trained based on a smaller than preset number of product manual samples combined with a general extraction framework.

4. The method according to claim 3, wherein, Methods for pre-training information extraction models include: Obtain a sample product manual; The product instruction manual sample is annotated with essential information; and The product manual sample, after element information annotation, is input into the general extraction framework for training until a preset cutoff condition is met, at which point the information extraction model is obtained. The general extraction framework includes a knowledge-enhanced pre-trained model.

5. The method according to claim 1, wherein, The step of matching the product element information with a preset element dictionary to determine whether there are any unrecognized entity elements includes: Obtain a product element information extraction dictionary, which includes entity element information obtained through a single extraction. The entity element information obtained through a single extraction is stored in a preset element dictionary, wherein the preset element dictionary includes an entity element field, and after storage, a mapping relationship exists between the entity element information and the entity element field; and When the preset element dictionary contains an entity element field that lacks entity element information with a mapping relationship, the entity element field is determined to be an unidentified entity element.

6. The method according to claim 1, wherein, The step of performing secondary identification on the unidentified entity elements based on regular expression matching rules to obtain the supplementary element information includes: Obtain a keyword list of unidentified entity elements, wherein the keyword list includes keywords associated with unidentified entity elements, and the keywords associated with unidentified entity elements include unidentified entity elements and their synonyms and / or near-synonyms; Sort the keywords associated with the unidentified entity elements from highest to lowest frequency; and Using regular expressions to locate the keywords associated with the unidentified entity elements according to the frequency sorting results, the entity element information that has a mapping relationship with the keywords associated with the unidentified entity elements is obtained as the supplementary element information.

7. The method according to claim 6, wherein, The method of using regular expressions to locate the associated keywords of the unidentified entity elements according to the frequency sorting results includes: Regular expressions are used to locate the keyword associated with the unidentified entity element that ranks first in frequency.

8. The method according to claim 6, wherein, After obtaining entity element information that has a mapping relationship with the keywords associated with the unidentified entity element, the method further includes: The supplementary element information is obtained by using auxiliary rules to verify the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element, including: Based on preset element format specifications, determine whether the entity element information that has a mapping relationship with the keywords associated with the unidentified entity element conforms to the preset element representation format; and When the entity element information that has a mapping relationship with the keyword associated with the unidentified entity element conforms to the preset element representation form, the entity element information is determined to be supplementary element information.

9. A method for processing elements in a product instruction manual, characterized in that, include: Extract the elements of the product manual; The extracted elements from the instruction manual are encapsulated into interface messages; as well as When the interface is invoked, the extracted elements from the specification and their associated information are returned to the front-end page. The product manual elements are extracted using the product manual element extraction method according to any one of claims 1 to 8.

10. A device for extracting elements from a product instruction manual, comprising: The acquisition module is configured to acquire images of product manuals. The recognition module is configured to perform text recognition on the product manual image based on an object detection model and a text recognition model, wherein the object detection model is used to recognize the line text position information of the product manual image; The first extraction module is configured to extract product element information by using an information extraction model to extract the text recognition results once. The information extraction model is fine-tuned and trained based on a sample of product manuals with a general extraction framework, which is less than a preset number. The second extraction module is configured to perform a secondary extraction of the text recognition results using fuzzy extraction rules to obtain supplementary element information; and The element extraction module is configured to obtain extractable elements from the instruction manual based on the product element information and the supplementary element information. The extractable elements in the specification include entity element fields and entity element information, wherein the entity element information corresponds to the entity element fields; The step of using fuzzy extraction rules to perform secondary extraction on the text recognition results to obtain supplementary element information includes: The product element information is matched with a preset element dictionary to determine whether there are any unrecognized entity elements; and The unidentified entity elements are re-identified based on regular expression matching rules to obtain the supplementary element information.

11. A device for processing elements in a product instruction manual, characterized in that, include: The acquisition module is configured to acquire images of product manuals. The recognition module is configured to perform text recognition on the product manual image based on an object detection model and a text recognition model, wherein the object detection model is used to recognize the line text position information of the product manual image; The first extraction module is configured to extract product element information by using an information extraction model to extract the text recognition results once. The information extraction model is fine-tuned and trained based on a sample of product manuals with a general extraction framework, which is less than a preset number. The second extraction module is configured to perform a secondary extraction of the text recognition results using fuzzy extraction rules to obtain supplementary element information; and The element extraction module is configured to obtain extractable elements for the instruction manual based on the product element information and the supplementary element information. The extractable elements for the instruction manual include entity element fields and entity element information, and the entity element information corresponds to the entity element fields. The result encapsulation module is configured to encapsulate the extracted elements from the instruction manual into interface messages; and The feedback module is configured to send the extracted elements of the instruction manual to the front-end page when the interface is called. The step of using fuzzy extraction rules to perform secondary extraction on the text recognition results to obtain supplementary element information includes: The product element information is matched with a preset element dictionary to determine whether there are any unrecognized entity elements; and The unidentified entity elements are re-identified based on regular expression matching rules to obtain the supplementary element information.

12. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images