Automatic component representation method and device based on fine tuning and knowledge graph

Abstract

The invention provides an automatic component representation method and device based on fine tuning and a knowledge graph, and the method comprises the steps: carrying out the understanding of the unstructured content of a line in the data of an electronic component through a large language model, and obtaining a first key feature attribute of the electronic component through word order recovery and attribute information extraction; obtaining a second key feature attribute of the electronic component from a picture in the electronic component data based on the visual and text combined understanding ability of the large language model; namely, important information about the electronic component is obtained from the picture or the table; and constructing the knowledge graph of the electronic component by using the two obtained key feature attributes. According to the method, a fine tuning strategy and a retrieval enhancement generation technology in a deep learning technology are fused, and unstructured Datasheet content is converted into highly structured information graph representation. The comprehensive features in the electronic component data are accurately obtained, the knowledge graph is constructed for comprehensive display, and the breadth and depth of information are ensured.

Description

Technical Field

[0001] The present application relates to the field of electronic information technology, and more specifically, to a method and device for automatic representation of components based on fine-tuning and knowledge graphs. Background Art

[0002] With the rapid development of the electronics industry, data related to electronic components has exploded. Given the diversity and complexity of electronic component attributes, such as package type, manufacturer, application field and other key attributes, these key attributes are often sparsely represented non-continuous parameters, which makes it difficult for traditional processing methods to effectively capture and utilize them.

[0003] Therefore, a new solution for automatic representation of components based on knowledge graphs is needed. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a method and device for automatic representation of components based on fine-tuning and knowledge graph.

[0005] The embodiments of this specification provide the following technical solutions: The embodiment of this specification provides a method for automatically representing components based on fine-tuning and knowledge graph, including: The large language model is used to understand the unstructured content of the text line in the electronic component data, and the first key characteristic attribute of the electronic component is obtained by word order recovery and attribute information extraction; the first key characteristic attribute includes important information about the electronic component hidden in the text line; Based on the joint visual and text understanding capabilities of the large language model, the second key characteristic attribute of the electronic component is obtained from the image in the electronic component data; the second key characteristic attribute includes obtaining important information about the electronic component from the image or table; The first key feature attribute and the second key feature attribute are used to construct a knowledge graph of electronic components.

[0006] The embodiment of this specification also provides a component automatic representation device based on fine-tuning and knowledge graph, including: A first acquisition module is used to understand the unstructured content of the text line in the electronic component data by using a large language model, and obtain the first key characteristic attribute of the electronic component by restoring the word order and extracting the attribute information; the first key characteristic attribute includes important information about the electronic component hidden in the text line; A second acquisition module is used to acquire a second key characteristic attribute of the electronic component from the image in the electronic component data based on the visual and text joint understanding ability of the large language model; the second key characteristic attribute includes acquiring important information about the electronic component from the image or table; A construction module is used to construct a knowledge graph of electronic components using the first key feature attribute and the second key feature attribute.

[0007] An embodiment of the present specification also provides an automatic component representation system based on fine-tuning and knowledge graph, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the automatic component representation method based on fine-tuning and knowledge graph described in the above technical solution.

[0008] An embodiment of the present specification also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the automatic component representation method based on fine-tuning and knowledge graph described in the above technical solution.

[0009] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: The present invention integrates the fine-tuning strategy and retrieval enhancement generation technology in deep learning technology, and designs a set of structured information extraction mechanisms by systematically combing the extensive properties of electronic components, aiming to transform the unstructured datasheet content into a highly structured information graph representation. Accurately obtain the comprehensive features in the electronic component data, build a knowledge graph for comprehensive display, and ensure the breadth and depth of information. This not only enables the characteristics of components to be fully displayed, but also provides a rich and accurate data foundation for subsequent similarity analysis, personalized recommendations and other applications. In addition, by constructing a knowledge graph, the extracted structured data is further organized into a highly correlated graph database, which improves the efficiency and accuracy of data utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 It is an overall schematic diagram of an automatic representation of components based on fine-tuning and knowledge graph of the present invention; Figure 2 It is a schematic diagram of the comparison of two versions of different prompt words used in the present invention; Figure 3 It is a schematic diagram of the change of loss value when using a large model to perform extraction task training in the present invention; Figure 4It is a schematic diagram of the change in accuracy when using a large model for extraction task training in the present invention; Figure 5 It is a schematic diagram of calculating the embedding results of electronic components by performing the GraphSAGE algorithm embedding operation on the knowledge graph in the present invention; Figure 6 It is a flow chart of the present invention for automatic representation of components based on fine-tuning and knowledge graph. DETAILED DESCRIPTION

[0012] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0013] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0014] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0015] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0016] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0017] Given the diversity and complexity of electronic component properties, such as package type, manufacturer, application field and other key attributes, these key attributes are often sparsely represented as non-continuous parameters, which makes it difficult for traditional processing methods to effectively capture and utilize them.

[0018] The traditional method can only extract component data for specific formats, and cannot obtain component data in complex formats. Or the traditional method is limited to extracting a small number of properties of electronic components, and cannot form a comprehensive understanding of the components.

[0019] Based on this, the embodiment of this specification proposes a new solution for automatic representation of components based on fine-tuning and knowledge graphs. By integrating the fine-tuning strategy in deep learning technology with the retrieval enhancement generation technology, a new method is created to transform the unstructured datasheet content into a highly structured information graph representation. Accurately obtain the comprehensive features in the electronic component data, build a knowledge graph for comprehensive display, and the generated vector representation not only highly condenses the key characteristics of the components, but also can accurately match user needs with the most suitable components, and promote the intelligence of the supply chain, ensuring that additional considerations such as environmental protection standards are reflected in the automatic decision-making process. In addition, the introduction of this method will also provide an important tool for the digital transformation of the electronic components industry, enhance the transparency of the product database, and improve the overall efficiency and sustainability of the industry.

[0020] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0021] like Figure 1 and Figure 6 As shown, the present invention provides an automatic component representation method based on fine-tuning and knowledge graph, including steps S601-S603. Step S601, use a large language model to understand the unstructured content of the text line in the electronic component data, and obtain the first key feature attribute of the electronic component; the first key feature attribute includes important information about the electronic component hidden in the text line. Step S602, based on the large language model's joint visual and text understanding capabilities, obtain the second key feature attribute of the electronic component from the picture in the electronic component data; the second key feature attribute includes important information about the electronic component obtained from the picture or table. Step S603, construct a knowledge graph of the electronic component with the first key feature attribute and the second key feature attribute.

[0022] In order to solve the shortcomings of traditional methods, an innovative structured extraction method of the present invention is proposed for the information processing of electronic component DataSheet (data manual), which can accurately obtain comprehensive feature information regardless of the complex format or layout of electronic component data. Specifically, the complex content is finely divided into three categories: text, tables and pictures, and a unique information extraction strategy is tailored for each category. The embodiment of this specification is described in detail using the example that the content layout rules in PDF format cannot be found. Electronic component data includes but is not limited to power management, information processing, sensors, etc.

[0023] It should be noted that, first of all, for electronic component data sheets in PDF format, people usually think that extracting text information from PDF is a simple task because the information is intuitively presented in front of them and humans can easily get what they need from it. However, it turns out that automatic text extraction has become extremely difficult due to the flexibility and inconsistency of the PDF format (such as FillingDB, 2024, source What's so hard about PDF text extraction? Retrieved from https: / / news.ycombinator.com / item?id=22473263). PDF was originally designed to precisely control the output format of documents rather than as a data input mode. Therefore, the text data in PDF is stored in the form of characters that record specific location information on the page, rather than in the form of paragraphs or text (such as FillingDB, 2024). This means that when a text or Word document is converted to PDF, most of the original semantic information will be lost, and all implicit document structures will be converted into empty characters on the page. In the process of building the database, FilingDB extracted data from a large number of PDFs and found that almost all previous assumptions about the structure of PDF documents were invalid. Therefore, processing PDF documents with various sources, styles and layouts is an extremely difficult task (FillingDB, 2024).

[0024] Secondly, datasheets for electronic components have some industry characteristics, which make it more difficult to convert them into valid data.

[0025] Such as the diversity of formats and structures: different manufacturers use different templates and presentation methods; data may be presented in various forms such as tables, charts, text, etc.; there is a lack of unified data structure standards.

[0026] Such as the complexity and correlation of parameters: a single component may contain a large number of technical parameters; there are complex interactions and constraints between parameters; and working conditions and environmental factors need to be considered comprehensively.

[0027] Such as the diversity of data expression methods: typical values ​​(Typical), minimum / maximum values ​​(Min / Max) mixed use; conditional parameters (values ​​under specific conditions); important information contained in notes and descriptions.

[0028] In step S601 of the present application, feature extraction is performed on text information, making full use of the excellent semantic understanding ability of the large language model. Through the main steps of word order recovery and attribute information extraction, the component attribute features hidden in the text are obtained from the disorganized or fragmented text lines in the original PDF document, that is, the first key characteristic attributes of electronic components are accurately captured, which include: application scenarios and a series of key characteristic attributes, such as input / output voltage, operating temperature, manufacturer, etc.

[0029] More important information in electronic components is expressed in the form of structural diagrams.

[0030] In step S602, the image content in the electronic data manual is processed. The present invention applies the fine-tuning technology of the large language model, leverages the powerful pre-training of the multimodal large language model, and combines its powerful ability in joint visual and text understanding to accurately identify and extract important information about electronic components in images or tables, such as symbols and pad packaging information (footprint).

[0031] It should be noted that the present invention obtains important information about electronic components by sorting out the feature relationships in the image through fine-tuning of the large language model and the powerful training function.

[0032] In some embodiments, tables and three views are embedded in the pictures.

[0033] It should be noted that the present invention does not limit the execution order of step S601 and step S602. Step S601 and step S602 can be performed simultaneously or in sequence, such as first performing step S601 and then performing step S602. Of course, step S602 can also be performed first and then performing step S601.

[0034] In step S603, the present invention obtains the first key characteristic attribute and the second key characteristic attribute for the unstructured electronic component data, and constructs a knowledge graph of structured electronic components.

[0035] In combination with the above embodiments, comprehensive feature extraction is performed on electronic component data to ensure the breadth and depth of information, so that the features of components can be fully acquired. After extracting key information from electronic component data in different application scenarios, the structured information extracted from text and images is integrated to generate a structured data representation. By constructing a knowledge graph, the extracted structured data is further organized into a highly correlated graph database, which improves the efficiency and accuracy of data utilization. It also provides a rich and accurate data foundation for subsequent similarity analysis, personalized recommendations and other applications. These data are stored in a database to provide services for downstream applications.

[0036] As mentioned above for electronic component data sheets in PDF format, some reasons that make text extraction from PDF files difficult (or even impossible) are recorded: PDFs sometimes have very small or hidden text on the page. PDFs sometimes have extra spaces between letters of a word, mostly for kerning. PDFs sometimes don't include spaces or replace them with other characters. PDFs sometimes use non-standard fonts and encodings. Reconstructing these empty characters into paragraphs or words in a PDF file is a very difficult task. The order of text and paragraphs in a PDF is difficult to determine.

[0037] It is very difficult to decide the order of text and paragraphs at these levels.

[0038] First, there is no absolute right answer to this order. Documents using traditional single-column layouts can exhibit a natural reading order, while documents using complex layouts make it more challenging to discern the reading order.

[0039] Secondly, even if the order answer is obvious to people, it is still a very difficult problem for a program to determine a stable paragraph order, and it may be even more difficult for artificial intelligence. For example, in some cases, even if the correct paragraph reading order can only be determined by semantic understanding. Therefore, the present application obtains the first key feature attribute by performing word order recovery and attribute information extraction.

[0040] In some embodiments, the first key characteristic attribute of the electronic component is obtained by word order recovery and attribute information feature extraction, including: corresponding word order recovery, extracting scattered text from the electronic component data for word order reorganization including: converting the input text into a vector representation; based on the sentence association analysis layer of the attention mechanism, calculating the semantic association between sentences; constructing a sentence dependency graph and solving the optimal word order; wherein, the optimal word order is obtained by combining the symbols in the text to assist in word order judgment.

[0041] For example, using a large language model to intelligently extract information from PDF documents includes two main steps: word order recovery and attribute information extraction. This can effectively improve the accuracy and efficiency of extracting key attribute information of electronic components from PDF documents.

[0042] Specifically, for word order recovery, a BERT-based word order analysis model is used to reorganize the word order of scattered text extracted from PDF. The pre-trained BERT model is used as an encoder to convert the input text into a vector representation; The sentence association analysis layer based on the attention mechanism calculates the semantic association between sentences; For example, the global word order optimization module based on graph structure builds sentence dependency graph and solves the optimal word order; For example, the rule enhancement module combines explicit features such as punctuation marks and paragraph marks to assist in word order judgment.

[0043] The above-mentioned process of word order recovery converts the input text into vector processing to provide a basis for mining hidden information in text lines, thereby obtaining the semantic correlation between sentences, and solving the optimal word order by using the sentence dependency graph, so as to obtain important information hidden in electronic components from disorganized or fragmented text lines. In the process of solving the optimal word order, the display features such as symbols in the text (such as punctuation marks, paragraph symbols) can also be combined to assist in determining the optimal word order, that is, the display features such as symbols in the text can be used to assist in word order arrangement to mine important information hidden in the text line, thereby improving the accuracy and completeness of the first key feature attribute of the text.

[0044] For intelligent information extraction, a scheme for extracting electronic component attribute information based on a large language model is designed, including: By using RAG dynamic retrieval enhancement, that is, establishing a vector database to store historical extraction cases, you can use algorithms such as cosine similarity to retrieve related cases, so that the most similar cases can be dynamically inserted into the prompt word template. The prompt word template design, by designing a structured attribute extraction template, clearly specifies the target field, so that you can design examples that include a variety of expressions, and you can also add field definition instructions and format specifications to the prompt word template.

[0045] At the same time, through the example enhancement mechanism, by collecting extracted samples in a standard format, normalizing different expressions, and building a sample library to facilitate retrieval and matching, relevant cases can be retrieved based on historical extraction cases, etc., to ensure the efficiency and accuracy of the extraction process.

[0046] In summary, intelligent information extraction is mainly divided into the following designs: First, the prompt word template design: 1) Design a structured attribute extraction template and clearly specify the target field; 2) Include a variety of expression examples; 3) Add field definition instructions and format specifications; Second, the example enhancement mechanism: 1) collects samples in a standard format; 2) normalizes different expressions; 3) builds an example library to facilitate retrieval and matching; Thirdly, RAG dynamic retrieval is enhanced by: 1) establishing a vector database to store historical extracted cases; 2) using algorithms such as cosine similarity to retrieve related cases; 3) dynamically inserting the most similar cases into the prompt word template.

[0047] In summary, by designing prompt word templates, example enhancement mechanism and RAG dynamic retrieval enhancement, setting diversified expression examples, collecting standard samples, and extracting cases most similar to historical cases as references, we can improve the ability to obtain important hidden information about electronic components from text lines and ensure the efficiency and accuracy of the extraction process.

[0048] In some embodiments, the second key characteristic attribute of the electronic component is obtained from the image in the electronic component data based on the joint visual and text understanding ability of the large language model, including: determining the feature position or table position of the image, and obtaining the second key characteristic attribute after sorting out the feature relationship in the image using the question-answering process; the question-answering process contains the source information, name, and specific size data value of the key characteristic attribute. Among them, the image is annotated and trained based on the powerful training resources of the large language model, and a set of highly optimized annotation and question-answering processes are set up using the fine-tuning technology of the large model. The question-answering process can accurately identify and extract the symbol (symbol) and pad packaging information (footprint) of the component.

[0049] Specifically, based on the big prediction model's ability to understand both vision and text, the image's feature position or table position in the image of the electronic component data is determined. After obtaining the feature data in the image, the feature relationship in the image is sorted out with the help of the question-answering process to obtain the second key feature attribute. For example, the question can be: Footprint image link This image describes the thickness of the device package. The answer is "1.5mm". That is, the large language model uses the fine-tuned multimodal model to parse the footprint and symbol for the image. In this way, the model can learn how to extract the component packaging information from the image.

[0050] In some embodiments, for image information extraction, especially footprint images and symbol pin tables in DataSheet, a layout analysis model is used to predict the coordinates of these images, and they are cut out to determine feature locations and table locations, etc.

[0051] In the embodiments of this specification, fine-tuning of the large language model not only utilizes its visual understanding ability, but also designs a question-answering process to sort out the relationship between the contents in the picture. For example, the question-answering process can be used to infer the positions of multiple image features, so as to accurately obtain the source, name, and specific dimensional data value of key feature attributes, and accurately identify and extract the symbol (symbol) and pad packaging information (footprint) of the component.

[0052] In some embodiments, it also includes: annotating the pictures, obtaining a picture data set of electronic components during training; the picture data set is set in the form of question-answer pairs, and the question-answer pairs are used to guide the large language model to sort out and associate feature relationships in the pictures to obtain important information.

[0053] In combination with the above embodiments, the embodiments of this specification use the powerful pre-trained resources of the large language model to design fine-tuning when processing image content, especially for tables and three-view images embedded in the images. During the training process, a large number of annotations are required for these images. For example, for a footprint image, it is necessary to annotate the various package size information of the components described in the image. The annotated dataset will be used to train the deepseek-vl-chat-6B multimodal large language model. During the fine-tuning process, the dataset is constructed in the form of question-answer pairs. For example, a question can be: " Footprint image link This image describes the thickness of the device package. How many millimeters is it? The answer is "1.5mm". In this way, the model can learn how to extract the component package information from the image.

[0054] like Figure 3 and Figure 4 As shown in the figure, the loss value of the fine-tuned multimodal model gradually decreases and the accuracy gradually improves during the training process, indicating that the model performs well in footprint and symbol parsing tasks. That is, when using a large multimodal language model for footprint information extraction tasks, the change in the loss value and the change in accuracy during model training are shown.

[0055] The fine-tuning dataset is carefully constructed to maximize the performance of the model in the specific field of electronic component DataSheet, thereby achieving efficient and accurate information extraction.

[0056] In some embodiments, the method further includes using a layout analysis model on the electronic component data to classify different areas in the electronic component data to obtain text, pictures, or pictures with embedded tables.

[0057] The pictures include the physical layout or packaging information of the components, the symbol diagram of the components and the corresponding pin function description. For example, the footprint picture includes the layout shape and size of the components on the circuit board. The symbol pin table lists the function of each pin in detail (such as input, output, power ground, etc.).

[0058] Specifically, in order to achieve efficient layout analysis, a layout analysis model based on deep learning is used. This model can automatically detect different areas in a PDF document and classify them into different types of content such as text, images, tables, etc. Specifically, we used the LayoutLMv3 model, which performs well in document image understanding tasks and can effectively handle complex PDF layouts.

[0059] For the extraction of image information, especially the footprint images and symbol pin tables in DataSheet, the layout analysis model is used to predict the coordinates of these images and extract them.

[0060] LayoutLMv3 is a multimodal pre-trained model that can process text and visual information simultaneously. It can accurately identify different elements in a document by combining the contextual information of the text and the visual layout of the document. We fine-tune the model to better adapt to the specific format and content of the electronic component DataSheet.

[0061] Or use the PyMuPDF tool to export text content from DataSheet. At the same time, use the layout analysis model to predict the coordinates of the DataSheet's footprint image and symbol pin table and extract the image.

[0062] In some embodiments, the method further includes: the large language model uses the prompt word to guide the first prompt designed by the model to process the text line content, focusing on the target information to obtain the first key feature information; and / or, using the prompt word to guide the second prompt designed by the model to focus on the target information in the picture or table content to obtain the second key feature information. The target information includes a text line, a picture, or a table embedded in a picture, etc.

[0063] In order to further improve the accuracy of extraction, the embodiments of this specification incorporate prompt word engineering technology. Through carefully designed prompt words, the model is guided to focus on the target information more accurately, effectively filter out irrelevant content, and ensure the efficiency and accuracy of the extraction process.

[0064] For example, the extraction of text information uses a large language model (such as the 32B Chat version of Tongyi Qianwen 1.5) for processing. The extraction effect of the model is optimized by designing a prompt word project. The prompt word project is based on the characteristics of the large language model used, such as using context information to design prompt words suitable for it to help the model better understand the text line, so as to more accurately extract the key attribute information of components (such as operating voltage, operating temperature, package type, etc.) from the text.

[0065] like Figure 2 As shown in the figure, the accuracy of extracting device attribute feature fields from text using two different versions of large language model prompt words is compared. In order to eliminate the influence of language model on prompt word preference, two open source large language models with similar parameter values ​​are used for comparative testing. They are the 32B Chat version of Tongyi Qianwen 1.5 and the 34B Chat version of Lingyi Wanwu. After comparison, it can be seen that the second prompt word optimized by prompt word engineering greatly improves the accuracy of extracting attribute feature fields in long context scenarios.

[0066] Compared with general prompt words, the carefully designed prompt words have significantly improved the accuracy and completeness of text information extraction. In this way, structured component attribute information can be effectively extracted from complex PDF documents.

[0067] In some embodiments, the prompt word guidance model can also be used to design the prompt word template in the above embodiments.

[0068] For example, in image processing, through meticulous prompt word engineering, the model can have a deeper understanding of the image context and ensure the accuracy of the extraction process.

[0069] Specifically, in the process of image processing guided by the question-and-answer process, different prompt words can be tried. By comparing the output quality of different prompt words, they can be applied to the electronic components scene process to obtain prompt words suitable for obtaining electronic components, so that the model can have a deeper understanding of the image context and ensure the accuracy of the extraction process.

[0070] In some embodiments, the first key feature attribute and the second key feature attribute are used to construct a knowledge graph of electronic components, including: converting the first key feature attribute and the second key feature attribute into entities and relationships and constructing them into a graph; sampling adjacent nodes and aggregating the features of adjacent nodes to fuse the graph structure of the knowledge graph and the attributes of the node itself, and generating an embedded representation of the node.

[0071] Specifically, the structured data extracted from the electronic component DataSheet is imported into the Neo4j graph database to construct a knowledge graph. The nodes in the knowledge graph represent entities such as components, manufacturers, and package types, and the edges represent the relationships between them (such as "the manufacturer produced the component", "the component has a package type", etc.). Each node also contains attribute information extracted from the DataSheet, such as the operating temperature of the component, input / output voltage, etc.

[0072] To generate an embedding representation of each node in the knowledge graph, the GraphSAGE algorithm in Neo4j is used. GraphSAGE generates node embeddings by sampling the neighbors of the node and aggregating their features. This method is particularly suitable for large-scale graph data and can handle newly added nodes in the graph. First, the knowledge graph is projected into Neo4j's graph data science library so that the GraphSAGE algorithm can process it. The key attributes of the components (such as voltage, temperature, etc.) are selected as the features of the nodes.

[0073] Then, we use the GraphSAGE algorithm to embed the nodes in the knowledge graph. We can generate embedding representations based on the properties of components (such as voltage, temperature, etc.) and the relationships between them. After the training is completed, we can use the trained GraphSAGE model to generate embedding representations for each node in the knowledge graph. Figure 5 This is a simple example of an embedding result (e.g., the embedding result of an electronic component calculated by performing a GraphSAGE algorithm embedding operation on a knowledge graph in a small amount of data scenario), showing the final embedding calculation effect. The generated embedding can be used for tasks such as similarity calculation and recommendation system.

[0074] A knowledge graph is constructed for the extracted structured data, and nodes (entities) and edges (relationships) are used to represent and store devices, manufacturers, etc. in the datasheet and their mutual relationships. The knowledge graph covers rich information such as the operating temperature, input / output voltage, and manufacturer of components. This makes the storage of device information no longer isolated, but interrelated and supportive. Finally, the nodes in the knowledge graph are embedded with the help of the GraphSAGE algorithm. This representation method integrates the graph structure of the knowledge graph and the attributes of the nodes themselves. It can convert various attributes of components and their mutual relationships into vector representations in mathematical space, thereby realizing efficient similarity calculation, recommendation and query.

[0075] The nodes and edges in the graph show the properties of components and their relationships, making the source and processing of data clearer and more transparent. Electronic component information is no longer isolated, but through the combination of nodes and relationships, a highly connected data network is formed, which greatly enhances the availability and analyzability of data.

[0076] In some embodiments, it also includes: based on the knowledge graph structure and the attributes of the nodes themselves, the various attributes of the components and their interrelationships are converted into vector representations in the data space, and the path query and subgraph matching of the knowledge graph database are set to provide services for downstream applications to obtain components similar to the target components.

[0077] The nodes in the knowledge graph are embedded with the help of the GraphSAGE algorithm. This representation method integrates the graph structure of the knowledge graph and the attributes of the nodes themselves. It can convert the various attributes of components and their relationships into vector representations in mathematical space, thereby achieving efficient similarity calculation, recommendation and query.

[0078] Figure 5 This is a simple embedding result example, showing the final embedding calculation effect. The generated embedding can be used for tasks such as similarity calculation and recommendation system.

[0079] Specific downstream applications: 1. The generated embedding vectors can be used for a variety of downstream tasks, such as: Similar component recommendation: By calculating the cosine similarity between the embedding vectors, other components similar to the target component can be quickly found, thereby recommending alternative components. 2. Supply chain optimization: By analyzing the relationship embedding between components and manufacturers, supply chain management can be optimized to find the most suitable supplier.

[0080] In combination with the above embodiments, the extracted text information and image information are integrated to generate a structured data representation. These data can be stored in a MySQL database for use by downstream business systems, or the Pandas library can be used to store structured data as a CSV file for subsequent scientific calculations and analysis.

[0081] The hardware facilities used in this application: CPU: Intel(R) Xeon(R) Gold 6230R; Graphics card: NVIDIA A10080G.

[0082] Software environment: Operating system: Ubuntu 20.04 Python version: 3.8 Related libraries: PyMuPDF, PyTorch, Hugging Face Transformers, OpenCV, Pandas Database: Neo4j, MySQL In combination with the above embodiments, the embodiments of this specification also provide a component automatic representation device based on fine-tuning and knowledge graph, including: A first acquisition module is used to understand the unstructured content of the text line in the electronic component data by using a large language model to obtain a first key characteristic attribute of the electronic component; the first key characteristic attribute includes important information about the electronic component hidden in the text line; A second acquisition module is used to acquire a second key characteristic attribute of the electronic component from the image in the electronic component data based on the visual and text joint understanding ability of the large language model; the second key characteristic attribute includes acquiring important information about the electronic component from the image or table; A construction module is used to construct a knowledge graph of electronic components using the first key feature attribute and the second key feature attribute.

[0083] A component automatic representation system based on fine-tuning and knowledge graph is also provided, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the component automatic representation method based on fine-tuning and knowledge graph described in any one of the above technical solutions.

[0084] A readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the automatic component representation method based on fine-tuning and knowledge graph as described in any one of the above technical solutions.

[0085] This invention creates a novel method by integrating the fine-tuning strategy in deep learning technology with the retrieval enhancement generation technology, aiming to transform the unstructured datasheet content into a highly structured information graph representation. The generated vector representation not only highly condenses the key characteristics of the components, but also allows effective similarity calculation in the mathematical space, thereby quickly retrieving alternative devices or similar devices, which lays a solid foundation for the automatic recommendation system in electronic product design. It can accurately match user needs with the most suitable components, promote the intelligence of the supply chain, and ensure that additional considerations such as environmental standards are reflected in the automatic decision-making process. In addition, the introduction of this method will also provide an important tool for the digital transformation of the electronic components industry, enhance the transparency of the product database, and improve the overall efficiency and sustainability of the industry.

[0086] Compared with the prior art, the present invention has achieved significant breakthroughs and optimizations in the field of electronic component information processing, and its beneficial effects are mainly reflected in the following aspects: Comprehensiveness and depth improvement: Most existing solutions are limited to extracting a small number of properties of electronic components, making it difficult to form a comprehensive understanding of the components.

[0087] The present invention systematically combs through the extensive properties of electronic components and designs a set of structured information extraction mechanisms to ensure the breadth and depth of information. This not only enables the characteristics of components to be fully displayed, but also provides a rich and accurate data foundation for subsequent similarity analysis, personalized recommendations and other applications. In addition, by constructing a knowledge graph, the extracted structured data is further organized into a highly correlated graph database, which improves the efficiency and accuracy of data utilization.

[0088] Innovation in technical means and synergistic effects: The present invention cleverly integrates the pre-training capabilities of large language models, significantly improves the semantic understanding capabilities when processing unstructured texts, and reduces the complexity and error rate of traditional algorithms. At the same time, the present invention innovatively combines the image understanding function of the multimodal model, especially when processing tables and graphic information in the datasheet, it demonstrates the advantages of cross-modal collaborative processing. This cross-modal information processing strategy greatly optimizes the information extraction process, reduces manual intervention, and improves the fluency and efficiency of the entire process. The introduction of knowledge graphs further strengthens the organization and management of data, so that electronic component information is no longer isolated, but through the combination of nodes and relationships, a highly connected data network is formed, greatly enhancing the availability and analyzability of data.

[0089] Engineering feasibility and cost-effectiveness: Through the above-mentioned technological innovations, the present invention significantly reduces the difficulty of extracting structured information from diverse data sources, which not only improves the feasibility of engineering implementation, but also optimizes cost-effectiveness. With the help of intelligent processing models, the development cycle is greatly shortened and labor costs are reduced accordingly. At the same time, the improvement in the degree of automation reduces human errors and avoids waste of resources. The application of knowledge graphs further optimizes the information query and management process, accelerates product design and supply chain management, especially in the task of quickly finding similar or alternative components, significantly improving efficiency and reducing inventory backlogs and risks. This dual technical and economic benefit has brought revolutionary changes to electronic design automation and related industries.

[0090] Data transparency and interpretability: Through the construction of the knowledge graph, the present invention significantly enhances the transparency and interpretability of data. The nodes and edges in the graph show the properties of components and their relationships, making the source and processing of the data clearer and more transparent. This not only helps with data quality control, but also greatly increases the user's trust in the system. In addition, the visualization tools of the graph database enable users to intuitively understand the complex relationships between components, so as to better grasp the basis for system recommendations and analysis results, further improving the user experience.

[0091] Support for advanced analysis and applications: Thanks to the construction of the knowledge graph, the present invention provides strong support for advanced analysis and applications. For example, through the path query, subgraph matching and other functions of the graph database, it is possible to quickly find components similar to or replaceable with the target components, accelerating product design and supply chain optimization. With the help of graph algorithms (such as the shortest path, community discovery, etc.), the implicit relationships and patterns between components can be revealed, providing a new perspective for complex analysis tasks such as market trend forecasting and supply chain risk assessment, further enhancing the market responsiveness and competitiveness of enterprises.

[0092] In summary, the present invention has successfully solved the comprehensive problem of electronic component information extraction by comprehensively applying advanced artificial intelligence technology, and has lowered the implementation threshold through innovative technology combinations, significantly improving the efficiency and economy of engineering practice. In particular, the application of knowledge graphs has significantly improved the transparency, interpretability and advanced analysis capabilities of data, surpassing the limitations of existing technologies and bringing new value to the industry.

[0093] The traditional way is as follows: The method proposed by Castellanos et al. is to automatically extract structured data from web pages. Since web pages are composed of HTML language, HTML syntax has a relatively strict organization and nested structure, which can reduce the difficulty of improving structured data. In comparison, the content layout rules in PDF are difficult to find, and automatic extraction is more difficult.

[0094] T. -P. Hong et al. proposed an automatic process for extracting dimensional parameters from three-view drawings. This process is divided into two stages. In the first stage, deep learning technology is used to detect three views in the data sheet, identify the text areas containing parameters in the drawings, and then identify the values ​​in these areas. In the second stage, two algorithms are designed: a k-nearest neighbor (k-NN)-based method and a statistical evaluation method, which are used to match the digitized parameters with the identified values. This work only interprets the three-segment picture part in the DataSheet, and the process is complicated and difficult to implement. The upstream tasks strongly rely on the accuracy of the underlying algorithm.

[0095] K. -C. Chen, et al. classified various charts (including tables) in the electronic component DataSheet and proposed the ECS-YOLO model for large-scale chart extraction and classification, aiming to accelerate the PCB design preparation process. This work only helps to improve the accuracy of chart classification tasks in DataSheet.

[0096] Tian F et al. proposed an automated tool that can extract dynamic and complex nonlinear characteristics from semiconductor data sheets to improve the power loss estimation model and achieve optimal power converter design. The tool uses the object recognition neural network CenterNet training to extract charts from the data sheet. Then, optical character recognition (OCR) and morphological image processing techniques are used to extract dynamic data from the charts. Finally, the acquired data is used to enhance the dynamic performance of the power switch and establish a more accurate power loss model as input for the design automation tool to improve design efficiency and accuracy. This work focuses on the construction of semiconductor power loss models, and its versatility needs to be expanded.

[0097] In general, the present invention systematically overcomes the difficulty of extracting structured information from diverse DataSheet contents through the above-mentioned hierarchical and differentiated methods, and builds a graph database on this basis, providing powerful data pre-processing support for electronic design automation tools.

[0098] The same or similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiment described later, since it corresponds to the method, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0099] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A component automatic representation method based on fine-tuning and knowledge graph, characterized in that: include: The large language model is used to understand the unstructured content of text lines in the electronic component data, and the first key characteristic attributes of the electronic components are obtained through word order recovery and attribute information extraction; The first key feature attribute includes important information about the electronic components hidden in the text line; Based on the joint visual and text understanding capabilities of the large language model, the second key characteristic attribute of the electronic component is obtained from the image in the electronic component data; the second key characteristic attribute includes obtaining important information about the electronic component from the image or table; The first key feature attribute and the second key feature attribute are used to construct a knowledge graph of electronic components.

2. The component automatic representation method based on fine-tuning and knowledge graph according to claim 1, characterized in that: The first key characteristic attributes of electronic components are obtained by word order recovery and attribute information feature extraction, including: Corresponding to word order recovery, extracting scattered text from electronic component data to reorganize word order includes: converting input text into vector representation; The sentence association analysis layer based on the attention mechanism calculates the semantic association between sentences; Construct a sentence dependency graph and solve the optimal word order; the optimal word order is also determined by combining the symbols in the text to assist in word order judgment; Corresponding to the attribute feature extraction, a vector database is established to store historical extraction cases, and a similarity algorithm is used to retrieve related cases so as to dynamically insert the most similar cases into the prompt word template.

3. The component automatic representation method based on fine-tuning and knowledge graph according to claim 1 is characterized in that: Based on the joint understanding ability of vision and text in the large language model, the second key feature attributes of electronic components are obtained from the images in the electronic component data, including: Determine the feature position or table position of the image, and use the question-and-answer process to sort out the feature relationship in the image to obtain the second key feature attribute; the question-and-answer process includes the source information, name, and specific size data value of the key feature attribute; or, also include: The images are labeled, and an image dataset of electronic components is obtained during the training process; the image dataset is set in the form of question-answer pairs, and the question-answer pairs are used to guide the large language model to sort out and associate feature relationships in the images to obtain important information.

4. The component automatic representation method based on fine-tuning and knowledge graph according to any one of claims 1 to 3, characterized in that: Also includes: A layout analysis model is used for electronic component data to classify different areas in the electronic component data to obtain text, pictures or pictures with embedded tables; wherein the pictures include physical layout or packaging information of components, symbol diagrams of components and corresponding pin function descriptions.

5. The component automatic representation method based on fine-tuning and knowledge graph according to any one of claims 1 to 3, further comprising: The large language model uses the prompt words to guide the first prompt designed by the model to process the content of the text line, focusing on the target information to obtain the first key feature information; And / or, a second prompt designed by using prompt words to guide the model focuses on target information in the picture or table content to obtain second key feature information.

6. The component automatic representation method based on fine-tuning and knowledge graph according to claim 1 is characterized in that: The first key feature attribute and the second key feature attribute are used to construct a knowledge graph of electronic components, including: Convert the first key feature attribute and the second key feature attribute into entities and relationships and construct them into a graph; By sampling adjacent nodes and aggregating the features of adjacent nodes, the graph structure of the knowledge graph and the properties of the node itself are integrated to generate an embedded representation of the node.

7. The component automatic representation method based on fine-tuning and knowledge graph according to claim 6 is characterized in that: Also includes: Based on the knowledge graph structure and the properties of the nodes themselves, the various properties of components and their relationships are converted into vector representations in the data space, and the path query and subgraph matching of the knowledge graph database are set to provide services for downstream applications to obtain components similar to the target components.

8. A component automatic representation device based on fine-tuning and knowledge graph, characterized in that: include: A first acquisition module is used to understand the unstructured content of the text line in the electronic component data by using a large language model, and obtain the first key characteristic attribute of the electronic component by restoring the word order and extracting the attribute information; the first key characteristic attribute includes important information about the electronic component hidden in the text line; A second acquisition module is used to acquire a second key characteristic attribute of the electronic component from the image in the electronic component data based on the visual and text joint understanding ability of the large language model; the second key characteristic attribute includes acquiring important information about the electronic component from the image or table; A construction module is used to construct a knowledge graph of electronic components using the first key feature attribute and the second key feature attribute.

9. A component automatic representation system based on fine-tuning and knowledge graph, characterized in that: include: A memory, a processor and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the component automatic representation method based on fine-tuning and knowledge graph as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the automatic component representation method based on fine-tuning and knowledge graph as described in any one of claims 1 to 7.

Images