Schematic diagram symbol automatic generation method, system and device based on VLM and medium

Abstract

The invention discloses a schematic diagram symbol automatic generation method, system and device based on VLM and a medium, and relates to the technical field of electronic design, the method comprises the following steps: inputting information to be processed into a trained visual language model for processing, and the processing process comprises the following steps: obtaining a PDF page number of the information to be processed in a PDF manual; performing target area detection in the image corresponding to the PDF page number, and obtaining an area image corresponding to the to-be-processed information; respectively extracting symbol information of the to-be-processed information from the region images subjected to resolution adjustment and direction correction; formatting parameters are formed through the PDF page number and the symbol information and output; the formatting parameters are imported into an EDA tool to be processed, and a symbol diagram which is created by the EDA tool and used for drawing a schematic diagram is obtained; the schematic diagram symbols are automatically generated based on the VLM, and the method has the advantages of being simple in process, good in generalization ability, high in adaptability, high in precision, capable of outputting multi-dimensional parameters in a formatted mode and the like.

Description

Technical Field

[0001] This invention relates to the field of electronic design technology, and more specifically, to a method, system, device, and medium for automatic generation of schematic symbols based on VLM. Background Technology

[0002] Existing methods for automatically generating schematic symbols rely on a combination of algorithms, including image recognition, OCR extraction, and semantic matching. The principle behind these algorithms is that schematic symbols are created based on device pin assignments, which are paired.<pin number,pin name> The list includes pin numbers describing pin location information and pin names describing pin function information; pin assignments exist in symbol diagrams or symbol tables; page numbers where the target might be located are retrieved using predefined keywords, such as Pin Assignment & Pin Configuration; then, image recognition technology is used to extract the target area from the symbol diagram or symbol table on the page; finally, OCR technology is used to extract text information from the target area, and manually written semantic matching rules are used to assemble the text information into...<pin number,pin name> Lists; however, this method is cumbersome and complex, prone to error accumulation; and it is not intelligent enough, requiring manual writing of rules to implement semantic matching rules, with poor generalization ability, high error rate, and inability to automatically output multi-dimensional formatted parameter lists.

[0003] Therefore, this application is hereby submitted. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device and medium for automatic generation of schematic symbols based on VLM. The automatic generation of schematic symbols based on VLM has the characteristics of simple process, good generalization ability, strong adaptability, high accuracy and multi-dimensional parameter formatted output.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] Firstly, this application provides a method for automatically generating schematic symbols based on VLM, including the following specific steps:

[0007] The information to be processed is input into the trained visual language model for processing, and the formatted parameters corresponding to the information to be processed are obtained.

[0008] The formatting parameters are imported into the EDA tool for processing, resulting in a symbol diagram created by the EDA tool and used to draw the schematic.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Furthermore, the information to be processed is input into the trained visual language model for processing, specifically as follows:

[0011] Obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, use a general VLM to check whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed;

[0012] Target region detection is performed on the image corresponding to the PDF page number, and the target image corresponding to the information to be processed is obtained. At the same time, the symbol macro information corresponding to the target image is extracted. The symbol macro information includes at least category, symbol image type, location information, package type, and device name.

[0013] From the target image after resolution adaptive adjustment and the pin region sub-image in the target image after text direction correction, the symbol details of the information to be processed, namely the pin number or pin sequence number and the corresponding pin name, are extracted in one go. The macroscopic symbol information and symbol details are integrated to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, where:

[0014] To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped.

[0015] When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details.

[0016] The formatting parameters are generated and output using PDF page numbers and symbol information.

[0017] Furthermore, the trained visual language model described above is obtained through the following method:

[0018] The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

[0019] Furthermore, the above training samples were obtained in the following way:

[0020] Step 4.1, Construction of page number retrieval training samples:

[0021] Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information;

[0022] Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information;

[0023] For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained".

[0024] Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image.

[0025] Step 4.2, Construction of training samples for target region detection and information extraction:

[0026] The training samples for this step are constructed using the positive sample set from step 4.1, and labeled with the target region location and size and target region information. The target region information is the macroscopic information of the symbol, including: category, package type, device name, pin region location and size, pin region text orientation, etc. A single sample usually contains multiple target regions.

[0027] Step 4.3, extracting symbol details from training samples for construction:

[0028] Similarly, the positive sample set from step 4.1 is used to construct the training samples for this step. Based on the annotation information from step 4.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name.

[0029] Step 4.4, Merging training samples:

[0030] The training sets of steps 4.1, 4.2, and 4.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

[0031] Secondly, this application provides a VLM-based automatic schematic symbol generation system, applicable to the VLM-based automatic schematic symbol generation method of any of the first aspects, including:

[0032] The parameter acquisition module is used to input the information to be processed into the trained visual language model for processing, and obtain the formatted parameters corresponding to the information to be processed.

[0033] The symbol generation module is used to import formatting parameters into the EDA tool for processing, and obtain the symbol diagram created by the EDA tool and used to draw the schematic diagram.

[0034] Furthermore, the information to be processed is input into the trained visual language model for processing, specifically as follows:

[0035] Obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, use a general VLM to check whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed;

[0036] Target region detection is performed on the image corresponding to the PDF page number, and the target image corresponding to the information to be processed is obtained. At the same time, the symbol macro information corresponding to the target image is extracted. The symbol macro information includes at least category, symbol image type, location information, package type, and device name.

[0037] From the target image after resolution adaptive adjustment and the pin region sub-image in the target image after text direction correction, the symbol details of the information to be processed, namely the pin number or pin sequence number and the corresponding pin name, are extracted in one go. The macroscopic symbol information and symbol details are integrated to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, where:

[0038] To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped.

[0039] When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details.

[0040] The formatting parameters are generated and output using PDF page numbers and symbol information.

[0041] Furthermore, the trained visual language model described above is obtained through the following method:

[0042] The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

[0043] Furthermore, the above training samples were obtained in the following way:

[0044] Step 8.1, Construction of page number retrieval training samples:

[0045] Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information;

[0046] Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information;

[0047] For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained".

[0048] Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image.

[0049] Step 8.2, Construction of training samples for target region detection and information extraction:

[0050] The training samples for this step are constructed using the positive sample set from step 8.1, and labeled with the target region location and size and target region information. The target region information is the macroscopic information of the symbol, including: category, package type, device name, pin region location and size, pin region text orientation, etc. A single sample usually contains multiple target regions.

[0051] Step 8.3, extracting symbol details and constructing training samples:

[0052] Similarly, the positive sample set from step 8.1 is used to construct the training samples for this step. Based on the annotation information from step 8.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name.

[0053] Step 8.4, Merging training samples:

[0054] The training sets of steps 8.1, 8.2, and 8.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

[0055] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of the first aspects.

[0056] Fourthly, this application provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to perform any of the methods in the first aspect.

[0057] Compared with the prior art, the present invention has at least the following beneficial effects:

[0058] This invention overcomes the problems of single-model single-task, complex workflow, manual writing of a large number of rules and semantic matching, lack of support for various parameter list outputs, weak generalization and low accuracy in the aforementioned prior art. At the same time, the VLM used in this invention can realize the simultaneous processing of multiple tasks in a single model, simplify the workflow and output a formatted parameter list. The VLM base model is trained with massive amounts of data, has high accuracy and semantic understanding ability, good generalization and does not require writing a large number of manual rules. Attached Figure Description

[0059] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0060] Figure 1 This is an overall flowchart of the generation method in an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the image to be processed in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the region result obtained in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram illustrating the display of target detection formatting parameters using a JSON formatting tool in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the box1 region in an embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the box2 region in an embodiment of the present invention;

[0066] Figure 7 This is a schematic diagram of the box3 region in an embodiment of the present invention;

[0067] Figure 8 This is a schematic diagram of the box4 region in an embodiment of the present invention;

[0068] Figure 9 This is a schematic diagram illustrating the display of the formatted parameter list using a JSON formatting tool in an embodiment of the present invention;

[0069] Figure 10 This is a flowchart of the generation method in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0071] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0072] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0073] In the description of the embodiments of the present invention, "multiple" means at least two.

[0074] Example 1: To address the problems of current schematic symbol generation methods, such as cumbersome and complex implementation processes leading to error accumulation, insufficient intelligence requiring manual rule writing for semantic matching, poor generalization ability, high error rates, and inability to automatically output multi-dimensional formatted parameter lists, this example provides a VLM-based automatic schematic symbol generation method, such as... Figure 10 As shown, the specific steps include the following:

[0075] S1, input the information to be processed into the trained visual language model for processing, and obtain the formatted parameters corresponding to the information to be processed.

[0076] The above involves inputting the information to be processed into a trained visual language model for further processing, such as... Figure 1 As shown, specifically:

[0077] S11, obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, and use a general VLM to determine whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed.

[0078] S12, perform target region detection in the image corresponding to the PDF page number, obtain the target image corresponding to the information to be processed, and simultaneously extract the symbol macro information corresponding to the target image. The symbol macro information includes at least category, symbol image type, location information, package type, and device name.

[0079] S13, from the target image after resolution adaptive adjustment and the pin area sub-image after text direction correction in the target image, extract the symbol details of the information to be processed in one go, namely the pin number or pin sequence number and the corresponding pin name. Integrate the symbol macro information and symbol details to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, where:

[0080] To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped.

[0081] When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details.

[0082] S14 generates formatting parameters from PDF page numbers and symbol information and outputs them.

[0083] Optionally, the trained visual language model described above is obtained in the following way:

[0084] The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

[0085] The training samples mentioned above were obtained in the following way:

[0086] Step 4.1, Construction of page number retrieval training samples:

[0087] Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information;

[0088] Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information;

[0089] For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained".

[0090] Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image.

[0091] Step 4.2, Construction of training samples for target region detection and information extraction:

[0092] The training samples for this step are constructed using the positive sample set from step 4.1, and labeled with the target region location and size and target region information. The target region information is the macroscopic information of the symbol, including: category, package type, device name, pin region location and size, pin region text orientation, etc. A single sample usually contains multiple target regions.

[0093] Step 4.3, extracting symbol details from training samples for construction:

[0094] Similarly, the positive sample set from step 4.1 is used to construct the training samples for this step. Based on the annotation information from step 4.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name.

[0095] Step 4.4, Merging training samples:

[0096] The training sets of steps 4.1, 4.2, and 4.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

[0097] S2. Import the formatting parameters into the EDA tool for processing to obtain the symbol diagram created by the EDA tool and used to draw the schematic diagram.

[0098] Example 2: To address the problems of current schematic generation methods requiring multiple algorithms such as image recognition, OCR extraction, and semantic matching for comprehensive processing, resulting in a cumbersome and complex implementation process and insufficient intelligence, this example provides a method for automatic generation of schematic symbols based on VLM. This method includes two aspects: 1. Training VLM; 2. Using VLM to automatically generate schematic symbols.

[0099] It should be noted that VLM is a visual language model developed based on GPT. The VLM base model is trained on massive amounts of data and possesses rich world knowledge. It also has capabilities such as image recognition, OCR extraction, and semantic understanding. Therefore, by fine-tuning the VLM base model using schematic symbol data, automatic generation of schematic symbols based on VLM can be achieved. This method features a simple process, good generalization ability, strong adaptability, high accuracy, and multi-dimensional parameter formatted output. Although the VLM base model is already very powerful in terms of general knowledge, its industry-specific knowledge is usually relatively weak. Therefore, it is necessary to use industry-specific data to fine-tune and train the VLM base model. The usage process of VLM determines how VLM should be trained. The following will further explain the process of automatic generation of schematic symbols using VLM and the VLM training method.

[0100] Automatic generation process of schematic symbols:

[0101] 1. Pin Assignment Information Page Number Retrieval: Pin Assignments exist in symbol diagrams and symbol lists. It is necessary to determine which page of the PDF contains the symbol diagram and symbol list. Similar expressions to Pin Assignment will appear in the image annotations of symbol diagrams and the headers of symbol lists to indicate that the image is a symbol diagram and the table is a symbol list. At the same time, symbol diagrams and symbol lists have unique graphic and image appearances. Based on these two characteristics, traditional methods use keyword retrieval and image recognition technology respectively.

[0102] In this embodiment, the PDF manual is converted into images, and all or batches of images are input into VLM, allowing VLM to infer the image sequence number of the Pin Assignment, i.e., the PDF page number. This capability is due to VLM being trained using massive amounts of image and video data. Unlike schemes that use separate keyword retrieval and image recognition, VLM has both capabilities, meaning it can understand both text and graphics in images, and can also understand various expressions related to Pin Assignments.

[0103] 2. Preprocessing for Pin Assignment Information Extraction: After obtaining the PDF page number where the Pin Assignment information is located, the corresponding page can be converted into an image and input into VLM to extract the Pin Assignment. However, due to various factors, this can easily lead to a large amount of VLM output illusion (i.e., many errors). Therefore, it is necessary to perform preprocessing to remove the factors that can easily cause illusion.

[0104] 2.1) Target Region Detection: Symbol images and symbol tables typically do not fill an entire page, and multiple symbols may exist on a single page. While VLM can handle these issues to some extent, to achieve better extraction results, this patent identifies and cuts out the symbol images and symbol tables from the page before extracting them separately, thus reducing interference from redundant information. Traditional image recognition technologies can only identify the location and category of the target, but VLM can identify more dimensions of information and output the detection results in a formatted manner: category, such as symbol image or symbol list; symbol image type, such as two-column, four-sided, or matrix; location information; device package type; device name, etc. Specific device package types and device names must be extracted in this step because they are usually present in the contextual information of the symbol images and symbol tables, such as headers and image annotations, which is difficult to achieve with other methods.

[0105] 2.2) Automatic resolution adjustment: When a page is exported at a fixed dpi, the position and size of the symbol image are obtained through target area detection. The symbol image usually does not fill a page, and some are very small. Small symbol images can easily cause VLM output illusions. In order to ensure that the resolution of the symbol image is sufficient, this patent design constrains the minimum resolution of the symbol image. By comparing the target size detected in step 2.1 with the minimum resolution, the required dpi scaling ratio is calculated, the page image is re-exported, and finally the symbol image with the required resolution is cropped for Pin Assignment information extraction.

[0106] 2.3) Pin Name Orientation Correction: Symbol diagrams for devices such as QFP, QFN, and PLCC exist in a four-sided pattern. These diagrams often have pin names with various orientations. Unaligned pin names can cause visual illusions in the VLM output. Humans also tend to misread tilted or upside-down text. Therefore, in the target area detection step, if a four-sided symbol diagram is detected, the position and size of the pin area and the pin name orientation information will also be output. After completing the automatic resolution adjustment in step 2.2, for four-sided symbol diagrams, in addition to cropping the target area itself, a pin area sub-image also needs to be cropped, and the pin area sub-image needs to be rotated and aligned according to the detected pin name orientation information.

[0107] 3. Pin Assignment Information Extraction: After preprocessing, this step has obtained a target area image with a resolution that meets the requirements for extracting Pin Assignment information. The image data is input into VLM to obtain a formatted parameter list: category, such as symbol image or symbol list; symbol image type, such as two-column, four-column, or matrix; pin names or alphanumeric labels for the top, bottom, left, and right sides of the symbol image; and Pin Assignment, i.e., the natural order.<pin number,pin name> List; specifically, for a four-sided symbol diagram, the image data input to the VLM includes: the symbol diagram body cut out in step 2.2, and the pin area sub-diagram with orientation correction described in step 2.3.

[0108] 4. Write the extraction results from step 3 and the page numbers from step 1 into the parameterized list from step 2 as the final output.

[0109] 5. Import the formatting parameters output from the above steps into EDA and create a symbol diagram for drawing the schematic. It should be noted that, ideally, the symbol diagram in the PDF manual can be copied into the symbol diagram used in the EDA tool.

[0110] VLM training methods:

[0111] When training VLM, the input is a list of images and system prompts. The image list can contain one or more images, and the output is the corresponding data annotations. The specific training steps of VLM include:

[0112] 1. Training in retrieving page numbers from Pin Assignment information:

[0113] 1.1) Basic Data:

[0114] Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information;

[0115] Negative sample set: A set of images converted from PDF pages that do not contain Pin Assignment information.

[0116] 1.21) For a single sample, a binary classification training method is adopted, such as: the system prompt word is designed as "Does the image contain a symbol image or symbol table?", with "containing" for positive samples and "not containing" for negative samples.

[0117] 1.22) Randomly sample multiple images from the positive and negative samples and shuffle them to form a training sample. Label the images as the sequence numbers of the positive sample images. The system prompt can be designed as "Find the images containing symbol diagrams or symbol tables and output the sequence numbers of the images".

[0118] 2. Pin Assignment target area detection and four-sided symbolic pin area detection:

[0119] Using the positive sample set from the previous step, label the target location and size according to the grounding format of the selected VLM. Different VLMs will have different formats, such as using pixel values ​​or scale values; a typical labeling format for a single target region is as follows:

[0120] {"Location Size":"(x,y,w,h)","Category":"Symbol List","Package Type":"Specific Value","Device Name":"Specific Value"};

[0121] {"Location Size":"(x,y,w,h)","Category":"Symbol Diagram","Symbol Diagram Type":"Two-Column / Array","Package Type":"Specific Value","Device Name":"Specific Value"};

[0122] {"Position Size":"(x,y,w,h)","Category":"Symbol Diagram","Symbol Diagram Type":"Square","Pin Area":{"Top":{"Position Size":"(x,y,w,h)","Character Orientation":"Top / Bottom / Left / Right"},"Bottom":{"Position Size":"(x,y,w,h)","Character Orientation":"Top / Bottom / Left / Right"},"Left":{"Position Size":"(x,y,w,h)","Character Orientation":"Top / Bottom / Left / Right"},"Right":{"Position Size":"(x,y,w,h)","Character Orientation":"Top / Bottom / Left / Right"}},"Package Type":"Specific Value","Device Name":"Specific Value"};

[0123] Positive samples typically contain multiple target regions, so the complete annotation format is: [region 1 annotation, region 2 annotation...].

[0124] 3. Pin Assignment Information Extraction Training:

[0125] Based on the annotation information of the data in step 2, the symbol diagram and symbol table are cropped out; for the four-sided symbol diagram, the pin area sub-diagram is also cropped out and aligned; for the four-sided symbol diagram, the image input includes the symbol diagram body and the aligned pin area sub-diagram; for other types of symbol diagrams and symbol tables, the image input is a single image.

[0126] For the labeling method of two-column / quadruple symbol diagrams: {"Category":"Symbol Diagram","Symbol Diagram Type":"Two-column / quadruple","Pin Number":{"Top":"Specific Value","Bottom":"Specific Value","Left":"Specific Value","Right":"Specific Value"},"Pin Assignment":{"Number 1":"Name 1","Number 2":"Name 2",...}};

[0127] For pin numbers in a two-column format: when "top" and "bottom" have values, "left" and "right" are empty, i.e., ""; conversely, when "left" and "right" have values, "top" and "bottom" are empty.

[0128] For the labeling method of array-type symbol diagrams: {"Category":"Symbol Diagram","Symbol Diagram Type":"Array","Pin Number":{"Row":"Specific Value","Column":"Specific Value"},"Pin Assignment":{"Serial Number 1":"Name 1","Serial Number 2":"Name 2",...}};

[0129] Two-column / four-sided symbol diagrams directly label the pin numbers on the diagram, while array diagrams label the rows and columns. Row and column labels have two forms: row labels are a letter sequence & column labels are a number sequence, or row labels are a number sequence & column labels are a letter sequence. Array-style pin numbers use a combination of letter and number labels. Pin numbers and pin labels record spatial location information from top to bottom and from left to right.

[0130] The symbol table does not contain pin location information, but it can contain multiple pin assignments. Its notation method is as follows:

[0131] [{"Name":"Name1","Pin Assignment":{"Serial Number 1":"Name1","Serial Number 2":"Name2",...}},

[0132] {"Name":"Name2","Pin Assignment":{"Serial Number 1":"Name1","Serial Number 2":"Name2",...}},...]

[0133] The name is usually related to the device's package type.

[0134] 4. Combined training:

[0135] After completing the data preparation in steps 1-3 above, the three steps can be combined for training, which was not possible with previous methods. Combined training can reduce the management difficulty of multiple models, reduce the GPU memory usage during inference, and improve the capabilities of similar tasks.

[0136] The following provides a detailed explanation of the schematic symbol generation method in this embodiment based on a specific PDF. For the overall processing flow, please refer to [link / reference needed]. Figure 1 :

[0137] Target extraction description: See Figure 2 The PDF sample contains symbol diagrams for three different encapsulation types: two-column, four-column, and array. Additionally, a symbol table at the bottom of the page summarizes the PinAssignments for all three encapsulation types.

[0138] Step 1, Pin Assignment information page number retrieval

[0139] 1.1) Use a PDF tool (such as pymupdf, pdf2image) to convert the PDF to an image using a dpi value of 1. Since different dpi values ​​will be used multiple times to convert the image, a dpi value of 1 is used here for differentiation. When retrieving page numbers for Pin Assignment information, a higher resolution is not required, so the dpi value of 1 can be set lower to improve efficiency.

[0140] 1.2) Input the image obtained in step 1.1 into the VLM to determine the page number containing Pin Assignment information; there are two methods: sequentially check if a single image contains Pin Assignment information; or input multiple images into the VLM and determine which image contains the Pin Assignment information; both methods are supported if they have been trained; after step 1, page number 7 is obtained as shown in the sample example, see [link to sample example]. Figure 2 .

[0141] Step 2, Pin Assignment Target Region Detection

[0142] 2.1) When performing target region detection, the input image usually needs to be clear. Therefore, a dpi value of 2 is used to convert page 7 of the PDF into an image. (dpi value 2 >= dpi value 1)

[0143] 2.2) Input the image obtained in step 2.1 into VLM to obtain the region result of the target area. An example of the result is shown below. Figure 3 As shown in the figure, boxes 1, 2, 3, 4, sub1, sub2, sub3, and sub4 in Figure 3 are used to illustrate and represent the position and size; the position and size parameters are (x, y, w, h), which are the center coordinates and width and height of the target; it should be noted that some systems use (x1, y1, x2, y2), which are the upper left and lower right corners of the target; this patent only uses the first method for example illustration.

[0144] The target detection formatted parameters for the example sample are output as follows:

[0145] [{"Location Size":"box1","Category":"Symbol Diagram","Symbol Diagram Type":"Two-Column","Package Type":"TSSOP","Device Name":"ADG3308"},

[0146] {"Size":"box2","Category":"Symbol Image","Symbol Image Type":"Surround","Pin Area":{"Top":{"Size":"sub1","Character Orientation":"Left"},"Bottom":{"Size":"sub3","Character Orientation":"Left"},"Left":{"Size":"sub4","Character Orientation":"Top"},"Right":{"Size":"sub2","Character Orientation":"Top"}},"Package Type":"LFCSP","Device Name":"ADG3308"},

[0147] {"Location Size":"box3","Category":"Symbol Diagram","Symbol Diagram Type":"Array","Package Type":"WLCSP","Device Name":"ADG3308-1"},

[0148] {"Location Size":"box4","Category":"Symbol List","Package Type":"TSSOP,WLCSP,LFCSP","Device Name":"ADG3308,ADG3308,ADG3308-1"}];

[0149] The results displayed by the JSON formatting tool are as follows: Figure 4 As shown.

[0150] Step 3, Pin Assignment Information Extraction and Preprocessing

[0151] 3.1) Automatic resolution adjustment: As can be seen from the sample examples, the page occupancy of box1, box2, and box3 is small, and the text is not clear enough; while the width of box4 is close to the page width, and the text is relatively clear; assuming that the required resolution is: the larger side is no greater than 1280 and the smaller side is no less than 640, then the target scaling ratio for each symbol image is: Rate = min(1280 / max(w,h), 640 / min(w,h)); the scaling ratio for the symbol table area does not need to be calculated. For the sake of subsequent unified description, the scaling ratio Rate for the symbol table area is set to 1.

[0152] 3.2) Target Region Cropping: Based on the formula dpi-value 3 = Rate * dpi-value 2, re-export the page for each target region using a tool (such as pymupdf, pdf2image), and then crop the target region according to the target detection results. Common tools like OpenCV can achieve this. Specifically, it is also necessary to crop the pin regions for the four-sided symbol images and place them according to the orientation of the pin names; this can also be done using tools like OpenCV.

[0153] 3.3) Pin assignment information extraction: For the cropped target area image, including the symbol image body and the aligned pin area sub-image, input the VLM to extract the pin assignment.

[0154] 3.31) Box1 region extraction, the VLM input image is as follows: Figure 5 As shown, VLM is based on Figure 5 The output formatting parameter list is as follows:

[0155] {"Category":"Symbol Diagram","Symbol Diagram Type":"Two-Column","Pin Number":{"Top":"","Bottom":"","Left":"1,2,3,4,5,6,7,8,9,10","Right":"20,19,18,17,16,15,14,13,12,11"},"Pin Assignment":{"1":"V_CCA","2":"A1",...,"19":"Y1","20":"V_CCY"}}.

[0156] 3.32) Box 2 region extraction, the VLM input image is as follows: Figure 6 As shown, VLM is based on Figure 6 The output formatting parameter list is as follows:

[0157] {"Category":"Symbol Diagram","Symbol Diagram Type":"Square","Pin Number":{"Top":"20,19,18,17,16","Bottom":"6,7,8,9,10","Left":"1,2,3,4,5","Right":"15,14,13,12,11"},"Pin Assignment":{"1":"A2","2":"A3",...,"19":"V_CCA","20":"A1"}}.

[0158] 3.33) Box 3 region extraction, the VLM input image is as follows: Figure 7 As shown, VLM is based on Figure 7 The output formatting parameter list is as follows: The VLM output formatting parameter list is as follows:

[0159] {"Category":"Symbol Diagram","Symbol Diagram Type":"Array","Pin Labels":{"Rows":"1,2,3,4","Columns":"a,b,c,d,e"},"Pin Assignments":{"a1":"V_CCY","a2":"Y1",...,"e3":"EN","e4":"A8"}}.

[0160] 3.34) Box 4 region extraction, the VLM input image is as follows: Figure 8 As shown, VLM is based on Figure 8 The output formatting parameter list is as follows: The VLM output formatting parameter list is as follows:

[0161] [{"Name":"TSSOP","Pin Assignment":{"1":"V_CCA","2":"A1",...}},

[0162] {"Name":"LFCSP","Pin Assignment":{"19":"V_CCA","20":"A1",...}},

[0163] {"Name":"WLCSP","Pin Assignment":{"a4":"V_CCA","a3":"A1",...}}].

[0164] 3.35) Write the above information into the target area detection result with "Detailed Information" as the key, and also write the page number as the final formatted output parameter:

[0165] [{"Page Number":"7","Position Size":"box1","Category":"Symbol Diagram","Symbol Diagram Type":"Two Columns","Package Type":"TSSOP","Device Name":"ADG3308","Details":{"Category":"Symbol Diagram","Symbol Diagram Type":"Two Columns","Pin Number":{"Top":"","Bottom":"","Left":"1,2,3,4,5,6,7,8,9,10","Right":"20,19,18,17,16,15,14,13,12,11"},"Pin Assignment":{"1":"V_CCA","2":"A1",...,"19":"Y1","20":"V_CCY"}}},

[0166] {"Page Number":"7","Position Size":"box2","Category":"Symbol Image","Symbol Image Type":"Square","Pin Area":{"Top":{"Position Size":"sub1","Character Orientation":"Left"},"Bottom":{"Position Size":"sub3","Character Orientation":"Left"},"Left":{"Position Size":"sub4","Character Orientation":"Top"},"Right":{"Position Size":"sub2","Character Orientation":"Top"}},"Package Type":"LFC"} SP", "Device Name": "ADG3308", "Details": {"Category": "Symbol Diagram", "Symbol Diagram Type": "Surround Type", "Pin Numbers": {"Top": "20, 19, 18, 17, 16", "Bottom": "6, 7, 8, 9, 10", "Left": "1, 2, 3, 4, 5", "Right": "15, 14, 13, 12, 11"},"Pin Assignments": {"1": "A2", "2": "A3", ...,"19": "V_CCA", "20": "A1"}}},

[0167] {"Page Number":"7","Position Size":"box3","Category":"Symbol Diagram","Symbol Diagram Type":"Array","Package Type":"WLCSP","Device Name":"ADG3308-1","Details":{"Category":"Symbol Diagram","Symbol Diagram Type":"Array","Pin Number":{"Row":"1,2,3,4","Column":"a,b,c,d,e"},"Pin Assignment":{"a1":"V_CCY","a2":"Y1",...,"e3":"EN","e4":"A8"}}},

[0168] {"Page Number":"7","Position Size":"box4","Category":"Symbol List","Package Type":"TSSOP,WLCSP,LFCSP","Device Name":"ADG3308,ADG3308,ADG3308-1","Details":[{"Name":"TSSOP","Pin Assignment":{"1":"V_CCA","2":"A1",...}},{"Name":"LFCSP","Pin Assignment":{"19":"V_CCA","20":"A1",...}},{"Name":"WLCSP","Pin Assignment":{"a4":"V_CCA","a3":"A1",...}}]}].

[0169] The results displayed using the JSON formatting tool are as follows: Figure 9 As shown.

[0170] Example 3: This application provides a VLM-based automatic schematic symbol generation system, applied to the VLM-based automatic schematic symbol generation method of Example 1, including:

[0171] The parameter acquisition module is used to input the information to be processed into the trained visual language model for processing, and obtain the formatted parameters corresponding to the information to be processed.

[0172] The symbol generation module is used to import formatting parameters into the EDA tool for processing, and obtain the symbol diagram created by the EDA tool and used to draw the schematic diagram.

[0173] Specifically, the process of inputting the information to be processed into the trained visual language model involves:

[0174] Obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, use a general VLM to check whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed;

[0175] Target region detection is performed on the image corresponding to the PDF page number, and the target image corresponding to the information to be processed is obtained. At the same time, the symbol macro information corresponding to the target image is extracted. The symbol macro information includes at least category, symbol image type, location information, package type, and device name.

[0176] From the target image after resolution adaptive adjustment and the pin region sub-image in the target image after text direction correction, the symbol details of the information to be processed, namely the pin number or pin sequence number and the corresponding pin name, are extracted in one go. The macroscopic symbol information and symbol details are integrated to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, where:

[0177] To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped.

[0178] When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details.

[0179] The formatting parameters are generated and output using PDF page numbers and symbol information.

[0180] Optionally, the trained visual language model described above is obtained in the following way:

[0181] The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

[0182] The training samples mentioned above were obtained in the following way:

[0183] Step 8.1, Construction of page number retrieval training samples:

[0184] Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information;

[0185] Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information;

[0186] For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained".

[0187] Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image.

[0188] Step 8.2, Construction of training samples for target region detection and information extraction:

[0189] The training samples for this step are constructed using the positive sample set from step 8.1, and labeled with the target region location and size and target region information. The target region information is the macroscopic information of the symbol, including: category, package type, device name, pin region location and size, pin region text orientation, etc. A single sample usually contains multiple target regions.

[0190] Step 8.3, extracting symbol details and constructing training samples:

[0191] Similarly, the positive sample set from step 8.1 is used to construct the training samples for this step. Based on the annotation information from step 8.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name.

[0192] Step 8.4, Merging training samples:

[0193] The training sets of steps 8.1, 8.2, and 8.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

[0194] Example 4: This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method of Example 1 or Example 2.

[0195] Example 5: This application provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the method of Example 1 or Example 2.

[0196] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0201] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically generating schematic symbols based on VLM, characterized in that, The specific steps include the following: The information to be processed is input into the trained visual language model for processing, and the formatted parameters corresponding to the information to be processed are obtained. The formatting parameters are imported into the EDA tool for processing to obtain a symbol diagram created by the EDA tool and used to draw the schematic diagram.

2. The method for automatic generation of schematic symbols based on VLM according to claim 1, characterized in that, The process of inputting the information to be processed into the trained visual language model for processing specifically involves: Obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, use a general VLM to check whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed; Target region detection is performed on the image corresponding to the PDF page number to obtain the target image corresponding to the information to be processed, and the symbol macro information corresponding to the target image is extracted at the same time. The symbol macro information includes at least category, symbol image type, location information, package type, and device name. From the target image after resolution adaptive adjustment and the pin region sub-image after text direction correction in the target image, the symbol details of the information to be processed, namely the pin number or pin sequence number and the corresponding pin name, are extracted in one go. The macroscopic symbol information and symbol details are integrated to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, wherein: To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped. When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details. The formatting parameters are generated and output using the PDF page numbers and symbol information.

3. The method for automatic generation of schematic symbols based on VLM according to claim 1, characterized in that, The trained visual language model is obtained through the following method: The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

4. The method for automatic generation of schematic symbols based on VLM according to claim 3, characterized in that, The training samples were obtained in the following way: Step 4.1, Construction of page number retrieval training samples: Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information; Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information; For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained". Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image. Step 4.2, Construction of training samples for target region detection and information extraction: The training samples for this step are constructed using the positive sample set from step 4.1, and labeled with the target region location and size and target region information; Target area information is essentially macroscopic symbol information, including: category, package type, device name, pin area location and size, pin area text orientation, etc.; a single sample typically contains multiple target areas; Step 4.3, extracting symbol details from training samples for construction: Similarly, the positive sample set from step 4.1 is used to construct the training samples for this step. Based on the annotation information from step 4.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name. Step 4.4, Merging training samples: The training sets of steps 4.1, 4.2, and 4.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

5. A schematic symbol automatic generation system based on VLM, characterized in that, include: The parameter acquisition module is used to input the information to be processed into the trained visual language model for processing, and obtain the formatted parameters corresponding to the information to be processed. The symbol diagram generation module is used to import the formatting parameters into the EDA tool for processing, and obtain the symbol diagram created by the EDA tool and used to draw the schematic diagram.

6. The VLM-based automatic schematic symbol generation system according to claim 5, characterized in that, The process of inputting the information to be processed into the trained visual language model for processing specifically involves: Obtain the PDF page number of the information to be processed in the PDF manual, wherein: convert the PDF to an image, use a general VLM to check whether each image contains the information to be processed, or input multiple images into the VLM at the same time to determine which image contains the information to be processed; Target region detection is performed on the image corresponding to the PDF page number to obtain the target image corresponding to the information to be processed, and the symbol macro information corresponding to the target image is extracted at the same time. The symbol macro information includes at least category, symbol image type, location information, package type, and device name. From the target image after resolution adaptive adjustment and the pin region sub-image after text direction correction in the target image, the symbol details of the information to be processed, namely the pin number or pin sequence number and the corresponding pin name, are extracted in one go. The macroscopic symbol information and symbol details are integrated to obtain the final symbol information. The symbol information includes at least the category, symbol diagram type, location information, package type, device name, pin number or pin sequence number, and pin name, wherein: To extract detailed symbol information from the information to be processed, it is necessary to calculate the dpi parameter when converting PDF to image based on the target image size and the expected target size. Then, the page image is re-exported using the calculated dpi parameter, and the target image with guaranteed clarity is cropped. When there is a pin area sub-image in the target image where the text orientation is not correct, the pin area sub-image needs to be cropped out and corrected; then the target image and the corrected pin area sub-image are simultaneously input into VLM to obtain symbol details. The formatting parameters are generated and output using the PDF page numbers and symbol information.

7. The VLM-based automatic schematic symbol generation system according to claim 5, characterized in that, The trained visual language model is obtained through the following method: The pre-trained visual language model is trained using completed training samples until it reaches the training termination condition. The visual language model that reaches the training termination condition is then defined as the trained visual language model.

8. The VLM-based automatic schematic symbol generation system according to claim 7, characterized in that, The training samples were obtained in the following way: Step 8.1, Construction of page number retrieval training samples: Positive sample set, which is a set of images converted from PDF pages containing Pin Assignment information; Negative sample set: a set of images converted from PDF pages that do not contain Pin Assignment information; For single image samples, a binary classification training method is used, with positive samples being "contained" and negative samples being "not contained". Specifically, multiple images are randomly sampled from the positive and negative samples and their order is shuffled to form a multi-image training sample, which is labeled with the sequence number of the positive sample image. Step 8.2, Construction of training samples for target region detection and information extraction: The training samples for this step are constructed using the positive sample set from step 8.1, and labeled with the target region location and size and target region information; Target area information is essentially macroscopic symbol information, including: category, package type, device name, pin area location and size, pin area text orientation, etc.; a single sample typically contains multiple target areas; Step 8.3, extracting symbol details from training samples for construction: Similarly, the positive sample set from step 8.1 is used to construct the training samples for this step. Based on the annotation information from step 8.2, the target region image is cropped. If the text in the pin region is not facing upwards, the pin region image needs to be cropped and straightened. The target image and the possible straightened pin region image are used as a sample, and its annotation is the symbol details, including: pin number or pin sequence number, pin name. Step 8.4, Merging training samples: The training sets of steps 8.1, 8.2, and 8.3 are combined, and VLM is trained simultaneously to support different task capabilities of the multi-task system.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of any one of claims 1-4.

Images