Code generation method and device based on circuit diagram and electronic equipment

Abstract

Embodiments of the invention provide a circuit diagram-based code generation method and apparatus, and an electronic device. The method comprises the steps of obtaining an image of a to-be-processed circuit diagram; processing the image of the circuit diagram to obtain a circuit netlist and a netlist confidence coefficient corresponding to the circuit diagram; wherein the netlist confidence is determined according to basic components of the circuit diagram extracted from the circuit diagram; and generating a circuit code corresponding to the circuit diagram through a code generation model according to the image of the circuit diagram, the circuit netlist and the netlist confidence coefficient. In this way, it is considered that the circuit netlist and the confidence coefficient can represent the characteristics of the circuit diagram, and therefore the accuracy of the circuit code generated through the code generation model can be higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuits, and in particular to a code generation method and device based on a circuit diagram and an electronic device. BACKGROUND

[0002] In the process of integrated circuit design, a technician can manually write corresponding circuit code based on a circuit diagram, so as to facilitate subsequent simulation and other operations on the circuit based on the circuit code. However, the efficiency of manually writing the circuit code is low.

[0003] At present, a circuit diagram image can be directly input into a large model, so that the large model generates circuit code in combination with an input prompt word. This can effectively improve the efficiency of generating circuit code.

[0004] However, the large model can generate incorrect circuit code due to image blur, component overlap, or text recognition error, so that the accuracy of the generated circuit code is low. SUMMARY

[0005] Embodiments of the present application provide a code generation method and device based on a circuit diagram and an electronic device, to improve the accuracy of the generated circuit code.

[0006] In a first aspect, an embodiment of the present application provides a code generation method based on a circuit diagram, comprising:

[0007] obtaining an image of a circuit diagram to be processed;

[0008] processing the image of the circuit diagram to obtain a circuit netlist corresponding to the circuit diagram and a netlist confidence; wherein the netlist confidence is determined according to basic constituent elements of the circuit diagram extracted in the circuit diagram;

[0009] inputting the image of the circuit diagram, the circuit netlist, and the netlist confidence into a code generation model, and generating, by the code generation model, circuit code corresponding to the circuit diagram according to the image of the circuit diagram, the circuit netlist, and the netlist confidence.

[0010] In a possible implementation, the generating, by the code generation model, of the circuit code corresponding to the circuit diagram according to the image of the circuit diagram, the circuit netlist, and the netlist confidence comprises:

[0011] performing visual feature recognition on the image of the circuit diagram by a feature extraction module in the code generation model to obtain image features corresponding to the image of the circuit diagram;

[0012] generating, by the code generation model, the circuit code corresponding to the circuit diagram according to the image features, the circuit netlist, and the netlist confidence.

[0013] In a possible implementation, the code generation model comprises a multi-modal large model.

[0014] The generating, by the code generation model, of the circuit code corresponding to the circuit diagram according to the image feature, the circuit netlist, and the netlist confidence comprises:

[0015] The generating, by the multi-modal large model, of the circuit code corresponding to the circuit diagram according to the image feature, the circuit netlist, the netlist confidence, and a prompt word comprises:

[0016] The attention degree of the multi-modal large model to the circuit netlist is proportional to the netlist confidence, and the attention degree of the multi-modal large model to the image feature is inversely proportional to the netlist confidence when the multi-modal large model generates the circuit code.

[0017] In a possible implementation, the generating, by the multi-modal large model, of the circuit code corresponding to the circuit diagram according to the image feature, the circuit netlist, the netlist confidence, and the prompt word comprises:

[0018] The multi-modal large model processes the image feature, the circuit netlist, and the netlist confidence according to the prompt word to generate the circuit code corresponding to the circuit diagram.

[0019] The multi-modal large model comprises a self-attention layer and a multi-cross attention layer.

[0020] The self-attention layer is configured to process the generated circuit code.

[0021] The multi-cross attention layer is configured to calculate attention outputs of a first attention head and a second attention head in parallel, and determine an attention degree to the circuit netlist and an attention degree to the image feature according to the attention outputs, wherein the attention outputs are determined according to the netlist confidence, the image feature, and a feature of the circuit netlist.

[0022] In a possible implementation, the processing of the image of the circuit diagram to obtain the circuit netlist and the netlist confidence corresponding to the circuit diagram comprises:

[0023] The image of the circuit diagram is processed to obtain component information, connection line information, and text information in the circuit diagram.

[0024] The circuit netlist and the netlist confidence corresponding to the circuit diagram are determined according to the component information, the connection line information, and the text information.

[0025] In one possible implementation, the component information includes component data and component confidence, the connection information includes connection lines and connectivity confidence, and the text information includes text data and text confidence.

[0026] The process of processing the circuit diagram image to obtain component information, connection information, and text information in the circuit diagram includes:

[0027] Target detection is performed on the image of the circuit diagram. The component data in the circuit diagram is determined based on the intersection-union ratio of the recognition results. The component confidence is determined based on the first recognition success rate of the recognition results. The component confidence is proportional to the first recognition success rate.

[0028] The connection lines in the circuit diagram image are identified using a segmentation model to obtain the connection line data in the circuit diagram, and the connectivity confidence is determined based on the second recognition success rate of the recognition results; the connectivity confidence is proportional to the second recognition success rate.

[0029] Text recognition is performed on the image of the circuit diagram to obtain the text data and text confidence level in the circuit diagram.

[0030] In one possible implementation, the text confidence score includes at least one of character confidence score and word confidence score; the character confidence score is the confidence score of a single character in the text data, and the word confidence score is determined based on the confidence scores of all characters that make up the word or the word semantics.

[0031] In one possible implementation, determining the circuit netlist corresponding to the circuit diagram based on the component information, the connection information, and the text information includes:

[0032] The component information, the connection line information, and the text information are fused together to extract the attribute information of each component data.

[0033] Based on the component information, the connection line information, the text information, and the attribute information of each component data, the circuit netlist corresponding to the circuit diagram is determined.

[0034] In one possible implementation, determining the netlist confidence level corresponding to the circuit diagram based on the component information, the connection information, and the text information includes:

[0035] The netlist confidence level corresponding to the circuit diagram is determined based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram.

[0036] In one possible implementation, determining the netlist confidence level corresponding to the circuit diagram based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram includes:

[0037] The circuit netlist, the component confidence, the connectivity confidence, the text confidence, and the image features corresponding to the circuit diagram are input into the multimodal large model;

[0038] Using the multimodal large model, the netlist confidence level corresponding to the circuit diagram is determined based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram.

[0039] Secondly, embodiments of this application provide a circuit diagram-based code generation apparatus, comprising:

[0040] The acquisition module is used to acquire images of the circuit diagram to be processed;

[0041] The processing module is used to process the image of the circuit diagram to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram; wherein, the netlist confidence level is determined based on the basic components of the circuit diagram extracted from the circuit diagram;

[0042] The generation module is used to generate circuit code corresponding to the circuit diagram based on the image of the circuit diagram, the circuit netlist, and the confidence level of the netlist using a code generation model.

[0043] In one possible implementation, the generation module is specifically used to perform visual feature recognition on the image of the circuit diagram through the feature extraction module in the code generation model to obtain the image features corresponding to the image of the circuit diagram; and to generate the circuit code corresponding to the circuit diagram through the code generation model based on the image features, the circuit netlist, and the netlist confidence.

[0044] In one possible implementation, the code generation model includes a multimodal large model.

[0045] The generation module is specifically used to generate circuit code corresponding to the circuit diagram using the multimodal large model based on the image features, the circuit netlist, the netlist confidence level, and cue words. The cue words instruct the multimodal large model to generate the circuit code based on the acquired image features, circuit netlist, and netlist confidence level. The attention the multimodal large model pays to the circuit netlist when generating the circuit code is directly proportional to the netlist confidence level, and the attention it pays to the image features is inversely proportional to the netlist confidence level.

[0046] In one possible implementation, the generation module is specifically used by the multimodal large model to process the image features, the circuit netlist, and the netlist confidence based on the prompt words, to generate the circuit code corresponding to the circuit diagram. The multimodal large model includes a self-attention layer and multiple cross-attention layers.

[0047] The self-attention layer is used to process the generated circuit code;

[0048] The multiple cross-attention layer is used to compute the attention outputs of the first attention head and the second attention head in parallel, and to determine the degree of attention to the circuit netlist and the degree of attention to the image features based on the attention outputs; wherein, the attention outputs are determined based on the netlist confidence, the image features, and the features of the circuit netlist.

[0049] In one possible implementation, the processing module is specifically used to process the image of the circuit diagram to obtain component information, connection line information, and text information in the circuit diagram; and to determine the circuit netlist and netlist confidence level corresponding to the circuit diagram based on the component information, the connection line information, and the text information.

[0050] In one possible implementation, the component information includes component data and component confidence, the connection information includes connection lines and connectivity confidence, and the text information includes text data and text confidence.

[0051] The processing module is specifically used to perform target detection on the image of the circuit diagram, determine the component data in the circuit diagram based on the intersection-union ratio of the recognition results, and determine the component confidence based on the first recognition success rate of the recognition results; the component confidence is proportional to the first recognition success rate; identify the connecting lines in the image of the circuit diagram through a segmentation model to obtain the connecting line data in the circuit diagram, and determine the connectivity confidence based on the second recognition success rate of the recognition results; the connectivity confidence is proportional to the second recognition success rate; and perform text recognition on the image of the circuit diagram to obtain the text data and text confidence in the circuit diagram.

[0052] In one possible implementation, the text confidence score includes at least one of character confidence score and word confidence score; the character confidence score is the confidence score of a single character in the text data, and the word confidence score is determined based on the confidence scores of all characters that make up the word or the word semantics.

[0053] In one possible implementation, the processing module is specifically used to perform fusion processing on the component information, the connection line information, and the text information to extract the attribute information of each component data; and to determine the circuit netlist corresponding to the circuit diagram based on the component information, the connection line information, the text information, and the attribute information of each component data.

[0054] In one possible implementation, the processing module is specifically configured to determine the netlist confidence level corresponding to the circuit diagram based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram.

[0055] In one possible implementation, the processing module is specifically configured to input the circuit netlist, the component confidence, the connectivity confidence, the text confidence, and the image features corresponding to the image of the circuit diagram into a multimodal large model; and through the multimodal large model, determine the netlist confidence corresponding to the circuit diagram based on the image features corresponding to the circuit netlist, the component confidence, the connectivity confidence, the text confidence, and the image features corresponding to the image of the circuit diagram.

[0056] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0057] The memory stores computer-executed instructions;

[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0060] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0061] The circuit diagram-based code generation method, apparatus, and electronic device provided in this application acquire an image of a circuit diagram to be processed; process the image of the circuit diagram to obtain a circuit netlist and netlist confidence level corresponding to the circuit diagram; wherein, the netlist confidence level is determined based on the basic constituent elements of the circuit diagram extracted from the circuit diagram; and generate circuit code corresponding to the circuit diagram using a code generation model based on the image of the circuit diagram, the circuit netlist, and the netlist confidence level. In this way, considering that the circuit netlist and confidence level can characterize the features of the circuit diagram, the accuracy of the circuit code generated by the code generation model can be improved. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 A schematic diagram illustrating the framework of a circuit diagram-based code generation method provided in this application embodiment;

[0064] Figure 2 A schematic flowchart illustrating a code generation method based on a circuit diagram provided in this application embodiment;

[0065] Figure 3 A schematic diagram of a circuit diagram provided in an embodiment of this application;

[0066] Figure 4 A flowchart illustrating a method for obtaining a circuit netlist and netlist confidence level, provided for an embodiment of this application;

[0067] Figure 5 A schematic diagram of an architecture for determining netlist confidence is provided for an embodiment of this application;

[0068] Figure 6 A flowchart illustrating a method for generating circuit code using a code generation model, provided in an embodiment of this application;

[0069] Figure 7 This application provides a schematic diagram of the overall framework of a code generation system according to an embodiment of the present application.

[0070] Figure 8 A schematic diagram of a circuit diagram-based code generation device provided in this application embodiment;

[0071] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0074] First, let me explain the terms used in this application:

[0075] HDL: Hardware Description Language, a programming language that uses text to describe the structure and behavior of digital or analog circuits.

[0076] Verilog-A: Verilog Analog, a Verilog-based analog / mixed-signal hardware description language (HDL) used for modeling, simulating, and verifying analog circuits, RF circuits, and mixed-signal systems at the transistor or behavioral level. It can be directly used for functional verification in SPICE-like simulators (such as Spectre and HSPICE) and is a code format supported by this application.

[0077] OCR: Optical Character Recognition, a technology that automatically converts text regions in an image into editable text.

[0078] CV: Computer Vision, is a technical field that studies how computers can automatically "understand" the real world from images or videos, covering tasks such as image classification, object detection, semantic segmentation, and OCR. In this application, the CV module generally refers to the module responsible for visual processing flows such as component recognition, connection segmentation, and text detection of circuit diagram images.

[0079] In the integrated circuit design process, technicians can manually write corresponding circuit code based on circuit diagrams to facilitate subsequent operations such as circuit simulation. However, manually writing circuit code is inefficient.

[0080] Currently, circuit diagram images can be directly input into large models, allowing the large models to generate circuit codes based on the input prompts. This effectively improves the efficiency of generating circuit codes.

[0081] However, large models may generate incorrect circuit codes due to image blurring, component overlap, or text recognition errors, resulting in low accuracy of the generated circuit codes. Furthermore, manual modification of the circuit codes may be required afterward, making end-to-end automated processing impossible.

[0082] Based on this, embodiments of this application provide a code generation method based on circuit diagrams. When generating circuit code corresponding to a circuit diagram using a code generation model, not only is the circuit diagram input into the model, but also the circuit netlist determined based on the circuit diagram image and the netlist confidence level are input into the code generation model. The code generation model generates the circuit code corresponding to the circuit diagram based on the circuit diagram image, the circuit netlist, and the netlist confidence level. In this way, the code generation model can generate circuit code based on more relevant circuit information, improving the accuracy of the generated circuit code, reducing the need for subsequent manual corrections, and enabling end-to-end automated processing.

[0083] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0084] Figure 1 This is a schematic diagram illustrating the framework of a circuit diagram-based code generation method provided in an embodiment of this application.

[0085] like Figure 1 As shown, the image of the circuit diagram to be processed can be processed to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram. The circuit netlist, netlist confidence level and the image of the circuit diagram are all input into the code generation model, so that the code generation model can generate the circuit code corresponding to the circuit diagram based on the circuit netlist, netlist confidence level and the image of the circuit diagram.

[0086] For details, please refer to the following: Figure 2 Related descriptions.

[0087] Figure 2 This application provides a flowchart illustrating a code generation method based on a circuit diagram, as shown in the embodiments below. Figure 2 As shown, the method includes

[0088] S201. Obtain the image of the circuit diagram to be processed.

[0089] The obtained circuit diagram image can be input by the user or obtained from a database or the Internet. This application embodiment does not specifically limit the source of the circuit diagram image.

[0090] Figure 3 This is a schematic diagram of a circuit diagram provided in an embodiment of this application.

[0091] like Figure 3 As shown, circuit diagrams typically include components, connecting lines, and text data. This text data describes information about the components and the corresponding connections between the connecting lines.

[0092] It should be noted that the text in the obtained circuit diagram image is usually in English, for example, Figure 4 The text in the text.

[0093] The embodiments in this application are only for reference. Figure 3 The circuit diagram shown is used as an example for illustration and does not constitute any limitation.

[0094] S202. Process the circuit diagram image to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram.

[0095] The netlist confidence level is determined based on the basic components of the circuit diagram extracted from the circuit diagram.

[0096] The basic components of a circuit diagram include components, connecting lines, and text data.

[0097] In this application, the circuit diagram image can be processed to identify components, connecting lines and text data in the circuit diagram, and further, a corresponding circuit netlist and netlist confidence level can be generated based on the identified information.

[0098] S203. Input the circuit diagram image, circuit netlist, and netlist confidence level into the code generation model. The code generation model generates the circuit code corresponding to the circuit diagram based on the circuit diagram image, circuit netlist, and netlist confidence level.

[0099] In this application, the code generation model can be a multimodal large model or a pre-trained model. The embodiments of this application do not specifically limit the code generation model.

[0100] The generated circuit code can be HDL code, such as one or more of Verilog code, Verilog-A code, VHDL code, System Verilog code, etc. This application embodiment does not specifically limit the circuit code.

[0101] The circuit diagram-based code generation method provided in this application, when generating circuit code corresponding to a circuit diagram using a code generation model, not only inputs the circuit diagram into the model, but also inputs the circuit netlist determined based on the circuit diagram image and the netlist confidence level into the code generation model. The code generation model generates the circuit code corresponding to the circuit diagram based on the circuit diagram image, the circuit netlist, and the netlist confidence level. In this way, the code generation model can generate circuit code based on more relevant circuit information, improving the accuracy of the generated circuit code, reducing the need for subsequent manual correction, and enabling end-to-end automated processing.

[0102] The methods for obtaining the circuit netlist and netlist confidence corresponding to the circuit diagram, as well as the method for generating circuit code through the code generation model, will be explained below.

[0103] Figure 4 A flowchart illustrating a method for obtaining a circuit netlist and netlist confidence level, provided in an embodiment of this application, is shown below. Figure 4 As shown, in this embodiment... Figure 2 Based on the embodiments, the method for obtaining the circuit netlist and netlist confidence is described in detail. The method includes:

[0104] S401. Process the circuit diagram image to obtain component information, connection line information and text information in the circuit diagram.

[0105] In this application, component information includes component data and component confidence, connection information includes connection lines and connectivity confidence, and text information includes text data and text confidence.

[0106] Processing circuit diagram images to obtain component information, connection information, and text information can include: performing object detection on the circuit diagram image, determining component data in the circuit diagram based on the intersection-over-union ratio of the recognition results, and determining component confidence based on the recognition success rate; identifying connection lines in the circuit diagram image using a segmentation model to obtain connection line data in the circuit diagram, and determining connectivity confidence based on the accuracy of the recognition results; and performing text recognition on the circuit diagram image to obtain text data and text confidence in the circuit diagram.

[0107] Among them, component confidence is directly proportional to recognition success rate. Connectivity confidence is directly proportional to accuracy.

[0108] For example, component data may include components and data such as the location of components. This application embodiment does not limit the specific content of component data.

[0109] Determining component data and component confidence can specifically include: using object detection algorithms such as YOLO11 or other object detection models to perform object detection on the circuit diagram image and identify components in the circuit diagram image. Specifically, when detecting components, the object detection algorithm can obtain data such as the predicted bounding box and position of the component.

[0110] Furthermore, the cross-union ratio (CUP) between the predicted bounding boxes and the pre-stored or pre-labeled ground truth bounding boxes is calculated, and the identified components and their confidence levels are determined based on the CUP.

[0111] Determining the identified component data and component confidence level based on the cross-connection-to-union ratio (CTU) can include: for each component's corresponding CTU, if the CTU is greater than or equal to a preset CTU, the component is determined to be a successfully identified component; if the CTU is less than the preset CTU, the component is determined to be a failed identified component. The component data corresponding to the successfully identified components is then determined as the component data in the circuit diagram.

[0112] The embodiments of this application do not limit the specific value of the preset crossover-union ratio.

[0113] Furthermore, based on the number of successfully identified components and the number of unsuccessfully identified components, a first identification success rate is determined, and the component confidence level is determined based on this first identification success rate.

[0114] For example, when considering the number of successfully identified components and the number of failed identified components, taking the number of successfully identified components as S1 and the number of failed identified components as F1, and assuming that the first identification success rate is the same as the component confidence level, the identification success rate can be determined as Ccomp = S1 / (S1 + F1).

[0115] In this way, the confidence level of a component is determined based on the proportion of the number of successfully identified components to the total number of identified components, so that the determined confidence level can characterize the success rate of component identification.

[0116] For example, identifying connection lines in a circuit diagram image using a segmentation model to obtain connection line data in the circuit diagram, and determining connectivity confidence based on the accuracy of the identification results, may include: using a semantic segmentation model (e.g., UNet++, which is not limited in this embodiment) to identify connection lines in the circuit diagram image; determining whether connection line identification was successful based on the direction of the connection line and the corresponding component data; identifying successfully identified connection lines and unsuccessfully identified connection lines; and determining the connection line data corresponding to the successfully identified connection lines as the connection line data in the circuit diagram.

[0117] Furthermore, based on the number of successfully identified connections and the number of unsuccessfully identified connections, a second identification success rate is determined, and a connectivity confidence level is determined based on this second identification success rate.

[0118] For example, when considering the number of successfully identified connections and the number of failed connections, taking the number of successfully identified connections as S2 and the number of failed connections as F2, and assuming the second identification success rate is the same as the connectivity confidence, the second identification success rate can be determined as Cconn = S2 / (S2 + F2).

[0119] In this way, by determining the connectivity confidence level by the number of successfully identified connections and the number of failed connections, the determined connectivity confidence level can represent the success rate of connection identification.

[0120] For example, performing text recognition on an image of a circuit diagram to obtain text data and text confidence scores in the circuit diagram may include: using OCR technology to perform text recognition on the image of the circuit diagram, recognizing the text in the circuit diagram, the position of the text in the image of the circuit diagram, and the character confidence score of each character in the text, thereby obtaining text data and text confidence scores.

[0121] In this application, text confidence may include not only character confidence but also word confidence.

[0122] Character confidence is the predicted probability of a character during text recognition. Word confidence is determined based on the confidence of all characters that make up a word or the word's semantics.

[0123] Specifically, word confidence can be derived by combining the comprehensive features of character confidence or word semantics, and by using word plausibility factors to distinguish between valid words and garbled / misspelled text.

[0124] It should be noted that when the text is English, the character confidence score is the confidence score of each letter, and the word confidence score is the confidence score of a single English word composed of the characters. When the text is Chinese, the character confidence score is the confidence score of each Chinese character, and the word confidence score can be the confidence score of a word composed of characters determined by semantics.

[0125] In this way, by determining two types of confidence scores, namely character confidence score and word confidence score, the representation of text confidence scores becomes more comprehensive.

[0126] Therefore, this application determines component data, component confidence levels, connectors, connector confidence levels, text data, and text confidence levels to facilitate the subsequent determination of the circuit netlist and netlist confidence levels. This also ensures that the determined circuit netlist and netlist confidence levels have high accuracy.

[0127] S402. Based on the component information, connection information, and text information, determine the circuit netlist and netlist confidence level corresponding to the circuit diagram.

[0128] In this application, determining the circuit netlist corresponding to the circuit diagram based on component information, connection information, and text information may include: performing fusion processing on component information, connection information, and text information to extract attribute information for each component data; and determining the circuit netlist corresponding to the circuit diagram based on component information, connection information, text information, and attribute information for each component data.

[0129] In this way, based on component information, connection information, text information, and attribute information of each component data, the complete information of the circuit diagram can be obtained, making the generated circuit netlist more accurate, thereby improving the accuracy of the circuit code generated subsequently based on the circuit netlist.

[0130] In this application, determining the netlist confidence level corresponding to the circuit diagram based on component information, connection information, and text information may include: determining the netlist confidence level corresponding to the circuit diagram based on the circuit netlist, component confidence level, connectivity confidence level, text confidence level, and image features corresponding to the image of the circuit diagram.

[0131] The image features corresponding to the circuit diagram image can be identified through image recognition algorithms or models, and this application embodiment does not limit this.

[0132] In this way, by using multiple pieces of information to determine the confidence level of the netlist, the accuracy of the determined confidence level of the netlist is higher.

[0133] One possible implementation is to determine the weights corresponding to each confidence level based on the image features corresponding to the images of the circuit netlist and the circuit diagram, and then perform a weighted calculation on the confidence levels based on the weights corresponding to each level and each confidence level to obtain the netlist confidence level.

[0134] The method for determining the weights corresponding to each confidence level based on the image features corresponding to the circuit netlist and circuit diagram can be based on a pre-defined strategy. For example, the weights for different confidence levels can be determined based on the pre-stored correspondence between the circuit netlist, image features, and weights.

[0135] Another possible implementation, Figure 5 This is a schematic diagram of an architecture for determining netlist confidence, provided as an embodiment of this application.

[0136] like Figure 5As shown, the image features corresponding to the circuit netlist, component confidence, connectivity confidence, text confidence, and circuit diagram image can be input into the multimodal large model. Through the multimodal large model, the netlist confidence corresponding to the circuit diagram is determined based on the image features corresponding to the circuit netlist, component confidence, connectivity confidence, text confidence, and circuit diagram image.

[0137] In this way, the confidence level of the netlist can be determined by using a multimodal large model. Since multimodal large models have strong processing capabilities, the accuracy of the determined netlist confidence level can be improved.

[0138] The method for obtaining a circuit netlist and netlist confidence level provided in this application determines the circuit netlist and netlist confidence level corresponding to the circuit diagram based on component information, connection information, and text information. This results in a higher accuracy of the determined circuit netlist and a more accurate representation of the reliability of the generated netlist. Consequently, the accuracy of the circuit code generated subsequently based on the circuit diagram image, circuit netlist, and netlist confidence level is further improved.

[0139] Figure 6 A flowchart illustrating a method for generating circuit code using a code generation model, as provided in this application embodiment, is shown below. Figure 6 As shown, in this embodiment... Figure 2 Based on the embodiments, the method for generating circuit code through code generation model is described in detail. The method includes:

[0140] S601. Through the feature extraction module in the code generation model, visual feature recognition is performed on the image of the circuit diagram to obtain the image features corresponding to the image of the circuit diagram.

[0141] For example, code generation models can include multimodal large models. For instance, a code generation model could be a ViT (Vision Transformer) encoder-decoder model.

[0142] In the code generation model, the feature extraction module can be an encoder. The encoder can perform visual feature recognition on the image of the circuit diagram to obtain the image features corresponding to the circuit diagram image.

[0143] S602. Based on image features, circuit netlist, and netlist confidence, generate circuit code corresponding to the circuit diagram using a code generation model.

[0144] It should be understood that since image features and circuit netlists need to be input into the code generation model for processing, they can be processed into a language that can be recognized by the code generation model. For example, the circuit netlist can be tokenized into a sequence so that it can be processed by the code generation model.

[0145] In this application, generating circuit code corresponding to a circuit diagram may include: generating circuit code corresponding to a circuit diagram using a multimodal large model based on image features, circuit netlist, netlist confidence level, and prompt words; wherein, prompt words are used to instruct the multimodal large model to generate circuit code based on the acquired image features, circuit netlist, and netlist confidence level.

[0146] Among them, the attention paid to the circuit netlist by the multimodal large model when generating circuit code is directly proportional to the netlist confidence, and the attention paid to image features is inversely proportional to the netlist confidence.

[0147] It should be noted that the multimodal large model here can be the decoder in the encoder-decoder model.

[0148] For example, a multimodal large model can be a pre-trained model. The model training process may include:

[0149] Training data: Inputs are: circuit diagram image, cue words, circuit netlist, netlist confidence; outputs are: circuit code (e.g., Verilog-A code).

[0150] Training: In order for the model to learn to use confidence, the model is trained directly on the original data and then processed to obtain the corresponding circuit code (e.g., Verilog-A code).

[0151] During training, the model receives images of circuit diagrams, cue words, circuit netlists, and netlist confidence levels. The model learns autonomously by relying more on the circuit netlist when the netlist confidence is high, meaning it pays more attention to the circuit netlist. Conversely, it relies more on image features when the netlist confidence is low, meaning it pays more attention to image features. This allows the model to correct potential errors and thus achieve robust handling of uncertainty.

[0152] The embodiments in this application are merely illustrative of the above-described model training process and do not constitute any limitation.

[0153] In this way, during the circuit code generation process, the multimodal large model adjusts the attention level to the circuit netlist and image features based on the netlist confidence level. This makes the attention level more consistent with reality, allows for more precise control over the model's dependence on information at different levels, and improves the robustness of processing complex circuit diagrams. Furthermore, it enhances the accuracy of the generated circuit code.

[0154] In this application, generating circuit codes corresponding to circuit diagrams using a multimodal large model based on image features, circuit netlists, netlist confidence levels, and cue words may include:

[0155] The multimodal large model processes image features, circuit netlists, and netlist confidence based on prompt words to generate circuit codes corresponding to the circuit diagram.

[0156] The multimodal large model includes self-attention layers and multiple cross-attention layers:

[0157] The self-attention layer is used to process the generated circuit code.

[0158] Multiple cross-attention layers are used to compute the attention outputs of the first attention head and the second attention head in parallel, and to determine the degree of attention to the circuit netlist and the degree of attention to the image features based on the attention outputs; wherein, the attention output is determined based on the netlist confidence, image features and circuit netlist features.

[0159] The generated circuit code refers to the circuit code that has already been generated. It can be a sequence of multiple tokens that the model has progressively predicted using an autoregressive approach before the current generation step.

[0160] Multiple cross-attention layers can compute the attention outputs of a first attention head and a second attention head using a weighted attention fusion method. The first attention head focuses on image features, while the second attention head focuses on the circuit netlist.

[0161] For example, the attention output can be determined by a weighted fusion method based on netlist confidence, image features, and circuit netlist features.

[0162] Attention output can be determined using the following formula:

[0163]

[0164] in, For attention output, For the confidence level of the netlist, For text features, , For circuit netlist characteristics, , These are image features.

[0165] As can be seen from the above formula, in the first Attention, the netlist features are used for calculation, while in the second Attention, the image features are used for calculation.

[0166] This formula indicates that if the netlist features are poor, the model's attention output will be more inclined towards image features; conversely, if the image features are poor, the model's attention output will be more inclined towards netlist features, thereby improving the accuracy and robustness of the model's output.

[0167] In this way, by using a multi-attention mechanism, the model determines whether to pay more attention to the circuit netlist or the image features based on the netlist confidence during the circuit code generation process. This makes the generated circuit code more consistent with the content of the circuit diagram and improves the accuracy of the circuit code.

[0168] The method for generating circuit code using a code generation model in this application first performs visual feature recognition on an image to obtain image features, and then further generates circuit code corresponding to the circuit diagram based on the image features, circuit netlist, and netlist confidence. This allows the code generation model to generate circuit code based on relatively accurate and rich data, effectively improving the accuracy of the generated circuit code.

[0169] Based on the content described in the above embodiments, Figure 7 This is a schematic diagram of the overall framework of a code generation system provided in an embodiment of this application.

[0170] It should be noted that this code generation system is used to execute the circuit diagram-based code generation method described in the above embodiments.

[0171] The code generation system includes a structured information extraction and confidence assessment module and a multimodal code generation module.

[0172] like Figure 7 As shown, the structured information extraction and confidence assessment module includes a CV (Cyclic Array Transformation) module. This CV module processes the input circuit diagram image to obtain the circuit netlist and netlist confidence. Specifically, the methods for obtaining the circuit netlist and netlist confidence can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0173] Furthermore, the visual encoder in the multimodal code generation module can perform feature extraction processing on the input circuit diagram image to obtain the image features corresponding to the circuit diagram image. These image features, the circuit netlist, the netlist confidence score, and the prompt words are then input into the large model in the multimodal code generation module. This large model processes the image features, the circuit netlist, the netlist confidence score, and the prompt words to obtain the circuit code.

[0174] The large model performs the same operations as the decoder or multimodal large model described in the above embodiments, and can be found in the description of the above embodiments, which will not be repeated here.

[0175] Therefore, the circuit diagram-based code generation method provided in this application introduces a CV algorithm to process the circuit diagram image, generating corresponding netlist confidence scores and circuit netlists. The netlist confidence scores are then incorporated into the generation result calculation, helping the model better balance the influence of circuit netlist features and image features on the generation result. This significantly improves the efficiency of circuit diagram processing, enhances the robustness and accuracy of automated circuit diagram generation, and reduces the cost of subsequent manual calibration of the model's generated results. Furthermore, this application achieves end-to-end circuit code generation.

[0176] Figure 8 A schematic diagram of a circuit diagram-based code generation device provided in this application embodiment is shown below. Figure 8 As shown, the circuit diagram-based code generation device 70 provided in this embodiment includes:

[0177] The acquisition module 801 is used to acquire an image of the circuit diagram to be processed.

[0178] The processing module 802 is used to process the image of the circuit diagram to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram; wherein, the netlist confidence level is determined based on the basic components of the circuit diagram extracted from the circuit diagram.

[0179] The generation module 803 is used to generate circuit code corresponding to the circuit diagram based on the circuit diagram image, circuit netlist, and netlist confidence level through the code generation model.

[0180] In one possible implementation, the generation module 803 is specifically used to perform visual feature recognition on the image of the circuit diagram through the feature extraction module in the code generation model to obtain the image features corresponding to the image of the circuit diagram; and to generate the circuit code corresponding to the circuit diagram through the code generation model based on the image features, the circuit netlist, and the netlist confidence.

[0181] In one possible implementation, the code generation model includes a multimodal large model.

[0182] The generation module 803 is specifically used to generate circuit codes corresponding to circuit diagrams using a multimodal large model based on image features, circuit netlists, netlist confidence levels, and cue words. The cue words instruct the multimodal large model to generate circuit codes based on the acquired image features, circuit netlists, and netlist confidence levels. Specifically, the multimodal large model's attention to the circuit netlist during circuit code generation is directly proportional to the netlist confidence level, while its attention to image features is inversely proportional to the netlist confidence level.

[0183] In one possible implementation, the generation module 803 is specifically used to process image features, circuit netlists, and netlist confidence scores based on prompt words in the multimodal large model, generating circuit code corresponding to the circuit diagram. The multimodal large model includes a self-attention layer and multiple cross-attention layers.

[0184] The self-attention layer is used to process the generated circuit code.

[0185] Multiple cross-attention layers are used to compute the attention outputs of the first attention head and the second attention head in parallel, and to determine the degree of attention to the circuit netlist and the degree of attention to the image features based on the attention outputs; wherein, the attention output is determined based on the netlist confidence, image features and circuit netlist features.

[0186] In one possible implementation, the processing module 802 is specifically used to process the image of the circuit diagram to obtain component information, connection line information and text information in the circuit diagram; and to determine the circuit netlist and netlist confidence level corresponding to the circuit diagram based on the component information, connection line information and text information.

[0187] In one possible implementation, the component information includes component data and component confidence, the connection information includes connection lines and connectivity confidence, and the text information includes text data and text confidence.

[0188] The processing module 802 is specifically used to perform target detection on the circuit diagram image, determine the component data in the circuit diagram based on the intersection-union ratio of the recognition results, and determine the component confidence based on the first recognition success rate of the recognition results; the component confidence is proportional to the first recognition success rate; identify the connecting lines in the circuit diagram image through a segmentation model to obtain the connecting line data in the circuit diagram, and determine the connectivity confidence based on the second recognition success rate of the recognition results; the connectivity confidence is proportional to the second recognition success rate; and perform text recognition on the circuit diagram image to obtain the text data and text confidence in the circuit diagram.

[0189] In one possible implementation, text confidence includes at least one of character confidence and word confidence; character confidence is the confidence of a single character in the text data, and word confidence is determined based on the confidence of all characters that make up a word or the word semantics.

[0190] In one possible implementation, the processing module 802 is specifically used to perform fusion processing on component information, connection line information and text information, extract attribute information of each component data, and determine the circuit netlist corresponding to the circuit diagram based on the component information, connection line information, text information and attribute information of each component data.

[0191] In one possible implementation, the processing module 802 is specifically used to determine the netlist confidence corresponding to the circuit diagram based on the circuit netlist, component confidence, connectivity confidence, text confidence, and image features corresponding to the image of the circuit diagram.

[0192] In one possible implementation, the processing module 802 is specifically used to input the image features corresponding to the circuit netlist, component confidence, connectivity confidence, text confidence, and the circuit diagram image into the multimodal large model; and through the multimodal large model, determine the netlist confidence corresponding to the circuit diagram based on the image features corresponding to the circuit netlist, component confidence, connectivity confidence, text confidence, and the circuit diagram image.

[0193] The circuit diagram-based code generation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0194] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the device 90 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus 904.

[0195] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to perform the above-described method.

[0196] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0197] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0198] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0199] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0200] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0201] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0202] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0203] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0204] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0205] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0207] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0208] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0209] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A code generation method based on circuit diagrams, characterized in that, include: Acquire an image of the circuit diagram to be processed; The circuit diagram image is processed to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram; wherein, the netlist confidence level is determined based on the basic components of the circuit diagram extracted from the circuit diagram; The image of the circuit diagram, the circuit netlist, and the netlist confidence level are input into the code generation model. The code generation model then generates the circuit code corresponding to the circuit diagram based on the image of the circuit diagram, the circuit netlist, and the netlist confidence level.

2. The method according to claim 1, characterized in that, The step of generating circuit code corresponding to the circuit diagram using the code generation model based on the image of the circuit diagram, the circuit netlist, and the netlist confidence level includes: The feature extraction module in the code generation model performs visual feature recognition on the image of the circuit diagram to obtain the image features corresponding to the image of the circuit diagram. The code generation model generates circuit code corresponding to the circuit diagram based on the image features, the circuit netlist, and the netlist confidence level.

3. The method according to claim 2, characterized in that, The code generation model includes a multimodal large model; The step of generating circuit code corresponding to the circuit diagram using the code generation model based on the image features, the circuit netlist, and the netlist confidence level includes: The multimodal large model generates circuit code corresponding to the circuit diagram based on the image features, the circuit netlist, the netlist confidence level, and the prompt words; wherein, the prompt words are used to instruct the multimodal large model to generate circuit code based on the acquired image features, circuit netlist, and netlist confidence level. The degree of attention the multimodal large model pays to the circuit netlist when generating the circuit code is directly proportional to the confidence level of the netlist, and the degree of attention it pays to the image features is inversely proportional to the confidence level of the netlist.

4. The method according to claim 3, characterized in that, The step of generating circuit code corresponding to the circuit diagram using the multimodal large model based on the image features, the circuit netlist, the netlist confidence level, and the cue words includes: The multimodal large model processes the image features, the circuit netlist, and the netlist confidence based on the prompt words to generate the circuit code corresponding to the circuit diagram; The multimodal large model includes a self-attention layer and multiple cross-attention layers: The self-attention layer is used to process the generated circuit code; The multiple cross-attention layer is used to compute the attention outputs of the first attention head and the second attention head in parallel, and to determine the degree of attention to the circuit netlist and the degree of attention to the image features based on the attention outputs; wherein, the attention outputs are determined based on the netlist confidence, the image features, and the features of the circuit netlist.

5. The method according to any one of claims 1-4, characterized in that, The process of processing the circuit diagram image to obtain the corresponding circuit netlist and netlist confidence includes: The circuit diagram image is processed to obtain component information, connection line information and text information in the circuit diagram; Based on the component information, the connection line information, and the text information, determine the circuit netlist and netlist confidence level corresponding to the circuit diagram.

6. The method according to claim 5, characterized in that, The component information includes component data and component confidence; the connection information includes connection lines and connectivity confidence; and the text information includes text data and text confidence. The process of processing the circuit diagram image to obtain component information, connection information, and text information in the circuit diagram includes: Target detection is performed on the image of the circuit diagram. The component data in the circuit diagram is determined based on the intersection-union ratio of the recognition results. The component confidence is determined based on the first recognition success rate of the recognition results. The component confidence is proportional to the first recognition success rate. The connection lines in the circuit diagram image are identified using a segmentation model to obtain the connection line data in the circuit diagram, and the connectivity confidence is determined based on the second recognition success rate of the recognition results; the connectivity confidence is proportional to the second recognition success rate. Text recognition is performed on the image of the circuit diagram to obtain the text data and text confidence level in the circuit diagram.

7. The method according to claim 6, characterized in that, The text confidence score includes at least one of character confidence score and word confidence score; the character confidence score is the confidence score of a single character in the text data, and the word confidence score is determined based on the confidence scores of all characters that make up the word or the word semantics.

8. The method according to claim 6 or 7, characterized in that, The step of determining the circuit netlist corresponding to the circuit diagram based on the component information, the connection line information, and the text information includes: The component information, the connection line information, and the text information are fused together to extract the attribute information of each component data. Based on the component information, the connection line information, the text information, and the attribute information of each component data, the circuit netlist corresponding to the circuit diagram is determined.

9. The method according to claim 8, characterized in that, The step of determining the netlist confidence level corresponding to the circuit diagram based on the component information, the connection information, and the text information includes: The netlist confidence level corresponding to the circuit diagram is determined based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram.

10. The method according to claim 9, characterized in that, The step of determining the netlist confidence level corresponding to the circuit diagram based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram includes: The circuit netlist, the component confidence, the connectivity confidence, the text confidence, and the image features corresponding to the circuit diagram are input into the multimodal large model; Using the multimodal large model, the netlist confidence level corresponding to the circuit diagram is determined based on the circuit netlist, the component confidence level, the connectivity confidence level, the text confidence level, and the image features corresponding to the image of the circuit diagram.

11. A code generation device based on circuit diagrams, characterized in that, include: The acquisition module is used to acquire images of the circuit diagram to be processed; The processing module is used to process the image of the circuit diagram to obtain the circuit netlist and netlist confidence level corresponding to the circuit diagram; wherein, the netlist confidence level is determined based on the basic components of the circuit diagram extracted from the circuit diagram; The generation module is used to generate circuit code corresponding to the circuit diagram based on the image of the circuit diagram, the circuit netlist, and the confidence level of the netlist using a code generation model.

12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images