Chip time sequence diagram labeling method and device based on multi-modal large language model

Abstract

The invention relates to a chip time sequence diagram labeling method and device based on a multi-mode large language model, and relates to the field of chip time sequence diagram analysis and automatic labeling, and the method comprises the steps: firstly, carrying out the line and column line recognition of a chip time sequence diagram, and obtaining a signal line block and a time line block; then, detecting an intersection point area of the signal line block and the time line block, and further obtaining an image slice; then, detecting to obtain an annotation state of the image slice; then connecting the continuous annotation states in series to obtain an event, and further obtaining a natural language conforming to the understanding of the large language model; and finally, constructing according to the cue word to obtain a second text description, inputting the second text description into the first large language model to obtain a text response returned by the first large language model, and constructing based on the text response to obtain a sample for training a second large language model. Compared with the prior art, the method has the advantages that the accuracy of label state recognition is improved, the efficiency of training sample generation is improved, and the degree of manual intervention is reduced.

Description

Technical Field

[0001] The present invention relates to the field of chip timing diagram analysis and automatic annotation, and in particular to a chip timing diagram annotation method and device based on a multimodal large language model. Background Art

[0002] Chip timing diagrams, as an important graphical tool for expressing signal interactions and timing relationships in integrated circuit design, are widely used in chip design, verification, and documentation. However, traditional manual interpretation of chip timing diagrams is not only inefficient and error-prone, but also has inherent limitations in understanding and consistency when faced with complex waveforms, diverse standards, and deep semantic logic.

[0003] With the development of artificial intelligence technology, especially breakthroughs in image and text understanding using multimodal large language models, it has become possible to develop intelligent models that can automatically understand chip timing diagrams. However, training such models relies heavily on large-scale, high-quality, and diverse annotated data, particularly training data in the form of question-and-answer pairs that reflect the inherent logic and semantics of the timing diagrams. Traditionally, annotated data used for training is often obtained through manual annotation, but manual annotation is costly. Understanding and annotating complex timing diagrams requires domain expertise, and manually creating large numbers of QA pairs is time-consuming, labor-intensive, and extremely expensive.

[0004] In this regard, although some existing technologies attempt to use some methods for automatic annotation, for example, Chinese patent CN119378564A discloses a large model data intelligent annotation method and system, which uses natural language processing technology based on deep learning to perform confidence assessment on each unlabeled text data in the unlabeled text data set, selects the text data corresponding to the minimum confidence as the representative sample data, and expands the corpus of the representative sample data. Then, by extracting semantic features and performing compensatory interactive fusion on the representative sample data and the representative sample data after corpus expansion, the common information and unique information between the two are fully utilized, thereby achieving comprehensive semantic understanding and intelligent annotation of the representative sample data. In this way, the efficiency and accuracy of data annotation can be significantly improved, while the need for manual intervention is greatly reduced, and the annotation cost is reduced. However, although this method has improved the efficiency and accuracy of data annotation, it has obvious shortcomings when processing chip timing diagrams. It focuses on the semantic processing of text data, fails to fully utilize the image structured information of the timing diagram, lacks effective processing capabilities for complex timing logic, signal waveforms and noise interference, resulting in limited accuracy and completeness of the annotation results. At the same time, this method is highly dependent on the knowledge of domain experts and cannot fully meet the diverse annotation needs of chip timing diagrams, affecting the subsequent model training effect. Summary of the Invention

[0005] The purpose of the present invention is to provide a chip timing diagram annotation method and device based on a multimodal large language model.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A chip timing diagram annotation method based on a multimodal large language model, comprising:

[0008] Step S1: Perform row and column line identification on the chip timing diagram to obtain signal line blocks and time line blocks respectively;

[0009] Step S2: detecting the intersection areas of the signal line block and the time line block, and extracting image slices based on each intersection area;

[0010] Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice, wherein, for the state labeling classification result of the text state, further identify the text content as the labeling state; for other state labeling classification results, use the state labeling classification result as the labeling state;

[0011] Step S4: constructing a first text description based on the signal line block and the time line block and in combination with the annotation status of each image slice;

[0012] Step S5: According to the preconfigured prompt words, the second text description is obtained in combination with the first text description, and the second text description is input into the first language model to obtain a text response returned by the first language model, and a sample for training the second language model is constructed based on the text response.

[0013] The signal line identification process in step S1 includes:

[0014] Step S1-1-1: summing the foreground pixel values ​​of each row in the chip timing diagram to obtain a first horizontal projection profile sequence;

[0015] Step S1-1-2: Obtain significant valley values ​​by peak-valley detection on the first horizontal projection profile sequence, and use the rows corresponding to all significant valley values ​​as separation bands, and use the rows between the separation bands as candidate strip regions;

[0016] Step S1-1-3: summing the foreground pixel values ​​of each column in each candidate strip area to obtain a first vertical projection contour sequence corresponding to each candidate strip area, and segmenting the candidate strip area into a text area and a waveform area based on the first vertical projection contour sequence;

[0017] Step S1-1-4: Based on the text area and waveform area corresponding to the same candidate strip area, the candidate strip area is divided into one or more signal line blocks and their corresponding text blocks in combination with the typical spacing and number of segments of the waveform and text.

[0018] The step S1-1-4 includes:

[0019] Step S1-1-4-1: Acquire the text area and the waveform area corresponding to the same candidate strip area;

[0020] Step S1-1-4-2: summing the foreground pixel values ​​of each row to obtain a second horizontal projection profile sequence corresponding to the text area and a third horizontal projection profile sequence corresponding to the waveform area;

[0021] Step S1-1-4-3: Determine whether the length of the second horizontal projection profile sequence is within the first preconfigured interval, and whether the length of the third horizontal projection profile sequence is within the second preconfigured interval. If so, output the waveform area as the signal line block, and output the text area as the text block corresponding to the signal line block, and execute step S1-1-4-10; otherwise, execute step S1-1-4-4;

[0022] Step S1-1-4-4: Initialize the first separation detection threshold;

[0023] Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are smaller than the first separation detection threshold to 0, calculate the number of segments in the second horizontal projection contour sequence that are not zero as the number of first segments and the length of each first segment, and determine whether the lengths of all first segments are within the first preconfigured interval. If so, execute step S1-1-4-7; otherwise, execute step S1-1-4-6;

[0024] Step S1-1-4-6: Determine whether the first separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5;

[0025] Step S1-1-4-7: Initialize the second separation detection threshold;

[0026] Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are smaller than the second separation detection threshold to 0, calculate the number of segments in the third horizontal projection contour sequence that are not zero as the second segment number and the length of each second segment, and determine whether the lengths of all second segments are within the second preconfigured interval. If so, execute step S1-1-4-10; otherwise, execute step S1-1-4-9;

[0027] Step S1-1-4-9: Determine whether the second separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8;

[0028] Step S1-1-4-10: output all the lines corresponding to each first segment as a signal line block, and output all the lines corresponding to the second segment with the shortest vertical distance as the text block corresponding to the signal line block.

[0029] The timeline identification process in step S1 includes:

[0030] Step S1-2-1: Use Hough transform algorithm to detect the chip timing diagram and pre-set a length threshold L min The threshold is used to distinguish valid timelines from noise or non-critical segments. For each vertical segment detected, its length L is calculated. If L≥L min , the line segment is considered to be a main vertical line segment; otherwise, it is considered to be noise or non-critical line segment and is excluded.

[0031] Step S1-2-2: For the main vertical line segments, filter and obtain the timeline blocks by length detection and position detection;

[0032] Step S1-2-3: Perform OCR recognition to extract text above the key timeline blocks, and match it with the time keyword set to obtain the time text corresponding to each timeline block.

[0033] The step S1-2-3 includes:

[0034] Step S1-2-3-1: After determining the position of the key timeline block, perform OCR recognition in a specific area directly above it to extract text. Specifically, set a vertical offset threshold hthreshold to determine the range of the area extending upward from the top of the timeline block. At the same time, set the horizontal range to be consistent with the horizontal projection range of the timeline block to ensure that OCR recognition is only performed in the area horizontally aligned with the timeline block. Perform OCR recognition within this rectangular area;

[0035] Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, all the extracted texts are matched according to the nearest matching principle to obtain the time text corresponding to each timeline block.

[0036] The step S2 specifically includes:

[0037] Step S2-1: Detecting the intersection area of ​​the signal line block and the time line block;

[0038] Step S2-2: Expanding each intersection area outward by a plurality of pixel distances to obtain a plurality of image slices.

[0039] The first classification model is a ResNet state type classifier. The state label classification results include at least text state, high level, low level, rising edge, falling edge, and high impedance state. These classification results can be further categorized into two categories: signal state and signal change. High level, low level, and high impedance state are signal states, indicating the stable state of the signal at a certain moment; rising edge and falling edge are signal changes, indicating the transition of the signal from one state to another. Text state is used to identify text information in image slices, such as signal names or other labeled text.

[0040] The step S4 comprises:

[0041] Step S4-1: For each signal line block, randomly select two time line blocks;

[0042] Step S4-2: Connect all the annotated states between the two selected timeline blocks in series according to the time sequence to construct a set of event sequences;

[0043] Step S4 - 3 : composing a first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence according to a pre-configured text composition template.

[0044] The prompt words include:

[0045] Role indication field: indicates that the first language model is a time series analysis expert;

[0046] Task type qualification field: Indicates that the task of the first language model is focused on timing behavior analysis and functional verification. The core task is to identify and explain the key timing constraints, causal relationships, and control dependencies between various signals based on the information provided by the input data.

[0047] Input format description field: used to describe in detail the format of the input data obtained based on the first text description;

[0048] Output Restriction Field: Indicates the format of the output text response.

[0049] A chip timing diagram annotation device based on a multimodal large language model includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. First, through computer vision technology, image slices are obtained by row and column line recognition, which can improve the effectiveness of image slice state annotation. All state annotations are concatenated in chronological order to construct a first text description, which can provide a textual representation of the image information of the timing diagram. Combined with the second text description constructed with prompt words, it can effectively use the high-level large language model to understand the temporal logic, interaction relationship, and dependency relationship between the first text descriptions of different signals, and generate a standardized structured representation as a training sample for the low-level large language model.

[0052] 2. This method uses horizontal projection combined with peak and valley detection to identify candidate strip regions, and then further identifies signal line blocks within these candidate strip regions. Compared to methods that simply set a threshold to determine signal line regions, this method automatically adapts to varying image quality and signal line complexity, eliminating the need for a fixed threshold and improving robustness and accuracy. It accurately identifies signal line boundaries, reduces missegmentation, enhances noise immunity, and flexibly handles complex signal line structures, providing a reliable foundation for subsequent annotation and analysis.

[0053] 3. The process of dividing the candidate strip area into one or more signal line blocks and their corresponding text blocks by specifically combining the typical spacing and number of segments of the waveform and text can effectively solve the problem of interference of noise in the image on the recognition of signal line blocks.

[0054] 4. Use the Hough transform algorithm to detect vertical line segments in the chip timing diagram. Use the length threshold to filter the main vertical line segments and exclude noise or non-critical line segments shorter than the threshold. Then, based on the proximity distance threshold and slope threshold, cluster and merge adjacent and collinear short line segments to form a complete timeline block. Compared with the vertical projection method, even in the presence of noise or discontinuity, setting a length threshold can exclude short line segments generated by noise and reduce misidentification. By clustering and merging the proximity distance and slope thresholds, the accuracy of line segment recognition is further improved to ensure that the line segments are continuous in space and consistent in direction. These characteristics make this method more robust when dealing with complex images and noise interference, and can more reliably extract timeline blocks, providing a solid foundation for subsequent parsing and annotation. 5. The text of the timeline block is obtained by matching the time keyword set, which can improve the accuracy of text recognition.

[0055] 6. Specially designed prompts clearly indicate the role and task type of the large language model, allowing it to focus on temporal behavior analysis and functional verification, improving its understanding of temporal logic. Furthermore, by detailing the input data format and restricting the output format, the model accurately parses the input and generates high-quality annotation results, reducing manual intervention and improving annotation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1Schematic diagram of the main steps of the method of the present invention;

[0057] Figure 2 This is a timing diagram of a chip that needs to be analyzed in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0059] A chip timing diagram annotation method based on a multimodal large language model, such as Figure 1 As shown, including:

[0060] Step S1: Perform row and column line identification on the chip timing diagram to obtain signal line blocks and time line blocks respectively;

[0061] like Figure 2 The figure shows a timing diagram of a chip. Considering that the image quality of some chip timing diagrams is low, it is necessary to pre-process the chip timing diagram first. Specifically, in this embodiment, the process includes:

[0062] 1. Image enhancement: By applying image processing technologies such as contrast enhancement, sharpening, and noise suppression, the visual quality of timing diagram images is improved, making signal lines, waveform edges, and text annotations clearer and more distinct, and reducing the interference of blur and noise on subsequent accurate recognition.

[0063] 2. Perspective Correction: To address image tilt and distortion caused by shooting angle or scanning deformation, the system detects key geometric features in the image and calculates the corresponding transformation matrix to restore the deformed image to a standard frontal plane view, ensuring the horizontal and vertical alignment of signal lines and time axes, laying the foundation for subsequent structured analysis.

[0064] 3. Size standardization: To meet the requirements of subsequent deep learning models for fixed input size, original time series graphs of different sizes are uniformly adjusted to the preset dimensions by maintaining the aspect ratio and performing necessary padding operations. This ensures the consistency of the input data, thereby improving the processing efficiency and generalization ability of the model.

[0065] The signal line identification process in step S1 includes:

[0066] Step S1-1-1: Sum the foreground pixel values ​​of each row in the chip timing diagram to obtain a first horizontal projection profile sequence. The first horizontal projection profile sequence is a one-dimensional array, where each element corresponds to a row, the array index represents the vertical position of the image, and the array value represents the total number of foreground pixel values ​​in the corresponding row;

[0067] Step S1-1-2: Obtain significant valley values ​​by peak-valley detection on the first horizontal projection profile sequence, and use the rows corresponding to all significant valley values ​​as separation bands, and use the rows between the separation bands as candidate strip regions;

[0068] Specifically, the peak and valley detection method may be to derive the first horizontal projection profile sequence, find the zero-crossing points of the derivative, and take the rows with the smallest values ​​between the zero-crossing points as potential significant valley values.

[0069] The rows corresponding to the significant valley values ​​are regarded as blank intervals, which can be used as separation bands as long as an appropriate threshold is set.

[0070] Step S1-1-3: summing the foreground pixel values ​​of each column in each candidate strip area to obtain a first vertical projection contour sequence corresponding to each candidate strip area, and segmenting the candidate strip area into a text area and a waveform area based on the first vertical projection contour sequence;

[0071] Step S1-1-4: Based on the text area and waveform area corresponding to the same candidate strip area, the candidate strip area is segmented into one or more signal line blocks and their corresponding text blocks in combination with the typical spacing and number of segments of the waveform and text. Specifically, the step includes:

[0072] Step S1-1-4-1: Acquire the text area and the waveform area corresponding to the same candidate strip area;

[0073] Step S1-1-4-2: summing the foreground pixel values ​​of each row to obtain a second horizontal projection profile sequence corresponding to the text area and a third horizontal projection profile sequence corresponding to the waveform area;

[0074] Step S1-1-4-3: Determine whether the length of the second horizontal projection profile sequence is within the first preconfigured interval, and whether the length of the third horizontal projection profile sequence is within the second preconfigured interval. If so, output the waveform area as the signal line block, and output the text area as the text block corresponding to the signal line block, and execute step S1-1-4-10; otherwise, execute step S1-1-4-4;

[0075] Step S1-1-4-4: Initialize the first separation detection threshold;

[0076] Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are smaller than the first separation detection threshold to 0, calculate the number of segments in the second horizontal projection contour sequence that are not zero as the number of first segments and the length of each first segment, and determine whether the lengths of all first segments are within the first preconfigured interval. If so, execute step S1-1-4-7; otherwise, execute step S1-1-4-6;

[0077] Step S1-1-4-6: Determine whether the first separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5;

[0078] Step S1-1-4-7: Initialize the second separation detection threshold;

[0079] Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are smaller than the second separation detection threshold to 0, calculate the number of segments in the third horizontal projection contour sequence that are not zero as the second segment number and the length of each second segment, and determine whether the lengths of all second segments are within the second preconfigured interval. If so, execute step S1-1-4-10; otherwise, execute step S1-1-4-9;

[0080] Step S1-1-4-9: Determine whether the second separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8;

[0081] Step S1-1-4-10: output all the lines corresponding to each first segment as a signal line block, and output all the lines corresponding to the second segment with the shortest vertical distance as the text block corresponding to the signal line block.

[0082] After each signal line block is successfully detected, the text content in the text block is extracted using OCR technology as the signal line name of the signal line block.

[0083] The timeline identification process in step S1 includes:

[0084] Step S1-2-1: Use Hough transform algorithm to detect the chip timing diagram and pre-set a length threshold L min The threshold is used to distinguish valid timelines from noise or non-critical segments. For each vertical segment detected, its length L is calculated. If L≥L min , then the line segment is considered to be the main vertical line segment; otherwise, it is considered to be noise or non-critical line segment and excluded;

[0085] Step S1-2-2: For the main vertical line segments, calculate the distance between each line segment and other line segments by length detection and position detection. If the distance between two line segments is less than the preset proximity distance threshold D threshold , then the two line segments are considered to be adjacent. Further check whether the adjacent line segments are collinear. If the difference in slopes of the two line segments is less than the preset slope threshold θ threshold , they are considered collinear. All adjacent and collinear short segments are clustered and merged to obtain timeline blocks;

[0086] Step S1-2-3: Perform OCR recognition and extract text near the key timeline blocks, and match it with the time keyword set to obtain the time text corresponding to each timeline block. Specifically, it includes:

[0087] Step S1-2-3-1: Perform OCR recognition on the image above the timeline block to extract text. Generally, the specific pixel distance can be adjusted according to actual conditions, for example, it can be designed to be proportional to the average vertical pixel length of the signal line block;

[0088] Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, all the extracted texts are matched according to the nearest matching principle to obtain the time text corresponding to each timeline block.

[0089] Generally, the elements in the time keyword set may include: t0, t1, tINIT2, Reset, etc., and regular maintenance may be performed to ensure its integrity.

[0090] Step S2: Detecting the intersection areas of the signal line block and the time line block, and extracting image slices based on each intersection area, specifically including:

[0091] Step S2-1: Detecting the intersection area of ​​the signal line block and the time line block;

[0092] Step S2-2: After each intersection area is expanded outward by a plurality of pixel distances, a plurality of image slices are obtained. Generally, in this embodiment, the specific pixel distance can be determined in the following manner:

[0093] The pixel distance is proportional to the average vertical pixel length of the signal line block. For example, if the average vertical pixel length of the signal line block is h, the extended pixel distance can be set to 0.5h to ensure that the extracted image slice can cover the intersection of the signal line and the time line and the key information around it.

[0094] Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice, wherein, for the state labeling classification result of the text state, further identify the text content as the labeling state; for other state labeling classification results, use the state labeling classification result as the labeling state;

[0095] In this embodiment, the first classification model is a ResNet state type classifier, and the state label classification results include at least text state, high level, low level, rising edge, falling edge, and high impedance state. These classification results can be further categorized into two categories: signal state and signal change. Among them, high level, low level, and high impedance state are signal states, indicating the stable state of the signal at a certain moment; while rising edge and falling edge are signal changes, indicating the transition of the signal from one state to another. The text state is used to identify text information in the image slice, such as the signal name or other labeled text.

[0096] Among them, the first classification model can be obtained through training, and the training process is carried out using the state annotation classification results corresponding to the image slicing machine. In this embodiment, the ResNet state type classifier is based on the ResNet-18 or ResNet-34 architecture, and its deep network structure can automatically learn from the original pixels and extract robust visual features for distinguishing different state types. The output layer of the model uses the Softmax activation function to output the probability that the input slice belongs to each predefined state type.

[0097] The text status belongs to a special status label classification result. For this type of result, the corresponding text content needs to be further obtained through OCR recognition, such as Valid, DES, NOP, etc.

[0098] Step S4: Based on the signal line block and the time line block, combined with the annotation status of each image slice, a first text description is constructed.

[0099] This process requires converting the previously acquired information into a coherent natural language or semi-structured text description that is easily understood by large language models. The focus is on clearly expressing time, signals, events, and the known relationships between them. The specific methods are as follows:

[0100] Step S4-1: For each signal line block, randomly select two time line blocks.

[0101] For example, the signal line ta and the signal line tb may be selected.

[0102] Step S4-2: Connect all the annotated states between the two selected timeline blocks in series according to the time sequence to construct a set of event sequences;

[0103] For example, the state changes of the signal line at these time points and the time periods between them, such as from high level to low level or stable state, are arranged in chronological order to form an event sequence of the signal. The stable state may be maintaining NOP, etc.

[0104] The name of each signal is then clearly associated with the sequence of events to which it corresponds.

[0105] Step S4 - 3 : composing a first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence according to a pre-configured text composition template.

[0106] The first text description generated may be: the signal CMD is in the NOP state during the time period td to te.

[0107] Step S5: According to the preconfigured prompt words, the second text description is obtained in combination with the first text description, and the second text description is input into the first language model to obtain a text response returned by the first language model, and a sample for training the second language model is constructed based on the text response.

[0108] Prompt words include:

[0109] Role indication field: indicates that the first language model is a time series analysis expert;

[0110] Task type qualification field: Indicates that the task of the first language model is focused on timing behavior analysis and functional verification. The core task is to identify and explain the key timing constraints, causal relationships, and control dependencies between various signals based on the information provided by the input data.

[0111] Input format description field: used to describe in detail the format of the input data obtained based on the first text description;

[0112] Output Restriction Field: Indicates the format of the output text response.

[0113] In this embodiment, the first large language model selects an online cloud-based large language model such as GPT4-o1 and Claude3.7 with powerful text comprehension and logical reasoning capabilities, and submits the constructed Prompt and textual time series data through its API interface.

[0114] Afterwards, the first language model performs inference. During this period, the first language model analyzes the signal behavior described in the text based on the received information and instructions, and infers the potential temporal dependencies.

[0115] In this embodiment, examples of prompt words are also provided as follows:

[0116] You are a senior digital logic design verification engineer specializing in timing behavior analysis and functional verification. Your core task is to identify and explain the key timing constraints, causal relationships, and control dependencies between signals based on the signal interaction sequences and timing diagrams extracted and documented from the chip timing diagrams provided below.

[0117] The format of a certain input data is as follows:

[0118] The behavior of each signal is described in chronological order in the following format:

[0119] Signal <signal name>:

[0120] - At <time point / time period> the status is <status value> [Optional: Event description <description>]

[0121] -Change from <state A> at <time point A> to <state B> at <time point B> [Optional: Associated timing parameter <parameter name>]

[0122] Correspondingly, the output format requirements are as follows:

[0123] Please return all identified timing dependencies as a JSON list. Each dependency object should contain the following fields:

[0124] - "source_signal": "The name of the source signal that triggers or affects the signal"

[0125] -"target_signal":"The name of the target signal to be affected"

[0126] -"dependency_type":"Dependency type (e.g. 'triggers_change', 'enables_event', 'data_setup_for', 'follows_state')"

[0127] -"timing_condition":"The time point, time period or specific condition description of the dependency"

[0128] -"reasoning":"Briefly describe your reasons for judging this dependency"

[0129] For example, the signal input is as follows:

[0130] Signal RESET_n:

[0131] - The status is High from t_start to t_a

[0132] -Change from t_a state High to t_b state Low, associated timing parameter tPW_RESET

[0133] -Change from t_b state Low to t_c state High

[0134] Signal CMD_VALID:

[0135] - The state is Low from t_start to t_b

[0136] -Change from t_b state Low to t_c state High, event description Command becomes valid signal CLK:

[0137] -Periodic square wave, rising edge at t_clk1, t_clk2,...

[0138] Also for example, the output is as follows: [

[0140] {

[0141] "source_signal":"RESET_n",

[0142] "target_signal":"CMD_VALID",

[0143] "dependency_type":"enables_event",

[0144] "timing_condition":"CMD_VALID becomes High after RESET_n changes from Low to High (t_b to t_c)",

[0145] "reasoning":"The CMD_VALID signal becomes valid (high level) only after the reset signal RESET_n is released (goes high), indicating that commands are invalid during the system reset and become valid only after the reset is complete."

[0146] } ]

[0148] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A chip timing diagram annotation method based on a multimodal large language model, characterized in that: include: Step S1: Perform row and column line identification on the chip timing diagram to obtain signal line blocks and time line blocks respectively; Step S2: detecting the intersection areas of the signal line block and the time line block, and extracting image slices based on each intersection area; Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice, wherein, for the state labeling classification result of the text state, further identify the text content as the labeling state; for other state labeling classification results, use the state labeling classification result as the labeling state; Step S4: constructing a first text description based on the signal line block and the time line block and in combination with the annotation status of each image slice; Step S5: According to the preconfigured prompt words, the second text description is obtained in combination with the first text description, and the second text description is input into the first language model to obtain a text response returned by the first language model, and a sample for training the second language model is constructed based on the text response.

2. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The signal line identification process in step S1 includes: Step S1-1-1: summing the foreground pixel values ​​of each row in the chip timing diagram to obtain a first horizontal projection profile sequence; Step S1-1-2: Obtain significant valley values ​​by peak-valley detection on the first horizontal projection profile sequence, and use the rows corresponding to all significant valley values ​​as separation bands, and use the rows between the separation bands as candidate strip regions; Step S1-1-3: summing the foreground pixel values ​​of each column in each candidate strip area to obtain a first vertical projection contour sequence corresponding to each candidate strip area, and segmenting the candidate strip area into a text area and a waveform area based on the first vertical projection contour sequence; Step S1-1-4: Based on the text area and waveform area corresponding to the same candidate strip area, the candidate strip area is divided into one or more signal line blocks and their corresponding text blocks in combination with the typical spacing and number of segments of the waveform and text.

3. The chip timing diagram annotation method based on a multimodal large language model according to claim 2, characterized in that: The step S1-1-4 includes: Step S1-1-4-1: Acquire the text area and the waveform area corresponding to the same candidate strip area; Step S1-1-4-2: summing the foreground pixel values ​​of each row to obtain a second horizontal projection profile sequence corresponding to the text area and a third horizontal projection profile sequence corresponding to the waveform area; Step S1-1-4-3: Determine whether the length of the second horizontal projection profile sequence is within the first preconfigured interval, and whether the length of the third horizontal projection profile sequence is within the second preconfigured interval. If so, output the waveform area as the signal line block, and output the text area as the text block corresponding to the signal line block, and execute step S1-1-4-10; otherwise, execute step S1-1-4-4; Step S1-1-4-4: Initialize the first separation detection threshold; Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are smaller than the first separation detection threshold to 0, calculate the number of segments in the second horizontal projection contour sequence that are not zero as the number of first segments and the length of each first segment, and determine whether the lengths of all first segments are within the first preconfigured interval. If so, execute step S1-1-4-7; otherwise, execute step S1-1-4-6; Step S1-1-4-6: Determine whether the first separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5; Step S1-1-4-7: Initialize the second separation detection threshold; Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are smaller than the second separation detection threshold to 0, calculate the number of segments in the third horizontal projection contour sequence that are not zero as the second segment number and the length of each second segment, and determine whether the lengths of all second segments are within the second preconfigured interval. If so, execute step S1-1-4-10; otherwise, execute step S1-1-4-9; Step S1-1-4-9: Determine whether the second separation detection threshold reaches the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8; Step S1-1-4-10: output all the lines corresponding to each first segment as a signal line block, and output all the lines corresponding to the second segment with the shortest vertical distance as the text block corresponding to the signal line block.

4. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The timeline identification process in step S1 includes: Step S1-2-1: Use Hough transform algorithm to detect the chip timing diagram and pre-set a length threshold L min The threshold is used to distinguish valid timelines from noise or non-critical segments. For each vertical segment detected, its length L is calculated. If L≥L min , then the line segment is considered to be the main vertical line segment; otherwise, it is considered to be noise or non-critical line segment and excluded; Step S1-2-2: For the main vertical line segments, filter out key vertical lines through length detection and position detection, and cluster and merge adjacent and collinear short line segments to obtain timeline blocks; Step S1-2-3: Perform OCR recognition to extract text above the key timeline blocks, and match it with the time keyword set to obtain the time text corresponding to each timeline block.

5. The chip timing diagram annotation method based on a multimodal large language model according to claim 4, characterized in that: The step S1-2-3 includes: Step S1-2-3-1: After determining the position of the key timeline block, perform OCR recognition in the specific area directly above it to extract text. Specifically, set a vertical offset threshold h threshold To determine the area extending upward from the top of the timeline block. At the same time, set the horizontal range to be consistent with the horizontal projection range of the timeline block to ensure that OCR recognition is only performed in the area horizontally aligned with the timeline block. Perform OCR recognition within this rectangular area; Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, all the extracted texts are matched according to the nearest matching principle to obtain the time text corresponding to each timeline block.

6. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The step S2 specifically includes: Step S2-1: Detecting the intersection area of ​​the signal line block and the time line block; Step S2-2: Expanding each intersection area outward by a plurality of pixel distances to obtain a plurality of image slices.

7. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The first classification model is a ResNet state type classifier, and the state labeling classification results include at least text state, high level, low level, rising edge, falling edge and high impedance state.

8. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The step S4 comprises: Step S4-1: For each signal line block, randomly select two time line blocks; Step S4-2: Connect all the annotated states between the two selected timeline blocks in series according to the time sequence to construct a set of event sequences; Step S4 - 3 : composing a first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence according to a pre-configured text composition template.

9. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that: The prompt words include: Role indication field: indicates that the first language model is a time series analysis expert; Task type qualification field: Indicates that the task of the first language model is focused on timing behavior analysis and functional verification. The core task is to identify and explain the key timing constraints, causal relationships, and control dependencies between various signals based on the information provided by the input data. Input format description field: used to describe in detail the format of the input data obtained based on the first text description; Output Restriction Field: Indicates the format of the output text response.

10. A chip timing diagram annotation device based on a multimodal large language model, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images