A method and apparatus for chip timing icon annotation based on a multimodal large language model
By combining computer vision and multimodal large language models, chip timing diagrams are automatically identified and labeled, solving the problems of low efficiency and insufficient accuracy in traditional methods, and achieving efficient and accurate chip timing diagram labeling and model training.
Patent Information
- Application Number
- CN202510805844.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-17
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-06-17
AI Technical Summary
Traditional manual interpretation of chip timing diagrams is inefficient and error-prone. Existing automatic annotation methods fail to fully utilize the structured information in the images, cannot handle complex timing logic and noise interference, rely on domain expert knowledge, and cannot meet the diverse annotation needs of chip timing diagrams.
By using computer vision technology to identify signal lines and timelines, combining a multimodal large language model to annotate image slice states, using Hough transform to detect timelines, designing specific prompt words to guide the large language model to perform temporal logic analysis, and generating structured annotation results.
It improves the accuracy and robustness of chip timing diagram annotation, reduces noise interference, reduces manual intervention, and improves annotation efficiency and model training effect.
Smart Images

Figure CN120673196B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of chip timing diagram analysis and automatic annotation, and in particular to a chip timing diagram annotation method and apparatus based on a multimodal large language model. Background Technology
[0002] Chip timing diagrams, as important graphical tools for expressing signal interactions and timing relationships in integrated circuit design, are widely used in chip design, verification, and documentation. However, traditional methods of manually interpreting chip timing diagrams are not only inefficient and error-prone, but also have inherent limitations in understanding and consistency when faced with complex waveforms, diverse standards, and deep semantic logic.
[0003] With the development of artificial intelligence technology, especially the breakthroughs of multimodal large language models in the field of image and text understanding, it has become possible to develop intelligent models that can automatically understand chip timing diagrams. However, the training of such models heavily relies on large-scale, high-quality, and diverse labeled data, especially training data in the form of question-and-answer pairs that can reflect the inherent logic and semantics of timing diagrams. Traditional labeled data used for training is usually obtained through manual annotation, but manual annotation is costly. Understanding and annotating complex timing diagrams requires domain expert knowledge, and manually creating a large number of QA pairs is time-consuming, labor-intensive, and extremely costly.
[0004] While some existing technologies attempt to automate annotation, such as the Chinese patent CN119378564A which discloses a method and system for intelligent annotation of large model data, this approach employs deep learning-based natural language processing to evaluate the confidence of each unlabeled text data in an unlabeled text dataset. It selects the text data with the lowest confidence as representative sample data, expands the corpus of this representative sample data, and then performs semantic feature extraction and compensatory interactive fusion on the representative sample data and the expanded corpus to fully utilize the shared and unique information between them, thereby achieving comprehensive semantic understanding and intelligent annotation of the representative sample data. This method can significantly improve the efficiency and accuracy of data annotation while greatly reducing the need for manual intervention and lowering annotation costs. However, although this method improves the efficiency and accuracy of data annotation, it has significant shortcomings when processing chip timing diagrams. It focuses on the semantic processing of text data and fails to fully utilize the image structured information of timing diagrams. It lacks effective processing capabilities for complex timing logic, signal waveforms, and noise interference, resulting in limitations in the accuracy and completeness of the annotation results. Meanwhile, this method relies heavily on domain expert knowledge, which cannot fully meet the diverse annotation needs of chip timing diagrams and affects the subsequent model training effect. Summary of the Invention
[0005] The purpose of this invention is to provide a chip timing icon annotation method and apparatus based on a multimodal large language model.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] A chip timing icon annotation method based on a multimodal large language model includes:
[0008] Step S1: Identify the row and column lines of the chip timing diagram to obtain signal line blocks and time line blocks respectively;
[0009] Step S2: Detect the intersection region of the signal line block and the time line block, and extract image slices based on each intersection region;
[0010] Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice. For the state labeling classification result that is a text state, the text content is further identified and used as the labeling state. For other state labeling classification results, the state labeling classification result is used as the labeling state.
[0011] Step S4: Based on the signal line block and time line block, and combined with the annotation status of each image slice, construct the first text description;
[0012] Step S5: According to the pre-configured prompt words, combine the first text description to obtain the second text description, and input the second text description into the first language model to obtain the text response returned by the first language model. Based on the text response, construct samples for training the second language model.
[0013] The signal line identification process in step S1 includes:
[0014] Step S1-1-1: Sum the foreground pixel values of each row in the chip timing diagram to obtain the first horizontal projection contour sequence;
[0015] Step S1-1-2: Obtain significant valley values from the first horizontal projection contour sequence by peak and valley detection, and use the rows corresponding to all significant valley values as separators, and the rows between the separators as candidate strip regions;
[0016] Step S1-1-3: Sum the foreground pixel values of each column in each candidate strip region to obtain the first vertical projection contour sequence corresponding to each candidate strip region, and segment the candidate strip region into text region and waveform region based on the first vertical projection contour sequence.
[0017] Step S1-1-4: Based on the text region and waveform region corresponding to the same candidate strip region, and combining the typical spacing and number of segments of the waveform and text, the candidate strip region is divided into one or more signal line blocks and their corresponding text blocks.
[0018] Step S1-1-4 includes:
[0019] Step S1-1-4-1: Obtain the text region and waveform region corresponding to the same candidate strip region;
[0020] Step S1-1-4-2: Sum the foreground pixel values of each row to obtain the second horizontal projection contour sequence corresponding to the text region and the third horizontal projection contour sequence corresponding to the waveform region.
[0021] Step S1-1-4-3: Determine whether the length of the second horizontal projection contour sequence is within the first pre-configured interval and whether the length of the third horizontal projection contour sequence is within the second pre-configured interval. If yes, the output waveform area is used as the signal line block and the output text area is used as the text block corresponding to the signal line block. Execute step S1-1-4-10. Otherwise, execute step S1-1-4-4.
[0022] Step S1-1-4-4: Initialize the first segmentation detection threshold;
[0023] Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are less than the first segmentation detection threshold to 0, calculate the number of remaining non-zero segments in the second horizontal projection contour sequence as the number of first segments and the length of each first segment, and determine whether the length of all first segments is within the first pre-configured interval. If yes, then execute step S1-1-4-7; otherwise, execute step S1-1-4-6.
[0024] Step S1-1-4-6: Determine whether the first separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5.
[0025] Step S1-1-4-7: Initialize the second segmentation detection threshold;
[0026] Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are less than the second segmentation detection threshold to 0. Calculate the number of remaining non-zero segments in the third horizontal projection contour sequence as the number of second segments and the length of each second segment. Determine whether the length of all second segments is within the second pre-configured interval. If yes, proceed to step S1-1-4-10; otherwise, proceed to step S1-1-4-9.
[0027] Step S1-1-4-9: Determine whether the second separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8.
[0028] Step S1-1-4-10: Output all lines corresponding to each first segment as a signal line block, and output all lines corresponding to the second segment with the closest vertical distance as the text block corresponding to that signal line block.
[0029] The timeline identification process in step S1 includes:
[0030] Step S1-2-1: Use the Hough transform algorithm to inspect the chip timing diagram, and pre-set a length threshold L. min This threshold is used to distinguish valid timelines from noise or non-critical segments. For each detected vertical segment, its length L is calculated. If L ≥ L... min If the line segment is considered to be a major vertical line segment, then it is considered a noise or non-critical line segment and is excluded.
[0031] Step S1-2-2: For the main vertical line segments, filter to obtain timeline blocks by length detection and position detection;
[0032] Step S1-2-3: Perform OCR recognition to extract text above the key timeline blocks, and match it with the time keyword set to obtain the time text corresponding to each timeline block.
[0033] Step S1-2-3 includes:
[0034] Step S1-2-3-1: After determining the location of the key timeline block, perform OCR recognition within a specific area directly above it to extract text. Specifically, set a vertical offset threshold hthreshold to determine the range extending upwards from the top of the timeline block. Simultaneously, set the horizontal range to match the horizontal projection range of the timeline block, ensuring that OCR recognition is performed only within the area horizontally aligned with the timeline block. Perform OCR recognition within this rectangular area.
[0035] Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, then match all the extracted text according to the nearest matching principle to obtain the time text corresponding to each timeline block.
[0036] Step S2 specifically includes:
[0037] Step S2-1: Detect the intersection area of the signal line block and the time line block;
[0038] Step S2-2: After expanding outward by multiple pixels based on each intersection region, multiple image slices are obtained.
[0039] The first classification model is a ResNet state type classifier. The state labeling classification results include at least text states, high level, low level, rising edge, falling edge, and high impedance state. These classification results can be further categorized into two main types: signal states and signal changes. High level, low level, and high impedance state belong to signal states, representing the stable state of the signal at a certain moment; while rising edge and falling edge belong to signal changes, representing the transition of the signal from one state to another. Text states are used to identify text information in image slices, such as signal names or other labeled text.
[0040] Step S4 includes:
[0041] Step S4-1: For each signal line block, randomly select two time line blocks;
[0042] Step S4-2: Concatenate all labeled states located between the two selected timeline blocks in chronological order to construct a set of event sequences;
[0043] Step S4-3: According to the pre-configured text template, compose the first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence.
[0044] The prompt words include:
[0045] Role indicator field: Indicates the role of the primary language model as a time series analysis expert;
[0046] Task type limiting field: Indicates that the task of the first major language model is to focus on temporal behavior analysis and functional verification. The core task is to identify and explain the key temporal constraints, causal relationships and control dependencies between signals based on the information provided by the input data.
[0047] Input format description field: Used to describe in detail the format of the input data obtained based on the first text description;
[0048] Output constraints field: Indicates the format of the output text response.
[0049] A chip timing icon annotation device based on a multimodal large language model includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] 1. First, image slices are obtained through computer vision technology by recognizing rows and columns, which can improve the effectiveness of image slice state annotation. Then, the first text description is constructed by concatenating all state annotations in time sequence, which can provide the textualization of image information of time sequence diagram. The second text description constructed by combining the prompt words can effectively use high-level large language models to understand the temporal logic, interaction relationship and dependency relationship between the first text descriptions of different signals, and generate standardized structured representations as training samples for low-level large language models.
[0052] 2. This method obtains candidate banded regions by combining horizontal projection with peak-valley detection, and further extracts signal line blocks from these regions. Compared to methods that directly set a threshold to obtain signal line regions all at once, this approach automatically adapts to different image qualities and signal line complexities, eliminating the need for a fixed threshold and improving robustness and accuracy. It can accurately identify signal line boundaries, reduce missegmentation, enhance noise resistance, and flexibly handle complex signal line structures, providing a reliable foundation for subsequent annotation and analysis.
[0053] 3. The process of dividing candidate strip regions into one or more signal line blocks and their corresponding text blocks by combining specific waveform and text spacing and segmentation number can effectively solve the problem of noise in the image interfering with the recognition of signal line blocks.
[0054] 4. The Hough transform algorithm is used to detect vertical line segments in the chip timing diagram. Major vertical line segments are filtered by length threshold, excluding noise or non-critical segments shorter than the threshold. Then, based on proximity and slope thresholds, nearby and collinear short line segments are clustered and merged to form complete timeline blocks. Compared to the vertical projection method, this approach is more robust even in the presence of noise or discontinuities. Setting a length threshold can exclude short line segments caused by noise, reducing false identification. Clustering and merging using proximity and slope thresholds further improves the accuracy of line segment identification, ensuring spatial continuity and consistent direction of line segments. These characteristics make this method more robust when dealing with complex images and noise interference, enabling more reliable extraction of timeline blocks and providing a solid foundation for subsequent parsing and annotation. 5. Using time keyword set matching to obtain the text of the timeline blocks can improve the accuracy of text recognition.
[0055] 6. Specially designed prompts clearly indicate the role and task type of the large language model, enabling it to focus on temporal behavior analysis and functional verification, thus improving its understanding of temporal logic. Simultaneously, by detailing the input data format and restricting the output format, the model ensures accurate input parsing and generates high-quality annotation results, reducing manual intervention and improving annotation efficiency. Attached Figure Description
[0056] Figure 1This is a schematic diagram of the main steps of the method of the present invention;
[0057] Figure 2 This is a timing diagram of a certain chip that needs to be analyzed in the embodiments of the present invention. Detailed Implementation
[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0059] A chip timing diagram annotation method based on a multimodal large language model, such as... Figure 1 As shown, it includes:
[0060] Step S1: Identify the row and column lines of the chip timing diagram to obtain signal line blocks and time line blocks respectively;
[0061] like Figure 2 The image shown is a timing diagram of a certain chip. Considering that some chip timing diagrams have low image quality, it is necessary to preprocess the chip timing diagrams first. Specifically, in this embodiment, the process includes:
[0062] 1. Image Enhancement: By applying image processing techniques such as contrast enhancement, sharpening, and noise suppression, the visual quality of the time series image is improved, making signal lines, waveform edges, and text annotations clearer and more distinct, and reducing the interference of blur and noise on subsequent accurate identification.
[0063] 2. Perspective Correction: To address issues such as image tilting and distortion caused by shooting angle or scanning deformation, the system detects key geometric features in the image and calculates the corresponding transformation matrix to restore the deformed image to a standard frontal view, ensuring that the signal lines and time axis are horizontally and vertically straight, thus laying the foundation for subsequent structured analysis.
[0064] 3. Size standardization: To adapt to the requirement of fixed input size for subsequent deep learning models, the original time series graphs of different sizes are uniformly adjusted to the preset dimensions by maintaining the aspect ratio through scaling and necessary padding operations, ensuring the consistency of input data, thereby improving the processing efficiency and generalization ability of the model.
[0065] The signal line identification process in step S1 includes:
[0066] Step S1-1-1: Sum the foreground pixel values of each row in the chip timing diagram to obtain the first horizontal projection contour sequence. The first horizontal projection contour sequence is a one-dimensional array, each element corresponds to a row, the array index represents the vertical position of the image, and the array value represents the total number of foreground pixel values in the corresponding row.
[0067] Step S1-1-2: Obtain significant valley values from the first horizontal projection contour sequence by peak and valley detection, and use the rows corresponding to all significant valley values as separators, and the rows between the separators as candidate strip regions;
[0068] Specifically, the peak and valley detection method can be to differentiate the first horizontal projection contour sequence, find the zero-crossing point of the derivative, and take the rows with the smallest values between the zero-crossing points as potential significant valley values.
[0069] The rows corresponding to significant valleys are used as blank intervals. As long as an appropriate threshold is set, they can be used as separators.
[0070] Step S1-1-3: Sum the foreground pixel values of each column in each candidate strip region to obtain the first vertical projection contour sequence corresponding to each candidate strip region, and segment the candidate strip region into text region and waveform region based on the first vertical projection contour sequence.
[0071] Step S1-1-4: Based on the text region and waveform region corresponding to the same candidate strip region, and combining the typical spacing and number of segments of the waveform and text, the candidate strip region is divided into one or more signal line blocks and their corresponding text blocks. Specifically, this includes:
[0072] Step S1-1-4-1: Obtain the text region and waveform region corresponding to the same candidate strip region;
[0073] Step S1-1-4-2: Sum the foreground pixel values of each row to obtain the second horizontal projection contour sequence corresponding to the text region and the third horizontal projection contour sequence corresponding to the waveform region.
[0074] Step S1-1-4-3: Determine whether the length of the second horizontal projection contour sequence is within the first pre-configured interval and whether the length of the third horizontal projection contour sequence is within the second pre-configured interval. If yes, the output waveform area is used as the signal line block and the output text area is used as the text block corresponding to the signal line block. Execute step S1-1-4-10. Otherwise, execute step S1-1-4-4.
[0075] Step S1-1-4-4: Initialize the first segmentation detection threshold;
[0076] Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are less than the first segmentation detection threshold to 0, calculate the number of remaining non-zero segments in the second horizontal projection contour sequence as the number of first segments and the length of each first segment, and determine whether the length of all first segments is within the first pre-configured interval. If yes, then execute step S1-1-4-7; otherwise, execute step S1-1-4-6.
[0077] Step S1-1-4-6: Determine whether the first separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5.
[0078] Step S1-1-4-7: Initialize the second segmentation detection threshold;
[0079] Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are less than the second segmentation detection threshold to 0. Calculate the number of remaining non-zero segments in the third horizontal projection contour sequence as the number of second segments and the length of each second segment. Determine whether the length of all second segments is within the second pre-configured interval. If yes, proceed to step S1-1-4-10; otherwise, proceed to step S1-1-4-9.
[0080] Step S1-1-4-9: Determine whether the second separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8.
[0081] Step S1-1-4-10: Output all lines corresponding to each first segment as a signal line block, and output all lines corresponding to the second segment with the closest vertical distance as the text block corresponding to that signal line block.
[0082] After each signal line block is successfully detected, OCR technology is used to extract the text content from the text block as the signal line name of that signal line block.
[0083] The timeline identification process in step S1 includes:
[0084] Step S1-2-1: Use the Hough transform algorithm to inspect the chip timing diagram, and pre-set a length threshold L. min This threshold is used to distinguish valid timelines from noise or non-critical segments. For each detected vertical segment, its length L is calculated. If L ≥ L... min If it is true, then the line segment is considered a major vertical line segment; otherwise, it is considered noise or a non-critical line segment and is excluded.
[0085] Step S1-2-2: For the main vertical line segments, calculate the distance between each line segment and other line segments using length detection and position detection. If the distance between two line segments is less than the preset proximity distance threshold D... threshold If the slope difference between the two line segments is less than a preset slope threshold θ, then the two line segments are considered adjacent. Further checks are performed to determine if the adjacent line segments are collinear. threshold If they are collinear, then they are considered to be collinear. All adjacent and collinear short segments are clustered and merged to obtain timeline blocks;
[0086] Step S1-2-3: Perform OCR recognition to extract text near key timeline blocks and match it with the time keyword set to obtain the time text corresponding to each timeline block. Specifically, this includes:
[0087] Step S1-2-3-1: Perform OCR recognition on the image above the timeline block to extract text. Generally, the specific pixel distance can be adjusted according to the actual situation, for example, it can be designed to be proportional to the average vertical pixel length of the signal line block.
[0088] Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, then match all the extracted text according to the nearest matching principle to obtain the time text corresponding to each timeline block.
[0089] Generally, the elements in the time keyword set can include: t0, t1, tINIT2, Reset, etc., and their integrity can be ensured through regular maintenance.
[0090] Step S2: Detect the intersection regions of the signal line block and the time line block, and extract image slices based on each intersection region, specifically including:
[0091] Step S2-1: Detect the intersection area of the signal line block and the time line block;
[0092] Step S2-2: After expanding outward by multiple pixel distances based on each intersection region, multiple image slices are obtained. Generally, in this embodiment, the specific pixel distance can be determined in the following way:
[0093] The pixel distance is proportional to the average vertical pixel length of the signal line block. For example, if the average vertical pixel length of the signal line block is h, the extended pixel distance can be set to 0.5h to ensure that the extracted image slices can cover the intersection of the signal line and the time line and the key information around them.
[0094] Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice. For the state labeling classification result that is a text state, the text content is further identified and used as the labeling state. For other state labeling classification results, the state labeling classification result is used as the labeling state.
[0095] In this embodiment, the first classification model is a ResNet state type classifier. The state labeling classification results include at least text states, high level, low level, rising edge, falling edge, and high impedance state. These classification results can be further categorized into two main types: signal states and signal changes. High level, low level, and high impedance state belong to signal states, representing the stable state of the signal at a certain moment; while rising edge and falling edge belong to signal changes, representing the transition of the signal from one state to another. Text states are used to identify text information in image slices, such as signal names or other labeled text.
[0096] The first classification model can be obtained through training. The training process uses the state labeling classification results corresponding to the image slicing machine. In this embodiment, the ResNet state type classifier is based on the ResNet-18 or ResNet-34 architecture. Its deep network structure can automatically learn and extract robust visual features from the original pixels to distinguish different state types. The output layer of the model uses the Softmax activation function to output the probability that the input slice belongs to each predefined state type.
[0097] Text status belongs to a special status labeling classification result. For this type of result, it is necessary to further identify the corresponding text content through OCR, such as Valid, DES, NOP, etc.
[0098] Step S4: Based on the signal line blocks and time line blocks, and combined with the annotation status of each image slice, construct the first text description.
[0099] This process requires transforming the previously obtained information into coherent natural language or semi-structured text descriptions that are easily understood by large language models. The focus is on clearly expressing time, signals, events, and the known relationships between them. Specific methods are as follows:
[0100] Step S4-1: For each signal line block, randomly select two time line blocks.
[0101] For example, signal line ta and signal line tb can be selected.
[0102] Step S4-2: Concatenate all labeled states located between the two selected timeline blocks in chronological order to construct a set of event sequences;
[0103] For example, the state changes of this signal line at these time points and the time periods in between, such as from high level to low level or a stable state, can be arranged in chronological order to form the event sequence of the signal. The stable state can be NOP, etc.
[0104] Then, the name of each signal is explicitly associated with its corresponding event sequence.
[0105] Step S4-3: According to the pre-configured text template, compose the first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence.
[0106] The first text description generated could be: Signal CMD is in state NOP during the time period td to te.
[0107] Step S5: According to the pre-configured prompt words, combine the first text description to obtain the second text description, and input the second text description into the first language model to obtain the text response returned by the first language model. Based on the text response, construct samples for training the second language model.
[0108] The prompt words include:
[0109] Role indicator field: Indicates the role of the primary language model as a time series analysis expert;
[0110] Task type limiting field: Indicates that the task of the first major language model is to focus on temporal behavior analysis and functional verification. The core task is to identify and explain the key temporal constraints, causal relationships and control dependencies between signals based on the information provided by the input data.
[0111] Input format description field: Used to describe in detail the format of the input data obtained based on the first text description;
[0112] Output constraints field: Indicates the format of the output text response.
[0113] In this embodiment, the first large language model is selected from online cloud-based large language models such as GPT4-o1 and Claude3.7, which have strong text understanding and logical reasoning capabilities, and the constructed Prompt and textualized time-series data are submitted through their API interfaces.
[0114] Then, the first language model performs inference. During this process, the first language model analyzes the signal behaviors described in the text based on the received information and instructions, and infers the potential temporal dependencies.
[0115] In this embodiment, the following example of prompt words is also provided:
[0116] You are currently a senior digital logic design verification engineer, specializing in timing behavior analysis and functional verification. Your core task is to identify and explain the key timing constraints, causal relationships, and control dependencies among the signals, based on the signal interaction sequences extracted and textualized from the chip timing diagram and their timing diagrams provided below.
[0117] The specific format of one of the input data is as follows:
[0118] The behavior of each signal is described in chronological order, in the following format:
[0119] Signal <signal name>:
[0120] - The status is <status value> [optional: event description <description>] at <time point / time period>.
[0121] - Change from state <state A> at <time point A> to state <state B> at <time point B> [Optional: Associate timing parameter <parameter name>]
[0122] Correspondingly, the output format requirements are as follows:
[0123] Please return all identified time-series dependencies as a JSON list. Each dependency object should contain the following fields:
[0124] -"source_signal": "The name of the source signal that triggers or affects it"
[0125] -"target_signal":"Name of the target signal being affected"
[0126] -"dependency_type":"dependency type (e.g., 'triggers_change', 'enables_event', 'data_setup_for', 'follows_state')"
[0127] -"timing_condition": "The time point, time period, or specific condition description of when the dependency occurs"
[0128] -"reasoning": Briefly state your reasons for identifying this dependency.
[0129] For example, the signal input is as follows:
[0130] Signal RESET_n:
[0131] - The state is High from t_start to t_a
[0132] - The transition from High state t_a to Low state t_b is associated with the timing parameter tPW_RESET.
[0133] - Change from state t_b (Low) to state t_c (High)
[0134] Signal CMD_VALID:
[0135] - The state is Low from t_start to t_b
[0136] - When the state changes from Low (t_b) to High (t_c), the event description is the Command becomes valid signal CLK:
[0137] - A periodic square wave with rising edges at t_clk1, t_clk2,...
[0138] Similarly, for example, the output is as follows: [
[0140] {
[0141] "source_signal":"RESET_n",
[0142] "target_signal":"CMD_VALID",
[0143] "dependency_type":"enables_event",
[0144] "timing_condition":"CMD_VALID changes to High after RESET_n changes from Low to High (from t_b to t_c)",
[0145] "reasoning": "The CMD_VALID signal becomes valid (high level) only after the RESET_n signal is released (goes high), indicating that commands are invalid during the system reset period and only become effective after the reset is complete."
[0146] } ]
[0148] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A chip timing diagram annotation method based on a multimodal large language model, characterized in that, include: Step S1: Identify the row and column lines of the chip timing diagram to obtain signal line blocks and time line blocks respectively; Step S2: Detect the intersection region of the signal line block and the time line block, and extract image slices based on each intersection region; Step S3: Input each image slice into the pre-configured first classification model one by one to obtain the state labeling classification result corresponding to each image slice. For the state labeling classification result that is a text state, the text content is further identified and used as the labeling state. For other state labeling classification results, the state labeling classification result is used as the labeling state. Step S4: Based on the signal line block and time line block, and combined with the annotation status of each image slice, construct the first text description; Step S5: According to the pre-configured prompt words, combine the first text description to obtain the second text description, and input the second text description into the first language model to obtain the text response returned by the first language model. Based on the text response, construct samples for training the second language model.
2. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, The signal line identification process in step S1 includes: Step S1-1-1: Sum the foreground pixel values of each row in the chip timing diagram to obtain the first horizontal projection contour sequence; Step S1-1-2: Obtain significant valley values from the first horizontal projection contour sequence by peak and valley detection, and use the rows corresponding to all significant valley values as separators, and the rows between the separators as candidate strip regions; Step S1-1-3: Sum the foreground pixel values of each column in each candidate strip region to obtain the first vertical projection contour sequence corresponding to each candidate strip region, and segment the candidate strip region into text region and waveform region based on the first vertical projection contour sequence. Step S1-1-4: Based on the text region and waveform region corresponding to the same candidate strip region, and combining the typical spacing and number of segments of the waveform and text, the candidate strip region is divided into one or more signal line blocks and their corresponding text blocks.
3. The chip timing diagram annotation method based on a multimodal large language model according to claim 2, characterized in that, Step S1-1-4 includes: Step S1-1-4-1: Obtain the text region and waveform region corresponding to the same candidate strip region; Step S1-1-4-2: Sum the foreground pixel values of each row to obtain the second horizontal projection contour sequence corresponding to the text region and the third horizontal projection contour sequence corresponding to the waveform region. Step S1-1-4-3: Determine whether the length of the second horizontal projection contour sequence is within the first pre-configured interval and whether the length of the third horizontal projection contour sequence is within the second pre-configured interval. If yes, the output waveform area is used as the signal line block and the output text area is used as the text block corresponding to the signal line block. Execute step S1-1-4-10. Otherwise, execute step S1-1-4-4. Step S1-1-4-4: Initialize the first segmentation detection threshold; Step S1-1-4-5: Set all elements in the second horizontal projection contour sequence that are less than the first segmentation detection threshold to 0, calculate the number of remaining non-zero segments in the second horizontal projection contour sequence as the number of first segments and the length of each first segment, and determine whether the length of all first segments is within the first pre-configured interval. If yes, then execute step S1-1-4-7; otherwise, execute step S1-1-4-6. Step S1-1-4-6: Determine whether the first separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the first separation detection threshold and return to step S1-1-4-5. Step S1-1-4-7: Initialize the second segmentation detection threshold; Step S1-1-4-8: Set all elements in the third horizontal projection contour sequence that are less than the second segmentation detection threshold to 0. Calculate the number of remaining non-zero segments in the third horizontal projection contour sequence as the number of second segments and the length of each second segment. Determine whether the length of all second segments is within the second pre-configured interval. If yes, proceed to step S1-1-4-10; otherwise, proceed to step S1-1-4-9. Step S1-1-4-9: Determine whether the second separation detection threshold has reached the pre-configured upper limit. If yes, exit; otherwise, increase the second separation detection threshold and return to step S1-1-4-8. Step S1-1-4-10: Output all lines corresponding to each first segment as a signal line block, and output all lines corresponding to the second segment with the closest vertical distance as the text block corresponding to that signal line block.
4. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, The timeline identification process in step S1 includes: Step S1-2-1: Use the Hough transform algorithm to inspect the chip timing diagram, and pre-set a length threshold L. min This threshold is used to distinguish between valid timelines and noise or non-critical line segments. For each detected vertical line segment, its length L is calculated. If L≥L min If it is true, then the line segment is considered a major vertical line segment; otherwise, it is considered noise or a non-critical line segment and is excluded. Step S1-2-2: For the main vertical line segments, key vertical lines are selected by length detection and position detection, and adjacent and collinear short line segments are clustered and merged to obtain timeline blocks; Step S1-2-3: Perform OCR recognition to extract text above the key timeline blocks, and match it with the time keyword set to obtain the time text corresponding to each timeline block.
5. The chip timing diagram annotation method based on a multimodal large language model according to claim 4, characterized in that, Step S1-2-3 includes: Step S1-2-3-1: After determining the location of the key timeline block, perform OCR recognition in the area directly above it to extract text. Specifically, set a vertical offset threshold h. threshold To determine the area extending upwards from the top of the timeline block, and at the same time, to set the horizontal range to be consistent with the horizontal projection range of the timeline block, ensuring that OCR recognition is only performed within the area horizontally aligned with the timeline block, and to perform OCR recognition within the resulting rectangular area; Step S1-2-3-2: Match the extracted text with the time keyword set. If the extracted text is in the time keyword set, then match all the extracted text according to the nearest matching principle to obtain the time text corresponding to each timeline block.
6. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, Step S2 specifically includes: Step S2-1: Detect the intersection area of the signal line block and the time line block; Step S2-2: After expanding outward by multiple pixels based on each intersection region, multiple image slices are obtained.
7. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, The first classification model is a ResNet state type classifier, and the state label classification results include at least text state, high level, low level, rising edge, falling edge, and high impedance state.
8. The chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, Step S4 includes: Step S4-1: For each signal line block, randomly select two time line blocks; Step S4-2: Concatenate all labeled states located between the two selected timeline blocks in chronological order to construct a set of event sequences; Step S4-3: According to the pre-configured text template, compose the first text description based on the name of the signal line block, the name of the selected timeline block, and the event sequence.
9. A chip timing diagram annotation method based on a multimodal large language model according to claim 1, characterized in that, The prompt words include: Role indicator field: Indicates the role of the primary language model as a time series analysis expert; Task type limiting field: Indicates that the task of the first major language model is to focus on temporal behavior analysis and functional verification. The core task is to identify and explain the key temporal constraints, causal relationships and control dependencies between signals based on the information provided by the input data. Input format description field: Used to describe in detail the format of the input data obtained based on the first text description; Output constraints field: Indicates the format of the output text response.
10. A chip timing icon annotation device based on a multimodal large language model, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-9.
Citation Information
Patent Citations
Large model data intelligent labeling method and system
CN119378564A
Concrete dam defect time sequence image intelligent identification method
CN114913150A
Multi-modal large model training method and system fusing time series data of Internet of Things
CN118296462A
