Image measurement method and image processing device

Abstract

The embodiments of the disclosure provide an image measurement method and an image processing device. The method comprises: obtaining an input image and a plurality of reference line segments in the input image; encoding the input image into a sequence vector, and integrating the sequence vector and a positional encoding vector into a first input vector; using a transformer encoder to generate a second input vector based on the first input; using a transformer decoder to generate a predicted offset result corresponding to each of a plurality of to-be-measured line segments based on the second input vector and query information; and generating a measurement result of each to-be-measured line segment based on the predicted offset result of each to-be-measured line segment.

Description

Technical Field

[0001] This invention relates to an image processing mechanism, and more particularly to an image measurement method and an image processing apparatus. Prior Technology

[0002] In traditional computer vision techniques, obtaining line segments formed by the locations to be measured in an image typically relies on feature detection and matching techniques. However, these methods often struggle to guarantee stability and accuracy when processing images with high variability or complex backgrounds.

[0003] Furthermore, when the measurement target needs to be adjusted or changed, it usually requires a lot of time and resources to redesign and optimize the detection algorithm, which significantly increases development costs and reduces efficiency. Summary of the Invention

[0004] In view of this, the present invention provides an image measurement method and an image processing apparatus, which can be used to solve the above-mentioned technical problems.

[0005] This invention provides an image measurement method, executed by an image processing device, comprising: acquiring an input image and a plurality of reference line segments in the input image, wherein each reference line segment has a corresponding first endpoint position and a second endpoint position, and the plurality of reference line segments includes at least a plurality of line segments to be measured; encoding the input image into a sequence vector, and integrating the sequence vector with a position encoding vector into a first input vector, wherein the position encoding vector indicates the first endpoint position and the second endpoint position corresponding to each reference line segment; generating a second input vector based on the first input vector using a converter encoder; generating a predicted offset result corresponding to each of the line segments to be measured using a converter decoder based on the second input vector and query information, wherein the query information indicates descriptive information of each of the line segments to be measured; and generating a measurement result for each of the line segments to be measured based on the predicted offset result of each of the line segments to be measured.

[0006] This invention provides an image processing apparatus, including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute: acquiring an input image and a plurality of reference line segments in the input image, wherein each reference line segment has a corresponding first endpoint position and a second endpoint position, and the plurality of reference line segments include at least a plurality of line segments to be measured; encoding the input image into a sequence vector, and integrating the sequence vector with a position encoding vector into a first input vector, wherein the position encoding vector indicates the first endpoint position and the second endpoint position corresponding to each of the reference line segments; generating a second input vector based on the first input vector using a converter encoder; generating a predicted offset result corresponding to each of the line segments to be measured using a converter decoder based on the second input vector and query information, wherein the query information indicates descriptive information of each of the line segments to be measured; and generating a measurement result for each of the line segments to be measured based on the predicted offset result of each of the line segments to be measured. Simple Explanation of the Diagram

[0007] Figure 1 is a schematic diagram of an imaging device according to one embodiment of the present invention. Figure 2 is a flowchart illustrating an image measurement method according to one embodiment of the present invention. Figure 3 is an application scenario diagram illustrating one embodiment of the present invention. Figure 4 is a schematic diagram showing the offset of the first and second endpoints of each line segment to be tested, based on Figure 3. Figure 5 is another application scenario diagram based on Figure 3. Implementation

[0008] Please refer to Figure 1, which is a schematic diagram of an imaging device according to one embodiment of the present invention. In different embodiments, the image processing device 100 may be implemented as various smart devices and / or computer devices, but is not limited thereto.

[0009] In Figure 1, the image processing device 100 includes a storage circuit 102 and a processor 104.

[0010] The storage circuit 102 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record multiple code or modules.

[0011] The processor 104 is coupled to the storage circuit 102 and may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, a processor based on an advanced RISC machine (ARM), and the like.

[0012] In an embodiment of the present invention, the processor 104 can access the modules and program code recorded in the storage circuit 102 to implement the image measurement method proposed in the present invention, the details of which are described below.

[0013] Please refer to Figure 2, which is a flowchart illustrating an image measurement method according to one embodiment of the present invention. The method of this embodiment can be executed by the image processing apparatus 100 of Figure 1. The details of each step in Figure 2 will be explained below with reference to the components shown in Figure 1. To make the concept of the embodiment of the present invention easier to understand, Figure 3 will also be used for explanation below, wherein Figure 3 is an application scenario diagram illustrating one embodiment of the present invention.

[0014] In step S210, the processor 104 obtains the input image 300 and a plurality of reference line segments 311~314, 321~324 in the input image 300, wherein the reference line segments 311~314, 321~324 include at least a plurality of line segments to be tested.

[0015] In the scenario of Figure 3, the input image 300 includes a stitched image formed by stitching together the image under test 31 and the auxiliary image 32. The image under test 31 and the auxiliary image 32 can be stitched together vertically to form the input image 300 (i.e., the stitched image). In other embodiments, the image under test 31 and the auxiliary image 32 can also be stitched together in other ways to form the corresponding stitched image, but it is not limited to this.

[0016] In Figure 3, reference line segments 311-314 and 321-324 include multiple line segments to be tested and multiple auxiliary line segments. The multiple line segments to be tested include, for example, reference line segments 311-314 located in the image to be tested 31, and the multiple auxiliary line segments include, for example, reference line segments 321-324 located in the auxiliary image 32.

[0017] In one embodiment, each line segment to be tested (e.g., reference line segments 311-314) is, for example, a first line segment randomly assigned to the image under test 31. In the scenario of FIG3, the image under test 31 is, for example, an image corresponding to a FinFET. In one embodiment, certain portions (or widths) of the FinFET in the image under test 31 are, for example, targets to be measured.

[0018] In one embodiment, the position and / or length of each line segment to be tested (e.g., reference line segments 311-314) can be set randomly or by the designer according to certain principles, but are not limited thereto.

[0019] In Figure 3, each auxiliary line segment (e.g., reference line segments 321-324) is, for example, a second line segment indicating the corresponding target length in the auxiliary image 32. In the scenario of Figure 3, the auxiliary image 32 is, for example, an image corresponding to another FinFET, and the actual length (or width) of certain parts of this FinFET can be indicated by the corresponding auxiliary line segments (e.g., reference line segments 321-324).

[0020] In other embodiments, the number of line segments to be tested in the image 31 and the number of auxiliary line segments in the auxiliary image 32 can be set according to the designer's needs, and are not limited to the pattern shown in FIG3.

[0021] In one embodiment, each reference line segment 311~314, 321~324 has a corresponding first endpoint position (e.g., left endpoint position) and a second endpoint position (e.g., right endpoint position).

[0022] In step S220, the processor 104 encodes the input image 300 into a sequence vector V00, and integrates the sequence vector V00 with the position encoding vector PV into a first input vector V01.

[0023] In Figure 3, the processor 104 may, for example, feed the input image 300 into a convolutional neural network (CNN) 33 to extract features of the input image 300 as a sequence vector V01 by the CNN 33, but may not be limited to this.

[0024] In one embodiment, the position encoding vector PV can indicate the first endpoint position and the second endpoint position corresponding to each reference line segment 311~314, 321~324.

[0025] Taking reference line segment 311 as an example, its corresponding first endpoint position and second endpoint position can be represented as the pixel coordinates of the left and right endpoints of reference line segment 311 in the image 31 under test. Taking reference line segment 312 as an example, its corresponding first endpoint position and second endpoint position can be represented as the pixel coordinates of the left and right endpoints of reference line segment 312 in the image 31 under test. Taking reference line segment 313 as an example, its corresponding first endpoint position and second endpoint position can be represented as the pixel coordinates of the left and right endpoints of reference line segment 313 in the image 31 under test. Taking reference line segment 314 as an example, its corresponding first endpoint position and second endpoint position can be represented as the pixel coordinates of the left and right endpoints of reference line segment 314 in the image 31 under test.

[0026] Taking reference line segment 321 as an example, its corresponding first endpoint position and second endpoint position can be represented, for example, as the pixel coordinates of the left and right endpoints of reference line segment 321 in auxiliary image 32. Taking reference line segment 322 as an example, its corresponding first endpoint position and second endpoint position can be represented, for example, as the pixel coordinates of the left and right endpoints of reference line segment 322 in auxiliary image 32. Taking reference line segment 323 as an example, its corresponding first endpoint position and second endpoint position can be represented, for example, as the pixel coordinates of the left and right endpoints of reference line segment 323 in auxiliary image 32. Taking reference line segment 324 as an example, its corresponding first endpoint position and second endpoint position can be represented, for example, as the pixel coordinates of the left and right endpoints of reference line segment 324 in auxiliary image 32.

[0027] In this case, the position encoding vector PV in FIG3 may indicate, for example, the pixel coordinates of the left and right endpoints of the reference line segments 311-314 in the image under test 31, and the pixel coordinates of the left and right endpoints of the reference line segments 321-324 in the auxiliary image 32, but may not be limited to this.

[0028] In some embodiments, the aforementioned pixel coordinates may be represented as corresponding vectors in the position encoding vector PV after a certain transformation, but this is not limited to this.

[0029] In one embodiment, after obtaining the sequence vector V00 and the position encoding vector PV, the processor 104 can integrate the sequence vector V00 and the position encoding vector PV into a first input vector V01.

[0030] In some embodiments, the processor 104 may, for example, use any existing method to integrate the sequence vector V00 and the position encoding vector PV into a first input vector V01 (e.g., concatenation), but is not limited thereto.

[0031] In step S230, the processor 104 uses the converter encoder 34 to generate the second input vector V02 based on the first input vector V01.

[0032] In one embodiment, the converter encoder 34 is, for example, a converter encoder in a detection transformer (DETR) model, and its detailed operating principle can be found in existing technical literature related to the DETR model, which will not be repeated here.

[0033] In some embodiments, the second input vector V02 is, for example, a feature matrix modeled by the transformer encoder 34 through a global context, which may contain spatial structure and semantic information, but is not limited thereto.

[0034] In step S240, the processor 104 uses the converter decoder 35 to generate prediction offset results P01~P04 corresponding to each line segment to be tested based on the second input vector V02 and query information Q01~Q04, wherein the query information Q01~Q04 indicates descriptive information of each line segment to be tested.

[0035] In one embodiment, the descriptive information for each line segment to be tested may include a semantic embedding vector corresponding to the name of each line segment to be tested. For example, assuming that the names of reference line segments 311-314 (i.e., the line segments to be tested illustrated in FIG. 3) are "Line Segment to be Tested 1", "Line Segment to be Tested 2", "Line Segment to be Tested 3", and "Line Segment to be Tested 4", the processor 104 may, for example, individually feed the above names into Bidirectional Encoder Representations from Transformers (BERT) to generate a semantic embedding vector corresponding to the name of each line segment to be tested by BERT. In this case, the semantic embedding vector corresponding to the name of each line segment to be tested may be determined as query information Q01-Q04, but is not limited to this.

[0036] In one embodiment, the converter decoder 35 is, for example, the converter decoder in the DETR model, and its detailed operating principle can be found in the existing technical literature related to the DETR model, which will not be repeated here.

[0037] In one embodiment, the plurality of line segments to be tested (e.g., reference line segments 311-314) includes a first line segment to be tested (which is, for example, any of the line segments to be tested), and the predicted offset result corresponding to the first line segment to be tested includes a first offset and a second offset, wherein the first offset corresponds to the first endpoint position of the first line segment to be tested, and the second offset corresponds to the second endpoint position of the first line segment to be tested.

[0038] Taking reference line segment 311 as an example, its corresponding predicted offset result P01 includes, for example, a first offset corresponding to the first endpoint position of reference line segment 311 and a second offset corresponding to the second endpoint position of reference line segment 311. Taking reference line segment 312 as an example, its corresponding predicted offset result P02 includes, for example, a first offset corresponding to the first endpoint position of reference line segment 312 and a second offset corresponding to the second endpoint position of reference line segment 312. Taking reference line segment 313 as an example, its corresponding predicted offset result P03 includes, for example, a first offset corresponding to the first endpoint position of reference line segment 313 and a second offset corresponding to the second endpoint position of reference line segment 313. Taking reference line segment 314 as an example, its corresponding predicted offset result P04 includes, for example, a first offset corresponding to the first endpoint position of reference line segment 314 and a second offset corresponding to the second endpoint position of reference line segment 314, but is not limited to these.

[0039] In one embodiment, each of the first offsets may include, for example, a vertical offset component and a horizontal offset component associated with the corresponding first endpoint position. Similarly, each of the second offsets may include, for example, a vertical offset component and a horizontal offset component associated with the corresponding second endpoint position, but is not limited thereto.

[0040] In one embodiment, the components corresponding to the dashed boxes in FIG3 can be understood as a variant of a DETR model. To enable the variant of the DETR model shown to provide the aforementioned capabilities (e.g., generating predicted offset results P01~P04 in response to the input image 300), during the relevant training process of the DETR model in FIG3, the designer can feed specially designed training data into the DETR model in FIG3 to allow the DETR model in FIG3 to learn accordingly.

[0041] In some embodiments, the training data may include other images with similar properties to the input image 300. That is, each piece of training data may be a stitched image composed of two FinFET images (one of which may correspond to the image to be tested, and the other to an auxiliary image), and may be labeled with multiple auxiliary line segments and multiple line segments to be tested. The labeled auxiliary line segments may be line segments indicating the corresponding target length (e.g., the width at certain specific locations on the FinFET) in the FinFET image, and the labeled line segments to be tested may be multiple line segments randomly assigned, but are not limited to this.

[0042] In addition, each training data set may also include offset results that can offset the corresponding line segments to be tested to the appropriate positions.

[0043] Therefore, after feeding the aforementioned training data into the DETR model in Figure 3 during training, the DETR model in Figure 3 can learn how to correctly offset the line segment to be measured to the appropriate position. Thus, when an input image marked with one or more line segments to be measured is fed into the trained DETR model in Figure 3, the DETR model in Figure 3 can predict / determine how to offset the first and second endpoints of each line segment to be measured in order to correctly measure the corresponding target length, and thus determine the corresponding predicted offset result, but this is not limited to this.

[0044] In step S250, the processor 104 generates the measurement results of each line segment to be measured based on the predicted offset results P01 to P04 of each line segment to be measured (e.g., reference line segments 311 to 314).

[0045] Taking the first line segment to be tested as an example, the processor 104 can offset the position of the first endpoint of the first line segment to be tested according to the corresponding first offset, and offset the position of the second endpoint of the first line segment to be tested according to the corresponding second offset. Then, the processor 104 can determine the distance between the offset first endpoint position and the offset second endpoint position of the first line segment to be tested, and determine this distance as the measurement result corresponding to the first line segment to be tested.

[0046] Please refer to Figure 4, which is a schematic diagram showing the offset of the first and second endpoints of each line segment to be tested, based on Figure 3.

[0047] In Figure 4, the processor 104 may, for example, offset the first endpoint position (e.g., the left endpoint position) of the reference line segment 311 according to the first offset amount in the predicted offset result P01 to form the first endpoint position (e.g., the left endpoint position) of the reference line segment 311a. Additionally, the processor 104 may, for example, offset the second endpoint position (e.g., the right endpoint position) of the reference line segment 311 according to the second offset amount in the predicted offset result P01 to form the second endpoint position (e.g., the right endpoint position) of the reference line segment 311a.

[0048] Then, the processor 104 can determine the distance between the first endpoint position and the second endpoint position of the reference line segment 311a as the measurement result corresponding to the reference line segment 311a (or reference line segment 311).

[0049] Furthermore, the processor 104 may, for example, offset the first endpoint position (e.g., the left endpoint position) of the reference line segment 312 according to the first offset amount in the predicted offset result P02 to form the first endpoint position (e.g., the left endpoint position) of the reference line segment 312a. Additionally, the processor 104 may, for example, offset the second endpoint position (e.g., the right endpoint position) of the reference line segment 312 according to the second offset amount in the predicted offset result P02 to form the second endpoint position (e.g., the right endpoint position) of the reference line segment 312a.

[0050] Then, the processor 104 can determine the distance between the first endpoint position and the second endpoint position of the reference line segment 312a as the measurement result corresponding to the reference line segment 312a (or reference line segment 312).

[0051] For reference segments 313 and 314, the processor 104 can determine the corresponding reference segments 313a and 314a and the corresponding measurement results according to the above description, and the details will not be repeated here.

[0052] As can be seen from the above, the embodiments of the present invention utilize a modified DETR architecture for line segment adjustment, which can effectively capture and analyze the line segment features and their correspondences in the test image 31 and the auxiliary image 32. Therefore, the image processing device 100 can not only learn the line segment configuration in a single image (e.g., auxiliary image 32), but also automatically deduce the optimal mapping position of the corresponding line segment in another image (e.g., test image 31), thereby achieving accurate and automated line segment translation. This innovative method not only improves processing speed and flexibility, but can also be applied to fine-tuning of line segments in the same image.

[0053] It should be understood that although the input image 300 considered in the scenario of Figure 3 is a stitched image formed by stitching together the image to be tested 31 and the auxiliary image 32, in other embodiments, the method proposed in the embodiments of the present invention can also be applied to input images that only include the image to be tested 31.

[0054] Please refer to Figure 5, which is another application scenario diagram based on Figure 3. In the scenario of Figure 5, the input image 500 under consideration only includes the image to be tested 31. In this case, the reference line segments 311 to 314 consist only of the line segments to be tested, and each line segment to be tested is the first line segment randomly assigned to the image to be tested 31.

[0055] Based on this, the processor 104 can perform step S220 based on the input image 500 and the reference line segments 311-314 therein. In step S220, the processor 104 can encode the input image 500 into a sequence vector V00, and integrate the sequence vector V00 with the position encoding vector PV into a first input vector V01.

[0056] In the scenario shown in Figure 5, the position encoding vector PV may indicate only the position of the first endpoint and the position of the second endpoint of each line segment to be tested (i.e., reference line segments 311~314), but it may not be limited to this.

[0057] Afterwards, the processor 104 may continue to execute steps S230 to S240, and the details of each step can be referred to the relevant descriptions in Figures 3 and 4, which will not be repeated here.

[0058] In one embodiment, the components corresponding to the dashed boxes in FIG5 can be understood as a variant of a DETR model. To enable the variant of the DETR model shown to provide the aforementioned capabilities (e.g., generating predicted offset results P01~P04 in response to the input image 500), during the relevant training process of the DETR model in FIG5, the designer can feed specially designed training data into the DETR model in FIG5 to allow the DETR model in FIG5 to learn accordingly.

[0059] In some embodiments, the training data may include other images that have similar properties to the input image 500. That is, each piece of training data may also be a FinFET image, and may contain multiple line segments to be tested, which may be randomly assigned line segments, but are not limited thereto.

[0060] In addition, each training data set may also include offset results that can offset the corresponding line segments to be tested to the appropriate positions.

[0061] Therefore, after feeding the aforementioned training data into the DETR model in Figure 5 during training, the DETR model in Figure 5 can learn how to correctly offset the line segment to be measured to the appropriate position. Thus, when the trained DETR model in Figure 5 is fed with an input image marked with one or more line segments to be measured, the DETR model in Figure 5 can predict / determine how to offset the first and second endpoints of each line segment to be measured in order to correctly measure the corresponding target length, and thus determine the corresponding predicted offset result, but this is not limited to this.

[0062] It should be understood that although the above embodiments use FinFET images as examples for illustration, they are merely illustrative and not intended to limit the possible implementations of the present invention. For scenarios requiring measurements in other types of images, the methods proposed in the embodiments of the present invention can operate based on similar principles to achieve the required measurements.

[0063] In summary, this invention presents a novel end-to-end automated image measurement method. This method directly processes the entire process from image recognition to line segment mapping within the model, eliminating cumbersome post-processing steps and avoiding the high costs of retraining and redesigning post-processing to adapt to changing requirements.

[0064] Furthermore, the DETR variant structure of this invention enables the model to flexibly adapt to various measurement tasks. Unlike traditional methods that rely on explicit object definition and feature extraction, this technique utilizes deep learning to naturally learn the correlation between images and measured line segments, theoretically covering all scenarios that may need to be measured in the future. This design not only reduces the model's dependence on specific scenarios but also improves its adaptability to new and changing tasks.

[0065] Furthermore, due to the high degree of automation and versatility of the embodiments of the present invention, the technology of the present invention can be widely applied in various practical scenarios, such as industrial automation, medical image analysis, and remote sensing image processing. After obtaining sufficient training data, the model can accurately measure new images without further adjustments, greatly reducing operational complexity and cost, and improving work efficiency.

[0066] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0067] 100: Image processing device 102: Storage Circuit 104: Processor S210~S250: Steps 300, 500: Input images 31: Image to be tested 32: Auxiliary images 33:CNN 34: Converter Encoder 35: Converter Decoder 311a~314a, 311~314, 321~324: Reference line segments V00: Sequence Vector V01: First input vector V02: Second input vector Q01~Q04: Information Inquiry P01~P04: Predicted Migration Results PV: Location Encoding Vector

Claims

1. An image measurement method, executed by an image processing device, comprising: The process involves: acquiring an input image and multiple reference line segments within the input image, each reference line segment having a corresponding first endpoint position and a second endpoint position, and the multiple reference line segments including at least multiple line segments to be measured; encoding the input image into a sequence vector, and integrating the sequence vector with a position encoding vector into a first input vector, wherein the position encoding vector indicates the first endpoint position and the second endpoint position corresponding to each reference line segment; generating a second input vector based on the first input vector using a transformer encoder; generating a predicted offset result corresponding to each line segment to be measured using a transformer decoder based on the second input vector and query information, wherein the query information indicates descriptive information for each line segment to be measured; and generating a measurement result for each line segment to be measured based on the predicted offset result.

2. The method as described in claim 1, wherein the input image includes a test image, and the plurality of reference line segments include only the plurality of test line segments, and each test line segment is a first line segment randomly assigned to the test image.

3. The method as described in claim 1, wherein the input image includes a stitched image composed of a test image and an auxiliary image, the plurality of reference line segments include the plurality of test line segments and the plurality of auxiliary line segments, each of the test line segments is a first line segment randomly assigned in the test image, and each of the auxiliary line segments is a second line segment indicating a corresponding target length in the auxiliary image.

4. The method as described in claim 1, wherein encoding the input image into the sequence vector comprises: The input image is input into a convolutional neural network, and the features of the input image are extracted by the convolutional neural network as the sequence vector.

5. The method as described in claim 1, wherein the descriptive information of each of the test segments includes a semantic embedding vector corresponding to the name of each of the test segments.

6. The method as described in claim 1, wherein the plurality of line segments to be tested includes a first line segment to be tested, and the predicted offset result corresponding to the first line segment to be tested includes a first offset and a second offset, wherein the first offset corresponds to the first endpoint position of the first line segment to be tested, and the second offset corresponds to the second endpoint position of the first line segment to be tested.

7. The method of claim 6, wherein generating the measurement result for each of the measured line segments based on the predicted offset result of each of the measured line segments comprises: The first endpoint position of the first line segment to be tested is offset according to the first offset amount, and the second endpoint position of the first line segment to be tested is offset according to the second offset amount; the distance between the offset first endpoint position and the offset second endpoint position of the first line segment to be tested is determined, and the distance is determined to be the measurement result corresponding to the first line segment to be tested.

8. An image processing apparatus, comprising: Storage circuit, which stores program code; The processor, coupled to the storage circuit and accessing the code, executes the following: acquiring an input image and a plurality of reference line segments in the input image, wherein each reference line segment has a corresponding first endpoint position and a second endpoint position, and the plurality of reference line segments include at least a plurality of line segments to be measured; encoding the input image into a sequence vector, and integrating the sequence vector with a position encoding vector into a first input vector, wherein the position encoding vector indicates the first endpoint position and the second endpoint position corresponding to each of the reference line segments; generating a second input vector based on the first input vector using a converter encoder; generating a predicted offset result corresponding to each of the line segments to be measured using a converter decoder based on the second input vector and query information, wherein the query information indicates descriptive information of each of the line segments to be measured; and generating a measurement result for each of the line segments to be measured based on the predicted offset result for each of the line segments to be measured.

Images