Image recognition method and device

Abstract

The present invention discloses an image recognition method and device. After obtaining an image, multiple contour blocks in the image are detected, and based on the area of ​​each contour block, these contour blocks are screened to obtain multiple target blocks. Then, a text recognition algorithm is performed on each target block to obtain a corresponding string. And based on the position of each target block in the image and whether the corresponding string includes special characters, the string is filled into the corresponding position in the database.

Description

Technical Field

[0001] The present invention relates to an image processing method and device, and more particularly to an image recognition method and device. Background Art

[0002] With the progress of material science, microstructure has a great impact. Therefore, in order to understand the properties of the material itself, good microscopic analysis techniques and tools are necessary. Electron microscope is one of them. Among them, scanning electron microscope (SEM) has been widely used, which is mainly used to observe the surface morphology of objects. Using scanning electron microscope, it is easy to obtain nano-level imaging and measure it. For example, particle size, fiber diameter and membrane thickness. These analysis results help improve the process in R&D and production, thereby enhancing the results of product development.

[0003] At the factory end, operators will perform manual slicing of products at irregular and regular intervals and send them to the laboratory for processing to obtain SEM images for process monitoring. Afterwards, operators manually download SEM images from the cloud database, manually interpret the data in the SEM images, and manually fill them into spreadsheets to draw charts and perform analysis. Summary of the invention

[0004] The present invention provides an image recognition method and device, which can improve the time of manual processing and enhance the operation efficiency.

[0005] The image recognition method of the present invention includes: obtaining an image; detecting multiple contour blocks in the image; screening these contour blocks based on the area of ​​each contour block to obtain multiple target blocks; performing a text recognition algorithm on each target block to obtain a corresponding string; and filling the string into the corresponding position in the database based on the position of each target block in the image and whether its corresponding string includes special characters.

[0006] In one embodiment of the present invention, the step of selecting the contour blocks based on the area of ​​each contour block to obtain a plurality of target blocks includes: determining whether the area of ​​each contour block is greater than a specified area; and detecting the longest width and the longest height of each contour block whose area is greater than the specified area, and forming a target block with the longest width and the longest height.

[0007] In one embodiment of the present invention, before performing a text recognition algorithm on each target block, the method further includes: comparing the grayscale value of each pixel of each target block with a threshold value; and converting pixels with grayscale values ​​less than the threshold value to black, and converting pixels with grayscale values ​​greater than or equal to the threshold value to white.

[0008] In an embodiment of the present invention, the image recognition method further comprises: for each target block, calculating an average value of grayscale values ​​of a plurality of character regions to serve as the threshold value.

[0009] In one embodiment of the present invention, the step of filling the string into the corresponding position in the database includes: searching for the target block in the specified area of ​​the image to obtain the slice number, point information and slice type; displaying multiple fields corresponding to the slice type to the user interface; reading the strings corresponding to each target block one by one according to the reading order, and determining whether the string includes special characters; if the string includes special characters, determining that the string belongs to the thickness parameter; if the string does not include special characters, determining that the string belongs to the angle parameter; and filling the string into a corresponding field based on the position of each target block in the image determined by the reading order, and whether its corresponding string belongs to the thickness parameter or the angle parameter.

[0010] In one embodiment of the present invention, before detecting the contour block in the image, the method further includes: performing a screenshot process on the image to remove the shooting parameter area in the image.

[0011] In one embodiment of the present invention, the step of detecting the contour block in the image includes: executing an edge detection algorithm on the image to obtain an edge image; executing a binarization process on the edge image to obtain a binary image; and executing a dilation process and an erosion process on the binary image to obtain a contour image, wherein the contour image includes the contour block.

[0012] The image recognition device of the present invention comprises: a storage device storing one or more code snippets; and a processor coupled to the storage device. The processor is configured to read the code snippet to: obtain an image; detect multiple contour blocks in the image; filter the contour blocks based on the area of ​​each contour block to obtain multiple target blocks; perform a text recognition algorithm on each target block to obtain a corresponding string; and fill the string into a corresponding position in a database based on the position of each target block in the image and whether the corresponding string includes special characters.

[0013] Based on the above, through the image recognition method and device, multiple images can be recognized in a short time and written into the database, which greatly shortens the working hours and improves the working efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an image recognition device according to an embodiment of the present invention.

[0015] Figure 2 is a flow chart of an image recognition method according to an embodiment of the present invention.

[0016] Figure 3A to Figure 3D FIG. 4 is a schematic diagram of a target block search process according to an embodiment of the present invention.

[0017] Figure 4 is a schematic diagram of a user interface according to an embodiment of the present invention.

[0018] Figure 5 is a schematic diagram of a query screen according to an embodiment of the present invention.

[0019] Figure 6 is a schematic diagram of a modification screen according to an embodiment of the present invention.

[0020] Figure 7 FIG. 4 is a schematic diagram of an abnormal warning screen according to an embodiment of the present invention.

[0021] Figure 8 FIG. 4 is a schematic diagram of an abnormal warning screen according to an embodiment of the present invention.

[0022] Wherein, the reference numerals are:

[0023] 100: Image recognition device

[0024] 110: Processor

[0025] 120: Storage device

[0026] S205~S225: Steps of image recognition method

[0027] 300: Image

[0028] 300A: Contour Image

[0029] 300B:Filter results

[0030] 300C: Target Image

[0031] 301: Shooting parameter area

[0032] 311~314: text area

[0033] 321~325: Target block

[0034] 401~408: Parameter field

[0035] 410: Image Field

[0036] 421~424: Options DETAILED DESCRIPTION

[0037] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention.

[0038] Figure 1is a block diagram of an image recognition device according to an embodiment of the present invention. Figure 1 , the image recognition device 100 includes a processor 110 and a storage device 120 . The processor 110 is coupled to the storage device 120 .

[0039] The processor 110 is, for example, a central processing unit (CPU), a physical processing unit (PPU), a programmable microprocessor, an embedded control chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other similar devices.

[0040] The storage device 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or a combination of these devices. The storage device 120 stores a plurality of program code segments, which are executed by the processor 110 after being installed to implement the following image recognition method.

[0041] For example, the code snippet is an execution software, and the processor 110 can be set to start the execution software every 12 hours. After the execution software is executed, it will first connect to the cloud server to determine whether there are new images (such as SEM images). If there are no new images in the cloud server, the execution software is closed. If there are new images in the cloud server, the images are downloaded to the storage device 120. Then, the execution software performs the following image recognition method on each downloaded image.

[0042] Figure 2 is a flow chart of an image recognition method according to an embodiment of the present invention. Figure 2 In step S205, the processor 110 obtains an image. Here, the image is, for example, an image of the sample surface generated by scanning the surface of the sample using a focused electron beam using a scanning electron microscope (SEM). The image indicates data such as film thickness parameters and angle parameters.

[0043] After obtaining the image, the processor 110 may first perform a screenshot process on the image to remove the shooting parameter area in the image, and then continue to perform subsequent processing steps. Generally speaking, the SEM shooting parameters and scale information will be displayed below the image obtained by the SEM. In order to avoid misjudgment during the recognition process, the shooting parameter area can be deleted first, and then the target block can be searched.

[0044] The step of searching for the target block includes step S210 and step S215. In step S210, the processor 110 detects a plurality of contour blocks in the image. Next, in step S215, the processor 110 selects the contour blocks based on the area of ​​each contour block to obtain a plurality of target blocks.

[0045] Figure 3A to Figure 3D FIG. 1 is a schematic diagram of a target block search process according to an embodiment of the present invention. Figure 3A to Figure 3D To illustrate the target block search process. Figure 3A As shown, in addition to the text areas 311 to 314 , the image 300 also has a shooting parameter area 301 below it, which is used to mark the shooting parameters and scale information of the SEM. Before starting to search for the target block, the processor 110 first cuts off the shooting parameter area 301 in the image 300 .

[0046] Afterwards, the processor 110 performs an edge detection algorithm on the image of the cut-off parameter area 301 to obtain an edge image. That is, the processor 110 uses the edge detection algorithm to find the edge contours of multiple objects in the image. The edge detection algorithm is, for example, the Sobel algorithm. Next, the processor 110 performs a binarization process on the edge image to obtain a binary image. For example, pixels with a grayscale value less than a critical value are set to 0 (completely black), and pixels with a grayscale value greater than or equal to the critical value are set to 1 (completely white). Afterwards, the processor 110 performs a dilation process and an erosion process on the binary image to obtain the following. Figure 3B The outline image 300A shown here includes a plurality of outline blocks (white blocks).

[0047] After obtaining the contour image 300A, the processor 110 selects these contour blocks based on the area of ​​each contour block to obtain multiple target blocks. Here, a specified area is pre-set for comparison. The processor 110 determines whether the area of ​​each contour block in the contour image 300A is greater than the specified area. If the area of ​​the contour block is not greater than the specified area, it means that the contour block is noise, so the contour block is ignored and is not set as the target block. That is, the contour blocks with an area smaller than the specified area are filtered out, and the contour blocks with an area larger than the specified area are retained, such as Figure 3C Screening result 300B is shown.

[0048] After finding the contour blocks with an area larger than the specified area, the processor 110 detects the longest width and the longest height of these contour blocks in the screening result 300B, and forms the following image with the longest width and the longest height: Figure 3D The target blocks 321-325 in the target image 300C.

[0049] Before performing the text recognition algorithm, the processor 110 may further perform a special noise reduction algorithm on the target block, that is, for each target block, the average grayscale value of multiple character regions is calculated as a threshold value. Then, the processor 110 extracts the grayscale value of the target block from the original image (such as Figure 3A The grayscale values ​​of the corresponding pixels are read from the image 300 shown in the figure, and these grayscale values ​​are compared with the threshold value, and then the pixels with grayscale values ​​less than the threshold value are converted to black, and the pixels with grayscale values ​​greater than or equal to the threshold value are converted to white.

[0050] In step S220, a text recognition algorithm is performed on each target block to obtain a corresponding string. Here, Tesseract, an optical character recognition (OCR) engine, is used to execute the text recognition algorithm. In addition, a whitelist is set in the OCR engine to improve the recognition rate. In addition, the processor 110 first determines whether there are characters in the target block. If there are no characters, the target block is removed. Figure 3D For example, target blocks 321-324 correspond to Figure 3A Therefore, when the processor 110 determines that there is no character in the target block 325, it will remove the target block and only execute the text recognition algorithm on the target blocks 321-324 to obtain the corresponding character string.

[0051] Then, in step S225, based on the position of each target block in the image and whether the corresponding string includes special characters, the string is filled into the corresponding position in the database. First, the processor 110 searches for the target block in the designated area of ​​the image to obtain the slice number, point information and slice type, thereby displaying multiple fields corresponding to the slice type to the user interface. The designated area is, for example, located in the upper left corner of the image.

[0052] For example, Figure 4 is a schematic diagram of a user interface according to an embodiment of the present invention. In this embodiment, Figure 4The user interface shown includes an image field 410, a plurality of parameter fields 401-408, and a plurality of options 421-424. The image field 410 is used to display an image. The image includes 6 target blocks. The processor 110 first extracts a string from the target block in the upper left area (specified area) of the image, that is, obtains the string "6G0242J9Q-A-M1-TH". And, further obtains the slice number, point information, and slice type from the string according to a preset format. Among them, "6G0242J9Q" is the slice number, "A" is the point information, and "M1-TH" is the slice type. The processor 110 fills "6G0242J9Q", "A", and "M1-TH" into fields 401-403 respectively.

[0053] Next, the processor 110 displays a plurality of corresponding columns in the user interface according to the slice type. Table 1 shows that different slice types have different parameters. The processor 110 displays corresponding columns in the user interface according to these parameters.

[0054] Table 1

[0055]

[0056]

[0057] The processor 110 searches Table 1 according to the obtained slice type "M1-TH", and Figure 4 The user interface shown shows parameter columns 404 to 408 corresponding to the parameters "M1 Top Mo", "GSH", "PV", "GSH Taper", and "PV Taper" included in the slice types "M1-TH-h" and "M1-TH-t". After the user confirms that the values ​​automatically filled in each column are correct, he can press option 422 to write these data into the database. Options 423 and 424 are used to select the previous and next images for processing. Option 421 is used to delete the data.

[0058] The processor 110 then reads the strings corresponding to the other target blocks one by one according to the reading order, and determines whether the string includes special characters. In addition, the processor 110 determines the position of each target block in the image according to the reading order, and thereby determines which field to fill the string into according to the position of the target block and the presence or absence of special characters in the string. Specifically, special characters are, for example, "A" and "μm". If the string includes special characters, it is determined that the string belongs to the thickness parameter. If the string does not include special characters, it is determined that the string belongs to the angle parameter. The processor 110 fills the string into the corresponding field based on the position of the target block in the image and whether the corresponding string belongs to the thickness parameter or the angle parameter.

[0059] by Figure 4 For example, the processor 110 can obtain structural information based on the slice type "M1-TH", and determine which layer of data the read string belongs to according to a preset reading order (for example, from top to bottom or from bottom to top), and then determine the position of the target block in the image, and determine whether the string belongs to a thickness parameter or an angle parameter based on whether the string includes special characters, and then fill the currently read string into the corresponding field.

[0060] Through the above method, the user interface can be further designed to have functions such as query screen, modification screen, abnormal warning screen, etc. Figures 5 to 8 To explain. Figure 5 is a schematic diagram of a query screen according to an embodiment of the present invention. Figure 6 is a schematic diagram of a modification screen according to an embodiment of the present invention. Figure 7 FIG. 4 is a schematic diagram of an abnormal warning screen according to an embodiment of the present invention. Figure 8 FIG. 4 is a schematic diagram of an abnormal warning screen according to an embodiment of the present invention.

[0061] use Figure 5 The query screen shown can be used to easily and quickly query the database to view historical data. Figure 6 The modification screen shown in the figure allows the user to modify the abnormal sample. If the processor 110 cannot recognize the image, the user interface will display the following Figure 7 In addition, when the processor 110 determines that the entered value is stuck and may be abnormal, the display is as follows: Figure 8 The abnormal warning screen shown is used to remind the user to make corrections.

[0062] In summary, the image recognition method and device are combined with image processing, OCR engine and special noise reduction algorithm to automatically identify the data in the SEM image and upload the data to the database, so as to draw the electronic design automation (EDA) drawing and analyze it. Accordingly, multiple images can be downloaded for recognition in a short time, and the recognized data can be written into the database, which greatly shortens the working hours and improves the working efficiency. In addition, the user interface can be customized, and the interface and setting files can be adjusted according to different products to lower the threshold of use. Even if the user does not have the basics of data structures such as structured query language (SQL), it can be easily used.

[0063] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image recognition method, characterized in that: include: obtaining an image; Detecting a plurality of contour blocks in the image; Based on the area of ​​each of the contour blocks, the contour blocks are screened to obtain a plurality of target blocks; Performing a text recognition algorithm on each of the target blocks to obtain a corresponding word string; as well as Based on the position of each of the target blocks in the image and whether the corresponding string includes a special character, fill the string into a corresponding position in a database; The steps of filling the string into the corresponding position in the database include: Searching for the target block in a specified area of ​​the image to obtain a slice number, a point information, and a slice type; Displaying a plurality of fields corresponding to the slice type on a user interface; Read the character strings corresponding to the other target blocks one by one according to a reading order, and determine whether the character strings include the special character; If the string includes the special characters "A" and "μm", it is determined that the string belongs to a thickness parameter; If the string does not include the special character, determining that the string belongs to an angle parameter; and Based on the position of each of the target blocks determined by the reading order in the image, and the corresponding string belonging to the thickness parameter or the angle parameter, the string is filled into a corresponding one of the fields, specifically: Based on the position of the target block in the image and whether the corresponding string belongs to a thickness parameter or an angle parameter, the string is filled into the corresponding column; structural information is obtained according to the slice type, and the layer of data of the read string is known according to the preset reading order, so as to judge the position of the target block in the image, and determine whether the string belongs to a thickness parameter or an angle parameter according to whether the string includes special characters, and then fill the currently read string into the corresponding column.

2. The image recognition method according to claim 1, wherein: Based on the area of ​​each of the contour blocks, the step of screening the contour blocks to obtain the target blocks includes: Determining whether the area of ​​each of the contour blocks is greater than a specified area; and A longest width and a longest height of each of the contour blocks whose area is larger than the designated area are detected, and the target block is formed by the longest width and the longest height.

3. The image recognition method according to claim 1, wherein: Before the step of performing the text recognition algorithm on each of the target blocks, the method further includes: comparing the grayscale value of each pixel in each of the target blocks with a threshold value; and The pixels whose grayscale value is less than the threshold value are converted to black, and the pixels whose grayscale value is greater than or equal to the threshold value are converted to white.

4. The image recognition method according to claim 3, wherein: Also includes: For each of the target blocks, an average value of the grayscale values ​​of the plurality of character regions is calculated to serve as the threshold value.

5. The image recognition method according to claim 1, wherein: Before the step of detecting the contour blocks in the image, the method further includes: A cropping process is performed on the image to remove a shooting parameter area in the image.

6. The image recognition method according to claim 1, wherein: The step of detecting the contour blocks in the image comprises: Executing an edge detection algorithm on the image to obtain an edge image; Performing a binarization process on the edge image to obtain a binarized image; and A dilation process and an erosion process are performed on the binary image to obtain a contour image, wherein the contour image includes the contour blocks.

7. An image recognition device, characterized in that: include: a storage device storing one or more program code snippets; as well as a processor, coupled to the storage device, configured to read the program code segment to: obtaining an image; Detecting a plurality of contour blocks in the image; Based on the area of ​​each of the contour blocks, the contour blocks are screened to obtain a plurality of target blocks; Performing a text recognition algorithm on each of the target blocks to obtain a corresponding word string; as well as Based on the position of each of the target blocks in the image and whether the corresponding string includes a special character, fill the string into a corresponding position in a database; The processor is configured to: Searching for the target block in a specified area of ​​the image to obtain a slice number, a point information, and a slice type; Displaying a plurality of fields corresponding to the slice type on a user interface; Read the character strings corresponding to the target blocks one by one according to a reading order, and determine whether the character strings include the special character; If the string includes the special character, determining that the string belongs to a thickness parameter; If the string does not include the special character, determine that the string belongs to an angle parameter; as well as Based on the position of each of the target blocks determined by the reading order in the image, and the corresponding string belonging to the thickness parameter or the angle parameter, the string is filled into a corresponding one of the fields, specifically: The processor fills the string into the corresponding column based on the position of the target block in the image and whether the corresponding string belongs to the thickness parameter or the angle parameter; the processor obtains structural information according to the slice type, and knows which layer of data the read string belongs to according to the preset reading order, and then determines the position of the target block in the image, and determines whether the string belongs to the thickness parameter or the angle parameter according to whether the string includes special characters, and then fills the currently read string into the corresponding column.

8. The image recognition device according to claim 7, wherein: The processor is configured to: determine whether the area of ​​each of the contour blocks is greater than a specified area; and detect a maximum width and a maximum height of each of the contour blocks whose area is greater than the specified area, and form the target block with the maximum width and the maximum height.

9. The image recognition device according to claim 7, wherein: The processor is configured to: compare the grayscale value of each pixel of each of the target blocks with a threshold value; and convert the pixel whose grayscale value is less than the threshold value into black, and convert the pixel whose grayscale value is greater than or equal to the threshold value into white.

10. The image recognition device according to claim 9, characterized in that: The processor is configured to: for each of the target blocks, calculate an average value of the grayscale values ​​of a plurality of character regions to serve as the threshold value.

11. The image recognition device according to claim 7, wherein: The processor is configured to: before detecting the contour blocks in the image, perform a cropping process on the image to remove a shooting parameter area in the image.

12. The image recognition device according to claim 7, wherein: The processor is configured to: execute an edge detection algorithm on the image to obtain an edge image; execute binarization processing on the edge image to obtain a binarized image; and execute a dilation processing and an erosion processing on the binarized image to obtain a contour image, wherein the contour image includes the contour blocks.

Images