Image identification method using two-stage inference procedure

Abstract

This invention discloses an image recognition method employing a two-stage inference procedure. The image recognition method of this invention includes the following steps: capturing an image of the object to be detected; using a target detection model to detect multiple target images within multiple initially selected marked regions on the image of the object to be detected, thereby defining multiple first-order recognition target blocks on the image of the object to be detected, wherein the multiple target images are respectively within the multiple first-order recognition target blocks; and executing an unsupervised learning algorithm to identify the edges of the multiple target images, thereby converging the boundaries of the multiple first-order recognition target blocks, and using the converged boundaries of the multiple first-order recognition target blocks as the contours of the multiple target images detected by second-order inference.

Description

Image recognition method using a two-stage inference procedure The present invention relates to an image recognition method, and in particular to an image recognition method using a two-stage inference procedure. In recent years, the increasing application of image recognition technology in various fields has led to renewed interest in artificial neural networks (ANNs). This research is often referred to as deep learning, computer learning, and similar terms. Recent applications of ANNs include speech and image recognition. However, current image recognition technology requires manual dragging of multiple lines on the image toolbar to form a polygonal frame before execution. The polygonal frame is then used to roughly frame the target object to be identified in the image before accurate detection can be achieved. This method is extremely inconvenient, time-consuming, and labor-intensive. The present invention addresses the deficiencies of the prior art and provides an image recognition method using a two-stage inference procedure. The image recognition method using a two-stage inference procedure of the present invention includes the following steps: capturing an image of an object to be detected as an image of the object to be detected; using a target detection model to detect multiple detection target images within multiple initial selected mark areas on the image of the object to be detected, generating multiple first-order recognition target blocks, marking the multiple initial selected mark areas on the image of the object to be detected so that the multiple detection target images are respectively within the multiple first-order recognition target blocks; and executing an unsupervised learning algorithm to identify the edges of the multiple detection target images, converge the boundaries of the multiple first-order recognition target blocks, and use multiple second-order recognition mark blocks along the boundaries of the multiple first-order recognition target blocks after convergence to mark the multiple detection target images of the second-order recognition. As described above, the present invention provides an image recognition method that utilizes a two-stage inference procedure. In the image recognition method that utilizes a two-stage inference procedure of the present invention, the initial image marking method employed is to simply circle the object to be recognized by dragging a single rectangular point, then perform image recognition using a trained target detection model in the first inference stage, and finally, in the second inference stage, accurately identify the contours of the initially detected target object using an unsupervised clustering algorithm. The image recognition method that utilizes a two-stage inference procedure of the present invention utilizes a staged algorithm, but is more efficient than traditional image recognition algorithms and takes less time. To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and illustration and are not intended to limit the present invention. The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and each detail in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual dimensions. It is stated in advance. The following embodiments will further explain the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention. In addition, the term "or" used in this article may include any one or more combinations of the associated listed items depending on the actual situation. Please refer to Figures 1, 4 and 10, where Figure 1 is a schematic diagram of cavitations in the solder on a circuit board detected by an image recognition method using a two-stage inference procedure according to an embodiment of the present invention, Figure 4 is a flow chart of the steps of the image recognition method using a two-stage inference procedure according to an embodiment of the present invention, and Figure 10 is a block diagram of an image recognition system using a two-stage inference procedure according to an embodiment of the present invention. The image recognition method using a two-stage inference process according to an embodiment of the present invention may include steps S101 to S107 as shown in FIG1 , for example but not limited to being executed using an image recognition system (including an image capture device 100 and a processor 200 ) using a two-stage inference process as shown in FIG10 . The image recognition method using a two-stage inference process in an embodiment of the present invention is applied to detecting whether an image of a detection target appears in an image of one or more objects to be inspected. For example, it is applied to detecting whether voids (OD) appear in solder (e.g., solder) on a circuit board CT (e.g., a printed circuit board) as shown in FIG4 . It should be understood that the present invention is not limited to the size or number of images of the objects to be inspected. In step S101, the image capture device 100, such as but not limited to any device capable of capturing or shooting images, such as a camera or a mobile phone, captures an image of the object to be inspected as the object to be inspected image. For example, the image of the circuit board CT shown in FIG4 is captured, including images of the solder SR at the soldering points of circuit components, wires, pads, etc., and images of the surrounding areas. The image of the object to be inspected captured by the image capture device 100 may include multiple images of the entire area on the circuit board CT or multiple images of each of the multiple divided areas captured under the same environmental factors. It may also include multiple images of the entire area on the circuit board CT or multiple images of each of the multiple divided areas captured under different environmental factors such as different brightness or different color illumination. In step S102 , the processor 200 uses an object detection model, such as but not limited to a neural network model. For example, the steps of training and establishing this target detection model may include: the processor 200 collects multiple reference object images of multiple reference objects (for example, images of other circuit boards that have been inspected); the processor 200 identifies multiple reference object features common to multiple reference object images on the multiple reference object images (for example, multiple cavitation images in solder) as multiple reference basic features; and the processor 200 trains and establishes a target detection model based on the multiple reference basic features. In step S103 , the processor 200 uses an object detection model to detect a plurality of detection target images within a plurality of initially selected mark areas on the image of the object to be detected (eg, an image on a circuit board). In step S104, the processor 200 selects positions of a plurality of first-order recognition target blocks on the image of the object to be detected based on a detection result generated by detecting the current image of the object to be detected using the target detection model, and defines a plurality of first-order recognition target blocks. The plurality of detection target images are respectively within the defined plurality of first-order recognition target blocks. For example, the processor 200 can detect whether there are multiple reference basic features specified by the above-mentioned target detection model (such as multiple reference object features common to multiple cavitation images in solder on multiple circuit boards) in multiple initially selected marked areas and other areas on the image of the object to be tested; and each area on the image of the object to be tested where multiple reference basic features exist is used as a first-order recognition target block. In step S105 , the processor 200 executes an unsupervised learning algorithm, such as but not limited to a k-means algorithm. In step S106 , the processor 200 converges boundaries of a plurality of first-order recognition target blocks toward edges of a plurality of detection target images. The above-mentioned execution of the unsupervised learning algorithm, in which the processor 200 converges the boundaries of multiple first-order recognition target blocks toward the edges of multiple detection target images, includes the following procedures. First, the processor 200 repeatedly selects one of the multiple pixel points on the image of the object to be tested, including selecting one of the multiple pixel points in each of the multiple first-order recognition target blocks, and in particular, selecting one of the multiple pixel points constituting the reference basic features, as a center point. Then, the processor 200 calculates the distance between each other pixel point on the image of the object to be tested and this center point. Then, the processor 200 classifies those whose distance is less than a distance threshold value into the same group, and calculates the average distance of each group. Then, the processor 200 reselects the center point of each group based on the average distance of each group. The processor 200 executes the above-mentioned procedure with the updated center point multiple times. In this way, based on the center point obtained each time, an unsupervised learning algorithm is used to perform classification and grouping. According to the distribution of each group, the boundaries of the multiple first-order recognition target blocks are converged multiple times towards the edges of multiple detection target images on the image of the object to be detected. In step S107 , the processor 200 uses the boundaries of the multiple first-order recognition target blocks that have finally converged as the contours of the multiple detection target images detected by the second-order inference. Please refer to Figures 1, 2, 5, 6 and 10, where Figure 2 is a step flow chart of the image recognition method using a two-stage inference procedure according to an embodiment of the present invention, Figure 5 is a schematic diagram of the image of the object to be tested detected by the image recognition method using a two-stage inference procedure according to an embodiment of the present invention, and Figure 6 is a schematic diagram of using multiple initial marking patterns for marking in the image recognition method using a two-stage inference procedure according to an embodiment of the present invention. The image recognition method using a two-stage inference procedure according to an embodiment of the present invention may further include steps S201 to S203 as shown in FIG. 2 , for example but not limited to being executed by an image recognition system using a two-stage inference procedure as shown in FIG. 10 (including a processor 200 , an image capture device 100 , and a database 300 ) before the above-mentioned step S101 . In step S201 , the processor 200 receives a plurality of initial tagging requests (from a user interface of an electronic device used by a user). In step S202, the processor 200 selects a plurality of initial marking patterns according to the received plurality of initial marking requirements. In step S203 , the processor 200 places a plurality of initial marking patterns on the image of the object to be detected, respectively marking a plurality of initial selected marking areas respectively specified by the received plurality of initial marking requirements, such as but not limited to framing or circling a plurality of detection target images and the images around them. For example, a user may operate an electronic device to display an image of an object under test IM0 as shown in FIG5 . When the user can visually discern the locations of multiple inspection target images, such as the multiple cavitation images OD1 to OD16 in the solder material in the image of the object under test IM0 shown in FIG5 , the user may select several of the multiple initial marking pattern options in the toolbar of the user interface to generate multiple initial marking requests to be output to the processor 200. Next, the processor 200 may generate corresponding multiple initial marking patterns on the image of the object to be tested displayed on the user interface of the electronic device according to these multiple initial marking requirements to mark multiple initial selected marking areas respectively. For example, as shown in FIG. 6 , multiple initial marking patterns ET1 to ET16 are generated on the image of the object to be tested IM0 to mark / frame the cavitation images OD1 to OD16 respectively to form the image of the object to be tested IM1. If necessary, the user can also operate the electronic device to drag multiple initial marking patterns on the user interface to generate multiple frame dragging requests to be output to the processor 200, so as to request the processor 200 to drag the multiple initial marking patterns to multiple designated detection positions respectively, so as to mark (for example, frame or circle) multiple initial selection marking areas respectively. If necessary, the user can also operate the electronic device to adjust the sizes of multiple initial marking patterns used to mark multiple initial selection marking areas on the image of the object to be tested, so as to request the processor 200 to adjust the shapes and sizes of the multiple initial marking patterns on the image of the object to be tested displayed on the user interface of the electronic device. For example, but not limited to, using 16 initial marking patterns ET1~ET16 of different sizes on the image of the object to be tested IM1 as shown in Figure 6 to respectively select 16 areas with cavitation images OD1~OD16 (i.e., initial selection marking areas). As shown in Figure 5, in an embodiment of the present invention, rectangular hollow frames and square hollow frames are used as examples of multiple initial marking patterns for framing or selecting multiple initial selected marking areas. In practice, other patterns can be used as the multiple initial marking patterns to frame or circle multiple initial selected marking areas based on user needs. It should be understood that the present invention is not limited to the shape and size of the multiple initial marking patterns. After selecting a plurality of initial selected marking regions using a plurality of initial marking patterns as described above, ie, after executing steps S201 to S203 , the aforementioned steps S101 to S107 may be executed in sequence to accurately detect the contours of a plurality of detection target images. Please refer to FIG. 1 , FIG. 3 and FIG. 10 , wherein FIG. 3 is a flow chart of the steps of an image recognition method using a two-stage inference procedure according to an embodiment of the present invention. The image recognition method using a two-stage inference process according to the embodiment of the present invention may further include steps S301 to S305 , which are executed by, for example but not limited to, the image recognition system using a two-stage inference process (including the processor 200 , the image capture device 100 , and the database 300 ) as shown in FIG. 10 . After defining a plurality of first-order recognition target blocks (the aforementioned step S104 ), steps S301 and S302 may be performed sequentially. At step S301, the processor 200 selects a plurality of first-order identification mark patterns. In step S302 , the processor 200 uses a plurality of first-order recognition marking patterns to mark the plurality of first-order recognition target blocks on the image of the object to be detected, such as but not limited to framing or circling the plurality of first-order recognition target blocks. After marking a plurality of first-order recognition target blocks respectively using a plurality of first-order recognition marking patterns (step S302 ), the aforementioned steps S105 to S107 may be sequentially executed to detect contours of a plurality of detection target images through second-order inference. At step S303 , the processor 200 selects a plurality of second-order recognition mark tiles. In step S304 , the processor 200 uses a plurality of second-order recognition mark blocks to mark the contours of a plurality of detection target images detected by the second-order inference on the image of the object to be detected, for example, marking the periphery of a bubble in solder or the inner edge of a hollow solder. In step S305, the processor 200 uses a plurality of second-order recognition markers on the image of the object to be tested to mark the interior of the contours of the plurality of detection target images detected by the second-order inference. For example, this may mark the entire cavity in the solder, or more practically, mark the solder with a hollow portion (the portion of the solder not marked in the middle is a cavity). In other words, the plurality of second-order recognition markers cover the entire area of ​​the plurality of detection target images (e.g., the plurality of cavity images) detected by the second-order inference. It should be understood that the present invention may adjust or delete some of the steps described herein according to actual needs, for example, only executing step S304 or only executing step S305. Please refer to Figures 1 to 10, where Figure 7 is a schematic diagram of an image recognition method using a two-stage inference procedure to execute a first-stage inference procedure according to an embodiment of the present invention, Figure 8 is a schematic diagram of an image recognition method using a two-stage inference procedure to execute a second-stage inference procedure according to an embodiment of the present invention, and Figure 9 is a schematic diagram of an image recognition method using a two-stage inference procedure according to an embodiment of the present invention. The image recognition method using the two-stage inference process according to the embodiment of the present invention sequentially executes the first stage inference process and the second stage inference process as shown in FIG9 , which are described in detail as follows. First, as in FIG6, FIG7 and FIG9, the first stage inference procedure is performed. During the first-stage inference process, the processor 200 uses a plurality of initial marking patterns to mark a plurality of cavitation images OD1-OD16 in the solder in the object image IM1. The processor 200 then compares features of a plurality of regions in the object image IM1 with a plurality of reference base features of a plurality of reference object images IMref of the target detection model MD, particularly the plurality of initially selected marking regions marked with the plurality of initial marking patterns. The processor 200 then uses the plurality of initially selected marking regions with the plurality of reference base features as a plurality of first-order recognition target blocks and marks the plurality of first-order recognition target blocks in the object image IM1 with a plurality of first-order recognition marking patterns RN, thereby forming a plurality of first-order recognition mark images D1 (the original image with relatively blurred edges extracted through the first-stage inference process) as shown in FIG8 , thereby generating the object image IM2. Next, as shown in FIG8 and FIG9, the second stage inference procedure is performed. During the second-stage inference process, processor 200 executes an unsupervised learning algorithm (e.g., a clustering algorithm) to converge the boundaries of the object image IM2 toward the edges of the multiple detection target images. During these multiple convergence processes, the clarity of the object image IM2 increases. As shown in FIG8 , after the second-stage inference process, multiple detection target images D2 are formed on the object image IM2, each clearly demarcated by the clustering algorithm. Finally, as shown in Figures 8 and 9, the image recognition method using a two-stage inference process according to the present embodiment of the present invention generates an object image IM3 after executing the first and second stage inference processes on the object image IM0. Multiple detection target images D2, representing multiple cavitation images (i.e., multiple detection targets in this embodiment) detected in the object image IM3, can be labeled with multiple second-order identification marker images TN, as shown in Figure 8, to form multiple second-order identification marker images D3. If necessary, the above-mentioned object under test image IM0 , object under test image IM1 , object under test image IM2 , and object under test image IM3 may be stored in the database 300 . In practice, the image recognition method using the two-stage inference process of the present invention may further include the following steps. The processor 200 can calculate the area of ​​each of the multiple inspection target images detected by the second-order inference. The processor 200 can further calculate the actual area of ​​each of the multiple inspection targets (e.g., multiple cavitations in multiple solders at multiple solder joints) based on the area of ​​each inspection target image and relevant data such as the shooting distance between the image capture device 100 and the circuit board. The actual area can be stored in the database 300. Furthermore, the processor 200 may obtain images of the object to be tested of the same solder pile taken at different angles and execute the above-mentioned detection and calculation results after the image recognition method using the two-stage inference process of the present invention is used to calculate the volume of each of multiple detection targets (e.g., multiple cavitations in multiple solders at multiple welding points), which may be stored in the database 300. If necessary, the processor 200 may compare the area of ​​each of the multiple detection targets (e.g., multiple cavities in multiple solders at multiple welding points) with an area threshold value, so as to store relevant data such as the area size, volume size, and distribution position on the circuit board of each detection target (e.g., each cavitation in multiple solders at multiple welding points) having an area greater than an area threshold value as process adjustment reference information in the database 300. Additionally or alternatively, the processor 200 may calculate the area of ​​each of the plurality of solders of the plurality of solder joints, calculate the ratio of the area of ​​the cavitation within the solder of each solder joint to the solder, or calculate the ratio of the cavitation within the solder of each solder joint to the entirety (including the solder of each solder joint and the cavitation at its center), and compare the ratio with a cavitation ratio threshold value, so as to store the above-mentioned data, such as the area and ratio, of all cavitations having a ratio greater than a cavitation ratio threshold value and the solder in which they are located, in the database 300 as reference information for process adjustment. Additionally or alternatively, the processor 200 may calculate the volume of each of the plurality of solders, calculate the ratio of the volume of the cavitation within the solder of each soldering point to the volume of the solder, or calculate the ratio of the cavitation within the solder of each soldering point to the overall volume (including the solder of each soldering point and the cavitation at its center), and compare the ratio with a cavitation ratio threshold value, so as to store the above-mentioned volume and ratio and other related data of all cavitations having a ratio greater than a cavitation ratio threshold value and the solder in which they are located in as reference information for process adjustment in the database 300. In summary, the present invention provides an image recognition method using a two-stage inference procedure. In the image recognition method using a two-stage inference procedure of the present invention, the initial image marking method is to circle the object to be identified by simply dragging a rectangle, then use the trained target detection model to perform image recognition in the first inference stage, and finally use an unsupervised clustering algorithm in the second inference stage to accurately identify the outline of the initially detected target object. The image recognition method using a two-stage inference procedure of the present invention uses a staged algorithm, but it is more efficient than traditional image recognition algorithms and takes less time. The contents disclosed above are only preferred feasible embodiments of the present invention and do not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the contents of the description and drawings of the present invention are included in the scope of the patent application of the present invention. S101~S107, S201~S203, S301~S305: Step CT: Circuit board OD: Cavitation SR: Solder IM0: Test object image OD1~OD16: Cavitation images ET1~ET16: Initial marking pattern IM1: Test object image IM2: Test object image IMref: Reference object image MD: Target detection model RN: First-order recognition marking pattern D1: First-order recognition marking image D2: Detection target image D3: Second-order recognition marking image IM3: Test object image TN: Second-order recognition marking pattern Block 100: Image capture device 200: Processor 300: Database FIG1 is a flowchart of an image recognition method using a two-stage inference process according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the steps of an image recognition method using a two-stage inference process according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating the steps of an image recognition method using a two-stage inference process according to an embodiment of the present invention. FIG. 4 is a schematic diagram of cavitation in solder on a circuit board detected by an image recognition method using a two-stage inference procedure according to an embodiment of the present invention. FIG. 5 is a schematic diagram of an image of an object to be detected by an image recognition method using a two-stage inference procedure according to an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating a method for image recognition using a two-stage inference process and performing marking using multiple initial marking patterns according to an embodiment of the present invention. FIG. 7 is a schematic diagram illustrating an image recognition method using a two-stage inference process and executing a first-stage inference process according to an embodiment of the present invention. FIG. 8 is a schematic diagram illustrating the second-stage inference process of the image recognition method using the two-stage inference process according to an embodiment of the present invention. FIG. 9 is a schematic diagram of an image recognition method using a two-stage inference procedure according to an embodiment of the present invention. FIG. 10 is a block diagram of an image recognition system using a two-stage inference process according to an embodiment of the present invention. S101~S107: Steps

Claims

1. An image recognition method employing a two-stage inference procedure, comprising the following steps: using an image capturing device to capture an image of an object to be detected, as an object image to be tested; using a processor and a target detection model to detect multiple target images within multiple initially selected marked regions on the object image to be tested, generating a detection result, thereby defining multiple first-order recognition target blocks on the object image to be tested, wherein the multiple target images are respectively located within the multiple first-order recognition target blocks; and using the processor to execute an unsupervised learning algorithm to identify the edges of the multiple target images to be detected, converge the boundaries of the multiple first-order recognition target blocks, and use the converged boundaries of the multiple first-order recognition target blocks as the contours of the multiple target images detected by second-order inference; The step of the processor executing the unsupervised learning algorithm to converge the boundaries of multiple first-order recognition target blocks toward the edges of multiple detected target images includes the following procedure: repeatedly selecting one of the multiple pixels of the object image to be tested as a center point, calculating a distance between each of the other pixels and the center point, classifying those distances less than a distance threshold into the same group among multiple groups, calculating a distance average of the multiple distances for each group, reselecting the center point of each group based on the distance average of each group, performing group classification by repeating the above procedure, and gradually converging the boundaries of each first-order recognition target block toward the edges of the detected target image multiple times according to the distribution of each group.

2. The image recognition method employing a two-stage inference procedure as described in claim 1 further includes the following steps: using the image capturing device to capture an image on a circuit board as the image of the object under test.

3. The image recognition method employing a two-stage inference procedure as described in claim 1 further includes the following steps: The processor uses a type of neural network model as the target detection model.

4. The image recognition method employing a two-stage inference procedure as described in claim 1 further includes the following steps prior to using the target detection model: using the processor, based on multiple initial marking requirements, multiple initial marking patterns are used to mark multiple initial selection marking regions on the image of the object to be tested.

5. The image recognition method employing a two-stage inference procedure as described in claim 4, wherein the step of marking multiple initial selection mark regions on the image of the object to be tested using multiple initial mark patterns comprises: using the processor, according to multiple frame dragging requirements, dragging multiple initial mark patterns to multiple designated detection positions to mark multiple initial selection mark regions respectively.

6. The image recognition method employing a two-stage inference procedure as described in claim 5, wherein the step of marking multiple initial selection mark regions on the image of the object to be tested using multiple initial mark patterns further comprises: using the processor, adjusting the size and shape of the multiple initial mark patterns used to mark the multiple initial selection mark regions according to multiple frame range adjustment requirements.

7. The image recognition method employing a two-stage inference procedure as described in claim 1 further includes the following steps: using the processor, marking multiple first-order recognition mark patterns on the image of the object to be tested, respectively marking multiple first-order recognition target blocks.

8. The image recognition method employing a two-stage inference procedure as described in claim 7 further includes the following steps performed after executing the unsupervised learning algorithm: using the processor, replacing the plurality of the plurality of the first-order recognition marker patterns with a plurality of second-order recognition marker patches on the image of the object to be tested; using the processor, marking the contours of the plurality of detected target images on the image of the object to be tested along the boundaries of the converged plurality of the plurality of first-order recognition target blocks with the plurality of the second-order recognition marker patches; and using the processor, marking the interior of the contours of the plurality of detected target images on the image of the object to be tested with the plurality of the second-order recognition marker patches.

9. The image recognition method employing a two-stage inference procedure as described in claim 8 further includes the following steps: using the processor, calculating the area size of each of the plurality of detected target images detected by second-order inference.

Images