Image recognition method and system

Abstract

The present invention discloses an image recognition method and system. The method identifies wrist movement trajectories in an image sequence captured by an image capturer. Based on the wrist movement trajectory, the number of wrist movements and the distance of each wrist movement are calculated. A determination is made as to whether the number of movements meets a preset number and whether the distance of each movement is within a preset range. If the number of movements does not meet the preset number, or if the distance of at least one movement is outside the preset range, a re-execution notification is issued. If the number of movements meets the preset number, and the distance of each movement is within the preset range, the cleaning action is determined to be complete.

Description

Technical Field

[0001] The present invention relates to an image recognition mechanism, and more particularly to an image recognition method and system for monitoring personnel. Background Art

[0002] The manufacture of semiconductor devices is usually carried out by repeatedly depositing insulating or dielectric layer materials, conductive layer materials, and semiconductor layer materials on a semiconductor substrate, and patterning the various material layers using a lithography process to form circuit components and components on the semiconductor substrate. In the above-mentioned lithography process, a photomask is generally used to pattern the material layer. The so-called photomask refers to a product made by forming a fine pattern at the micron level on a glass plate. Generally speaking, before using the photomask, it is necessary to ensure that there are no fine particles attached to the surface to avoid affecting the process yield, so personnel are required to clean the photomask. While cleaning the photomask, another person is also required to supervise the operation status, such as confirming electrostatic protection, using an anti-static gun, and whether the cleaning action is carried out. Summary of the Invention

[0003] The present invention provides an image recognition method and system, which can confirm whether a person performs a cleaning action correctly and remind the person to do so.

[0004] The image recognition method of the present invention executes multiple steps through a processor, including: identifying a wrist movement trajectory in an image sequence captured by an image capturer, wherein the image sequence is captured when a user performs a cleaning action; based on the wrist movement trajectory, calculating the number of wrist movements and the movement distance of each wrist movement, wherein the number of movements is obtained by counting the number of times the wrist point and the shoulder point intersect; determining whether the number of movements meets a preset number and whether the movement distance of each movement is within a preset range; if the number of movements does not meet the preset number, or the movement distance of at least one movement is not within the preset range, issuing a re-execution notification; and if the number of movements meets the preset number and the movement distance of each movement is within the preset range, determining that the cleaning action is completed.

[0005] In one embodiment of the present invention, the image recognition method further includes executing the following steps through a processor: before identifying the wrist movement trajectory, driving an object detection model to perform object detection on the initial image obtained by the image capturer to determine whether the anti-static device overlaps with the anti-static box; and after determining that the anti-static device overlaps with the anti-static box, driving a posture evaluation model to identify the wrist movement trajectory in the image sequence.

[0006] In one embodiment of the present invention, the step of driving a posture evaluation model to identify a wrist movement trajectory in an image sequence includes: detecting a plurality of skeletons corresponding to a plurality of images in the image sequence through the posture evaluation model; and obtaining the wrist movement trajectory based on wrist points and shoulder points in these skeletons.

[0007] In one embodiment of the present invention, the image recognition method further includes executing the following steps by a processor: performing a smoothing process on the wrist movement trajectory to calculate the number of wrist movements and the movement distance of each wrist movement based on the smoothed wrist movement trajectory.

[0008] In one embodiment of the present invention, the step of determining whether the number of movements meets the preset number and whether the moving distance of each movement is within the preset range includes: determining, by a processor, whether the number of movements within a time interval meets the preset number; if the number of movements within the time interval does not meet the preset number, issuing a re-execution notification by the processor; if the number of movements within the time interval meets the preset number, determining, by the processor, whether multiple moving distances obtained within the time interval are all within the preset range; if at least one moving distance obtained within the time interval is not within the preset range, issuing a re-execution notification by the processor; and if all moving distances obtained within the time interval are within the preset range, determining, by the processor, that the cleaning action is completed.

[0009] The image recognition system of the present invention includes: an image capturer and a processor. The processor is coupled to the image capturer. The processor is configured to: identify the wrist movement trajectory in the image sequence captured by the image capturer, wherein the image sequence is taken when the user performs a cleaning action; based on the wrist movement trajectory, calculate the number of wrist movements and the movement distance of each wrist movement, wherein the number of movements is obtained by calculating the number of times the wrist point and the shoulder point intersect; determine whether the number of movements meets the preset number and whether the movement distance of each movement is within the preset range; if the number of movements does not meet the preset number, or the movement distance of at least one movement is not within the preset range, issue a re-execution notification; and if the number of movements meets the preset number and the movement distance of each movement is within the preset range, determine that the cleaning action is completed.

[0010] Based on the above, the present disclosure utilizes an image capture device to capture a sequence of images of a person performing cleaning actions and then performs image recognition algorithms on these images. This allows for verification of the correctness of the cleaning action and prompts the user to proceed. This reduces the risk of improper cleaning due to negligence, improves cleaning effectiveness, and eliminates the need for additional supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1is a block diagram of an image recognition system according to an embodiment of the present invention.

[0012] Figure 2 is a flow chart of an image recognition method according to an embodiment of the present invention.

[0013] Figures 3A to 3D FIG. 1 is a schematic diagram of a human skeleton according to an embodiment of the present invention.

[0014] Figure 4 is a graph of a movement trajectory according to an embodiment of the present invention.

[0015] Wherein, the reference numerals:

[0016] 110: Processor

[0017] 120: Storage element

[0018] 130: Image Capture

[0019] 310~340: Image

[0020] 410: Unsmoothed wrist point movement curve

[0021] 420: Smoothed wrist point movement curve

[0022] M:Mask

[0023] N1: Wrist point

[0024] N2: Shoulder point

[0025] S205~S225: Steps of image recognition method DETAILED DESCRIPTION

[0026] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this does not limit the present invention.

[0027] Figure 1 is a block diagram of an image recognition system according to an embodiment of the present invention. Figure 1 The image recognition system includes a processor 110, a storage device 120, and an image capturer 130. The processor 110 is coupled to the storage device 120 and the image capturer 130. In one embodiment, the processor 110 and the storage device 120 are disposed in the same electronic device, and the image capturer 130 can be coupled to the processor 110 via wired transmission or wireless transmission. In another embodiment, the processor 110, the storage device 120, and the image capturer 130 are disposed in the same electronic device.

[0028] The processor 110 is, for example, a central processing unit (CPU), a physical processing unit (PPU), a programmable microprocessor, an embedded control chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other similar devices.

[0029] The storage device 120 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, a hard disk, or other similar device, or a combination of these devices. The storage device 120 includes at least one program code segment, which, after being installed, is executed by the processor 110.

[0030] The image capturer 130 is, for example, a video camera or a still camera using a charge coupled device (CCD) lens or a complementary metal oxide semiconductor transistor (CMOS) lens. The image capturer 130 is positioned in front of the mask cleaning machine to capture images while the user is performing the cleaning operation.

[0031] Figure 2 FIG is a flow chart of an image recognition method according to an embodiment of the present invention. In this embodiment, each step is executed by the processor 110. Figure 2 In step S205, the wrist movement trajectory is identified in the image sequence captured by the image capturer 130. The image sequence includes multiple images captured while the user performs a cleaning action. The processor 110 can be configured to extract one image from the image sequence at every sampling frequency, thereby extracting multiple images within a time interval for image recognition. Alternatively, the processor 110 can be configured to extract each image included in the image sequence for image recognition.

[0032] In one embodiment, the storage device 120 includes a pose estimation model written in the Python programming language. The pose estimation model uses the PoseNet algorithm to perform real-time human pose estimation, thereby finding a skeleton composed of multiple points. In the PoseNet algorithm, the skeleton includes 17 points, and each point has a corresponding identification code (ID). The 17 points include nose (ID: 0), left eye (ID: 1), right eye (ID: 2), left ear (ID: 3), right ear (ID: 4), left shoulder (ID: 5), right shoulder (ID: 6), left elbow (ID: 7), right elbow (ID: 8), left wrist (ID: 9), right wrist (ID: 10), left hip (ID: 11), right hip (ID: 12), left knee (ID: 13), right knee (ID: 14), left ankle (ID: 15), and right ankle (ID: 16). In other embodiments, the pose estimation model may also use the MoveNet algorithm or the BlazePose algorithm.

[0033] After obtaining the skeletons in each frame of the image sequence, the wrist movement trajectory is obtained based on the wrist and shoulder points in these skeletons. For example, the coordinates of the two points with IDs 6 and 10 in each skeleton are taken to obtain the wrist movement trajectory. Alternatively, the coordinates of the two points with IDs 5 and 9 in each skeleton are taken to obtain the wrist movement trajectory. Alternatively, the coordinates of the four points with IDs 5, 6, 9, and 10 in each skeleton are taken to obtain the wrist movement trajectory.

[0034] After obtaining the wrist movement trajectory, in step S210, the number of wrist movements and the distance of each wrist movement are calculated based on the wrist movement trajectory. The number of movements is calculated by counting the number of times the wrist point intersects with the shoulder point. For example, moving the wrist point from one side of the shoulder point to the second side and then back to the first side is considered one movement.

[0035] Figures 3A to 3D is a schematic diagram of a human skeleton according to an embodiment of the present invention. Figures 3A to 3D The image sequence includes images 310-340, which were captured using the image capturer 130 while the user is performing a cleaning operation on the photomask M. In this embodiment, the pose estimation model utilizes the PoseNet algorithm to find a skeleton composed of 17 points, such as the 17 dots shown in images 310-340. In this embodiment, only the 17 points and the photomask M are depicted for illustration purposes.

[0036] For right-handed operation, the coordinates of wrist point N1 and shoulder point N2 on the right hand are used to obtain the wrist movement trajectory. As shown in images 310-340, the movement of wrist point N1 from one side of shoulder point N2 to the other side is considered a single movement. Processor 110 calculates the movement distance from wrist point N1 in image 310 to wrist point N1 in image 320. It then calculates the movement distance between wrist point N1 in images 320 and 330, and then calculates the movement distance between wrist point N1 in images 330 and 340. Finally, these three movement distances are summed to obtain the final movement distance of this wrist movement.

[0037] After obtaining the number of movements and the distance of each movement, in step S215, it is determined whether the number of movements meets the preset number and whether the distance of each movement is within the preset range. If either condition is not met, that is, the number of movements does not meet the preset number or at least one movement distance is outside the preset range, a re-execution notification is issued in step S220. If both conditions are met, that is, the number of movements meets the preset number and the distance of each movement is within the preset range, the cleaning operation is determined to be complete in step S225.

[0038] Furthermore, a predetermined number of movements can be pre-set within a time interval. For example, the time interval is set to 3 minutes, the preset number of movements is 5, and the preset range is 40 to 60 centimeters. The processor 110 first determines whether the number of movements within the time interval meets the preset number. For example, the processor 110 extracts an image sequence within 3 minutes from the image capturer 130 for processing. If the number of movements within the time interval does not meet the preset number, the processor 110 issues a re-execution notification. The processor 110 can use a voice signal or a text signal to implement the re-execution notification.

[0039] If the number of movements within the time interval meets the preset number, the processor 110 determines whether the multiple movement distances obtained within the time interval are all within the preset range. If at least one movement distance obtained within the time interval is not within the preset range, the processor 110 issues a re-execution notification. If all movement distances obtained within the time interval are within the preset range, the processor 110 determines that the cleaning operation is complete.

[0040] In addition, the storage element 120 may further include an object detection model. The object detection model is written in Python programming language, for example, and adopts the YOLO (You Only Look Once) algorithm. Before executing the posture evaluation model, an initial image is first captured by the image capturer 130 to identify the corresponding positions of the anti-static device and the electrostatic box in the initial image through the object detection model, and to determine whether the anti-static device overlaps with the electrostatic box accordingly. Only after determining that the anti-static device overlaps with the electrostatic box, the processor 110 will drive the posture evaluation model to identify the wrist movement trajectory in the image sequence. This step is used to ensure that electrostatic protection measures are properly implemented before performing a cleaning operation on the mask. The anti-static device is, for example, a destaticizing gun or an anti-static ring.

[0041] Furthermore, the yellow ambient light source may cause some deviations in the results of the posture estimation model, such as offsets in the detected points. Therefore, after obtaining the wrist movement trajectory, processor 110 may first smooth the wrist movement trajectory and then use the smoothed wrist movement trajectory to calculate the number of wrist movements and the distance of each wrist movement.

[0042] Figure 4 is a graph of a movement trajectory according to an embodiment of the present invention. Figure 4 In the graph shown, the horizontal axis represents time, and the vertical axis represents the X-coordinate position of the wrist point (X-coordinate). The solid line represents the unsmoothed wrist point movement curve 410, and the dashed line represents the smoothed wrist point movement curve 420. Each point on wrist point movement curve 410 represents the position of the wrist point obtained from multiple images included in the image sequence.

[0043] The following example illustrates the smoothing process. A lowpass Butterworth filter is created to obtain the transfer function coefficients [b, a]. Here, [b, a] = butter(n, Wc), where n is the filter order and Wc is the cutoff frequency. Next, the initial state zi is constructed for the steady-state step response using the lfilter filter. For example, the lfilter_zi filter is used to construct the initial state zi = lfilter_zi(b, a). Then, the lfilter filter is used to obtain z = lfilter(b, a, xn, zi = zi × xn[0]). Then, the lfilter filter is used again to obtain z2 = lfilter(b, a, z, zi = zi × z[0]). Alternatively, the filtfilt filter can be used to obtain y = filtfilt(b, a, xn). The above filters are used to smooth the wrist movement trajectory to avoid point drift caused by insufficient ambient brightness.

[0044] In summary, the present disclosure utilizes an image capture device to capture a sequence of images of a person performing cleaning actions and then performs image recognition algorithms on these images. This allows for verification of the correctness of the cleaning action and prompts the user to proceed. This reduces the risk of improper cleaning due to negligence, improves cleaning effectiveness, and eliminates the need for additional supervision.

[0045] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image recognition method, characterized in that: A processor performs a plurality of steps, including: Identifying a wrist movement trajectory in an image sequence captured by an image capturer, wherein the image sequence is captured when a user performs a cleaning action; Based on the wrist movement trajectory, calculating a number of wrist movements and a movement distance of each wrist movement, wherein the number of movements is obtained by calculating the number of times a wrist point intersects with a shoulder point; Determining whether the number of movements meets a preset number and whether the movement distance of each movement is within a preset range; If the number of moves does not meet the preset number, or the distance of at least one move is not within the preset range, issuing a re-execution notification; and If the number of movements matches the preset number and the movement distance of each movement is within the preset range, it is determined that the cleaning action is completed; The method further includes executing the following steps by the processor: Before identifying the wrist movement trajectory, driving an object detection model to perform object detection on an initial image obtained by the image capture device to determine whether an anti-static device overlaps with an anti-static box; and After determining that the anti-static device overlaps with the electrostatic box, a posture evaluation model is driven to identify the wrist movement trajectory in the image sequence.

2. The image recognition method according to claim 1, wherein: The step of driving the posture assessment model to identify the wrist movement trajectory in the image sequence includes: detecting, using the posture estimation model, a plurality of skeletons corresponding to the user in the plurality of images included in the image sequence; and The wrist movement trajectory is obtained based on the wrist points and shoulder points in the skeletons.

3. The image recognition method according to claim 1, wherein: The method further includes executing the following steps by the processor: A smoothing process is performed on the wrist movement trajectory to calculate the number of wrist movements and the movement distance of each wrist movement based on the smoothed wrist movement trajectory.

4. The image recognition method according to claim 1, wherein: The step of determining whether the number of movements meets the preset number and whether the movement distance of each movement is within the preset range includes: Determining, by the processor, whether the number of movements within a time interval meets the preset number; If the number of moves within the time interval does not meet the preset number, issuing the re-execution notification through the processor; If the number of movements within the time interval meets the preset number, the processor determines whether all movement distances obtained within the time interval are within the preset range; If at least one of all movement distances obtained within the time interval is not within the preset range, issuing the re-execution notification by the processor; and If all movement distances obtained within the time interval are within the preset range, the processor determines that the cleaning action is completed.

5. An image recognition system, characterized in that: include: an image capture device; a processor coupled to the image capture device, the processor being configured to: identifying a wrist movement trajectory in an image sequence captured by the image capturer, wherein the image sequence is captured when a user performs a cleaning action; Based on the wrist movement trajectory, calculating a number of wrist movements and a movement distance of each wrist movement, wherein the number of movements is obtained by calculating the number of times a wrist point intersects with a shoulder point; Determining whether the number of movements meets a preset number and whether the movement distance of each movement is within a preset range; If the number of moves does not meet the preset number, or the distance of at least one move is outside the preset range, a re-execution notification is issued; as well as If the number of movements matches the preset number and the movement distance of each movement is within the preset range, it is determined that the cleaning action is completed; The processor is configured to: Before recognizing the wrist movement trajectory, driving an object detection model to perform object detection on an initial image obtained by the image capture device to determine whether an anti-static device overlaps with an anti-static box; as well as After determining that the anti-static device overlaps with the electrostatic box, a posture evaluation model is driven to identify the wrist movement trajectory in the image sequence.

6. The image recognition system according to claim 5, wherein: The processor is configured to: detecting, using the posture estimation model, a plurality of skeletons corresponding to the user in the plurality of images included in the image sequence; and The wrist movement trajectory is obtained based on the wrist points and shoulder points in the skeletons.

7. The image recognition system according to claim 5, wherein: The processor is configured to: A smoothing process is performed on the wrist movement trajectory to calculate the number of wrist movements and the movement distance of each wrist movement based on the smoothed wrist movement trajectory.

8. The image recognition system according to claim 5, wherein: The processor is configured to: Determining whether the number of movements within a time interval meets the preset number; If the number of moves within the time interval does not meet the preset number, issuing the re-execution notification; If the number of movements within the time interval meets the preset number, determining whether all movement distances obtained within the time interval are within the preset range; If at least one of all movement distances obtained within the time interval is not within the preset range, issuing the re-execution notification; as well as If all movement distances obtained within the time interval are within the preset range, it is determined that the cleaning action is completed.

Images