Method, system and server for determining in-image sky track regions

Abstract

The application discloses a method, system and server for determining an aerial track area in an image, wherein the method comprises the following steps: S1, acquiring an RGB track image to be processed; S2, converting the RGB track image to obtain a single-channel H component image; S3, determining a set of hue variation points on a longitudinal line of the H component image and determining the aerial track area from the H component image based on the determined set of hue variation points. After the original image is acquired, the method does not perform conventional global median noise reduction or mean filtering processing, but directly converts the original image into an H component image, identifies hue variation points based on the tracking and locking principle of the first derivative mutation of the hue component, and determines the aerial track edge by combining the least square method. The processing method can instantaneously extract a standard aerial track area from the original image within 5 ms, thereby laying a foundation for the aerial track dust detection of a crown under the condition that the speed of the crown is not reduced.

Description

Technical Field

[0001] This invention relates to the field of semiconductor technology, and in particular to a method, system, and server for determining the aerial track region in an image of an overhead crane system used in semiconductor processing for detecting dust on the aerial track. Background Technology

[0002] Overhead crane (OHT) systems are key equipment in semiconductor factories used for handling materials such as wafer cassettes.

[0003] Semiconductor manufacturing plants have stringent requirements for the cleanliness level of their cleanroom environments. In AMHS systems, the overhead rails, which form the basis for the operation of overhead cranes (OHTs), are highly susceptible to accumulating micron-sized dust on their surfaces during long-term operation. This dust is not only a key hidden danger leading to wafer contamination but also exacerbates mechanical wear. Since the length of the overhead rails in FAB plants is typically over ten kilometers and requires continuous operation 24 / 7, how to efficiently clean the dust from these rails has always been a pain point affecting production efficiency.

[0004] Currently, the cleaning and maintenance of overhead rails mainly relies on dedicated "sweeping cranes." However, due to the lack of real-time dust distribution data on the surface of overhead rails throughout the plant, most existing cleaning solutions adopt a "blind sweeping" mode with fixed routes or fixed cycles, which is extremely time-consuming and lacks specificity, severely encroaching on the overhead rail resources of cranes used for normal production.

[0005] Visual inspection is desirable for detecting dust on aerial tracks; however, significant difficulties remain in practical applications. For example, the high speed of overhead cranes (typically >2m / s) requires image processing to have extremely high real-time performance. However, existing image processing algorithms typically use the Canny edge detection operator for image edge detection. This method takes too long to process a single frame of image and cannot achieve real-time detection without slowing down the speed. Summary of the Invention

[0006] The purpose of this invention is to solve the above-mentioned problems existing in the prior art and to provide a method, system and server for determining the aerial orbit region in an image.

[0007] The objective of this invention is achieved through the following technical solution: The method for determining the aerial orbit region in an image includes the following steps: S1, Obtain the RGB track image to be processed; S2, Based on the RGB track image conversion, a single-channel H component image is obtained; S3, determine a set of tonal variation points on a set of vertical lines in the H component image and determine the air track region from the H component image based on the determined set of tonal variation points.

[0008] Preferably, the RGB image is acquired by an image acquisition component, which includes a dark field light source located on the side of the aerial track, wherein the angle between the normal of the light-emitting end face of the dark field light source and the projection line of the normal onto the surface of the aerial track does not exceed 30°.

[0009] Preferably, the dark field light source is an LED linear array light source.

[0010] Preferably, the image acquisition component is mounted on the unmanned aerial vehicle. The image acquisition component also includes a camera with a lens that points vertically downward and focuses on the upper surface of the aerial track. The trigger pin of the camera is connected to the pulse output terminal of the encoder of the unmanned aerial vehicle's walking mechanism. The encoder controls the pulse output terminal to output pulses according to a set frequency to trigger the camera to acquire images.

[0011] Preferably, each pixel of the RGB track image is converted into a hue component value according to the following formula; ; Where H is the hue component value, R is the red component value of a pixel, G is the green component value of a pixel, and B is the blue component value of a pixel.

[0012] Preferably, S3 includes: S31, extract n vertical lines that are uniformly distributed along the horizontal direction of the H component image from the H component image, with each vertical line extending along the vertical direction of the H component image to the edge of the H component image; S32, along the vertical axis of the H component image, determine the tone change parameters between pixel pairs sequentially distributed on each vertical axis in a top-to-bottom or bottom-to-top order, and determine two tone change points on each vertical axis based on the tone change parameters between pixel pairs; each pixel pair is two adjacent pixels on the vertical axis. S33, Two straight lines are obtained by least-squares fitting based on all determined hue variation points; S34, the region located between the two straight lines in the H component image is determined as the aerial orbit region.

[0013] Preferably, in step S32, the tone transition parameter between each pixel pair is determined according to the following formula: ; Where t is the hue abrupt change parameter, h(x) p ,y iLet y be the ordinate of the Pth vertical line. i The hue component values ​​of the pixels; h(x) p ,y i-1 Let y be the ordinate of the Pth vertical line. i-1 The hue component values ​​of the pixels; Δ th This is the threshold for hue abrupt change.

[0014] Preferably, when the tone change parameter between a pair of pixels is determined to be 1, the pixel with the larger ordinate in the pair is designated as the tone change point. When the tone change parameter between a pixel pair is determined to be 0, the tone change parameter between the next pixel pair is determined. For a given vertical line, after identifying two hue variation points on the vertical line, the first-order differential scan of the vertical line is stopped, and the first-order differential scan of the pixels of the next vertical line is performed.

[0015] A system for determining aerial orbit regions in an image, including: Image acquisition unit, used to acquire RGB track images to be processed; A conversion unit is used to convert the RGB track image into a single-channel H component image; The region of interest determination unit is used to find hue variation points in the H component image by utilizing the first derivative abrupt changes in the hue of a set of vertical lines on the H component image, and to determine the air orbit region based on the hue variation points.

[0016] The server includes a memory and a processor, wherein the memory stores a program that can be executed by the processor, and when the program is executed, it implements the method for determining the aerial orbit region in the image as described above.

[0017] The advantages of the technical solution of this invention are mainly reflected in: The method of this invention, after acquiring the original image, does not perform conventional global median denoising or mean filtering. Instead, it directly converts it into an H-component image and identifies tonal variation points based on the tracking and locking principle of the first derivative mutation of the hue component. Combined with the least squares method, it determines the edge of the aerial track. This processing method can instantly extract a standard aerial track area from the original image within 5ms, laying the foundation for the overhead crane to perform aerial track dust detection without slowing down. Moreover, the highest positioning failure rate is strictly controlled within 5%, thus providing clean target data for subsequent dust detection.

[0018] This invention utilizes dark field illumination technology. In actual operation, the incident light path is changed by a side dark field light source system at a specific angle, so that the light emitted by the dark field light source sweeps across the surface of the air track at a very large incident angle. This effectively avoids the problem that diffuse light sources can easily cause strong overexposure on smooth metal surfaces, thus completely covering up micron-sized dust particles.

[0019] This invention uses high-frequency pulses from an encoder to trigger a camera to capture images. The images captured while the transport vehicle is moving at high speed are free of motion blur and spatial distortion, perfectly meeting the stringent industrial requirements for large-area, uninterrupted real-time monitoring of the AMHS overhead track in semiconductor cleanrooms. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the method for determining the aerial orbit region in an image according to the present invention; Figure 2 This is a schematic diagram showing the positional relationship between the image acquisition component and the aerial track of the present invention; Figure 3 This is a flowchart illustrating step S3 of the method for determining the aerial track region in an image according to the present invention. Figure 4 This is a flowchart illustrating the process of determining the tonal variation points of each longitudinal line in S3 of the method for determining the aerial track region in an image according to the present invention. Figure 5 This is a flowchart illustrating the S4 process of the present invention; Figure 6 This is a flowchart illustrating the S5 process of the present invention. Detailed Implementation

[0021] The objectives, advantages, and features of this invention will be illustrated and explained through the following non-limiting description of preferred embodiments. These embodiments are merely typical examples of applying the technical solutions of this invention, and all technical solutions formed by equivalent substitutions or equivalent transformations fall within the scope of protection claimed by this invention.

[0022] In the description of the solution, it should be noted that the terms "center," "upper," "lower," "left," "right," "front," "rear," "vertical," "horizontal," "inner," and "outer," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience and simplification of description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Example 1

[0023] The method for determining the aerial orbit region in the image disclosed in this invention is described below with reference to the accompanying drawings. Figure 1 As shown, the method for determining the aerial orbit region in the image includes the following steps: S1, Obtain the RGB track image to be processed; S2, Based on the RGB track image conversion, a single-channel H component image is obtained; S3, determine a set of tonal variation points on a set of vertical lines in the H component image and determine the air track region from the H component image based on the determined set of tonal variation points.

[0024] The RGB track image is acquired by an image acquisition component, and includes a track area and a background area. The size of a single frame of the RGB image is, for example, 2048*2048 pixels, but is not limited here. The image acquisition component is mounted on an unmanned aerial railcar (not shown in the figure), specifically on the traveling mechanism of the unmanned aerial railcar. The unmanned aerial railcar can be a conventional overhead crane for material handling, or any trolley structure including a traveling mechanism similar to an overhead crane, such as a trolley specifically for aerial rail maintenance (e.g., cleaning, inspection). Its specific structure can be set as needed, and is not limited here. Furthermore, when detecting particulate matter on the aerial rail, the moving speed of the unmanned aerial railcar is not less than 2 m / s. The specific speed can be set according to actual needs, such as 2.2 m / s, 2.5 m / s, 3 m / s, etc., and is not limited here. After acquiring each RGB track image, the image acquisition component can upload the acquired RGB track image, along with information such as the location of the unmanned aerial track vehicle and the acquisition time, to a server for image storage and analysis. The server then performs image analysis according to the steps described above.

[0025] As attached Figure 2 As shown, the image acquisition component includes a camera 100 and a dark-field light source 200. The camera can be any known feasible image acquisition device. In this embodiment, the camera is a CMOS area array camera with micron-level spatial optical resolution imaging capability. The spatial optical resolution of the camera can be configured as needed, for example, 1μm / pixel, 5μm / pixel, 10μm / pixel, etc., and is not limited here. The camera is located directly above the aerial track, with its lens pointing vertically downwards and focusing on the upper surface of the aerial track 300. Furthermore, the camera's trigger mode can be closely synchronized with its movement. The camera's focal length, object distance, and other shooting parameters can be set according to actual needs and are not limited here.

[0026] Furthermore, the camera's trigger pin is connected to the pulse output terminal of the encoder (incremental encoder) of the unmanned aerial vehicle's walking mechanism. The encoder controls the pulse output terminal to output pulses according to a set frequency to trigger the camera to acquire images. The set frequency is, for example, one pulse output for every 2mm movement of the unmanned aerial vehicle, thereby forcing the camera to perform high-frequency image acquisition at equidistant spatial intervals.

[0027] Existing conventional vision solutions typically employ LED diffused light sources, which easily create specular reflection blind spots on the highly reflective surface of the aerial track (smooth aluminum alloy). To overcome this defect, the dark field light source 200 is an LED linear array light source, which is disposed on the side of the aerial track. In a preferred embodiment, there are two dark field light sources symmetrically distributed on both sides of the aerial track. The angle α between the normal 220 of the emitting end face 210 of the dark field light source 200 and the projection line of the normal on the surface of the aerial track does not exceed 30°, more preferably not exceeding 20°, not less than 10°, and even more preferably between 14° and 16°. Using a dark field light source can effectively reduce specular reflection light, allowing the camera to receive more scattered light, which is beneficial for the camera to receive the scattered light generated by the dust particles marked on the aerial track to form bright spots. This effectively avoids the problem that diffused light sources can easily cause strong overexposure on smooth metal surfaces, thus completely masking micron-sized dust particles.

[0028] Meanwhile, to prevent vibrations during the movement of the unmanned aerial vehicle from affecting image acquisition, the image acquisition component is mounted on a mounting base (not shown in the figure), which is connected to the traveling mechanism via a shock absorber (not shown in the figure). Alternatively, the mounting base can be directly fixed to the traveling mechanism, and it can have its own vibration damping component for reducing vibration of the image acquisition component. Its specific structure can be designed as needed. For example, the mounting base includes a base frame fixed to the traveling mechanism, a spring shock absorber mounted on the base frame, and a mounting bracket connected to the spring shock absorber, on which the image acquisition component is mounted.

[0029] After the camera captures a high signal-to-noise ratio dark-field image, it is necessary to instantly extract the complex non-aerial track background from a single frame. Because the material of the aerial track in a semiconductor cleanroom exhibits extremely stable tonal characteristics under controlled lighting (a flat H-value curve with very low variability), while the tones of non-target background areas such as the hangers and quartz windows on both sides of the aerial track differ significantly from the track surface, the non-aerial track background can be quickly extracted using steps S2 and S3.

[0030] Correspondingly, in S2, after receiving the RGB image, the R, G, and B component values ​​of each pixel in the RGB image are mapped to the HSL color space of the cylindrical coordinate system in real time to obtain the HSL image, and then the hue component is extracted to obtain the H component image; wherein the hue component exhibits extremely strong mathematical robustness to fluctuations in ambient light.

[0031] Each pixel of the RGB track image is converted into a hue component value according to the following formula; ; Where H is the hue component value, R is the red component value of a pixel, G is the green component value of a pixel, and B is the blue component value of a pixel.

[0032] Of course, in other embodiments, instead of converting to the HSL color space, the hue component value of each pixel in the RGB image can be determined directly based on the RGB image and the above formula to obtain the H component image. Furthermore, a temporary cache area can be allocated directly in memory to specifically store the H component image, which does not fluctuate with changes in ambient lighting intensity.

[0033] Under controlled dark-field side lighting, the H values ​​distributed on the clean aluminum alloy surface of the aerial track exhibit extremely high consistency, forming a gentle curve with a coefficient of variation approaching zero. However, when light, due to spatial displacement, crosses the edge of the aerial track and cuts into a complex non-target background area, the color components undergo drastic changes. Therefore, for each column of pixels in the image, if the first-order rate of change of the hue component values ​​of two adjacent pixels in a column exceeds a set threshold, that point can be determined as the edge of the aerial track.

[0034] S3 is based on the above-mentioned rules to quickly determine the aerial orbit region in the H component image, as shown in the attached figure. Figure 3 As shown, step S3 includes the following steps: S31, extract n vertical lines uniformly distributed along the horizontal direction of the H component image from the H component image. Each vertical line extends along the vertical direction of the H component image to the edge of the H component image. Here, horizontal direction refers to the left-right direction of the image, and the aerial track extends horizontally on the image; vertical direction refers to the up-down direction of the image. The number of n can be determined as needed. For example, in a preferred embodiment, n is 16, meaning 16 vertical lines are extracted. The reason for choosing 16 vertical lines is that this minimizes computation and improves efficiency, reduces the influence of noise points, and ensures accurate fitting to guarantee the acquisition of reliable straight line equations for identifying the aerial track boundary. Correspondingly, taking the H component image as 2048*2048 pixels as an example, an image coordinate system is constructed with the upper left corner of the image as the origin, the horizontal axis as the X-axis (horizontal coordinate), and the vertical axis as the Y-axis (vertical coordinate). All pixels with a horizontal coordinate of 120 are determined as the first vertical line. With a step size of 120 pixels, 16 vertical lines are determined in sequence. The horizontal coordinates of the pixels on the 16 vertical lines are 120, 240, 360, 480...1920 in sequence.

[0035] S32, along the vertical axis of the H component image, determine the tone change parameters between pixel pairs sequentially distributed on each vertical line in a top-to-bottom or bottom-to-top order, and determine two tone change points on each vertical line based on the tone change parameters between pixel pairs; each pixel pair is two adjacent pixels on the vertical line; for example, taking the H component image as 2048*2048 pixels, for a vertical line with an abscissa of 120, among the 2048 pixels of the first vertical line, the pixel with ordinate 1 and the pixel with ordinate 2 are the first pixel pair, the pixel with ordinate 2 and the pixel with ordinate 3 are the second pixel pair, the pixel with ordinate 3 and the pixel with ordinate 4 are the third pixel pair, and so on, a vertical line can obtain a maximum of 2047 pixel pairs.

[0036] In step S32, the tone transition parameter between each pixel pair is determined according to the following formula: ; Where t is the hue abrupt change parameter, h(x) p ,y i Let y be the ordinate of the Pth vertical line. i The hue component values ​​of the pixels; h(x) p ,y i-1 Let y be the ordinate of the Pth vertical line. i-1 The hue component values ​​of the pixels; Δ th This is the threshold for hue abrupt change. The Δ... th You can set it as needed, for example, to 30, 35, etc., there is no limit here.

[0037] For example, in the first vertical line mentioned above (horizontal coordinate is 120), the pixel with vertical coordinate 501 (hue component value is 100) and the pixel with vertical coordinate 500 (hue component value is 150) form a pixel pair. Calculate the hue transition parameter t=|h(x1,y) between this pixel pair. 501 )- h(x1,y 500 )|=|150-100|=50>Δ th If the hue transition parameter between this pixel pair is 1, then the hue transition parameter between these pixel pairs is equal to 1.

[0038] As attached Figure 4 As shown, when the tone change parameter between a pair of pixels is determined to be 1, the pixel with the larger ordinate in the pair is determined as the tone change point. For example, in the pixel pair in the example above, the pixel with the ordinate of 501 is the tone change point.

[0039] When the hue abrupt change parameter between a pair of pixels is determined to be 0, the hue abrupt change parameter between the next pair of pixels is determined. For example, if the hue abrupt change parameter between the first pair of pixels in the example above is 0, then the hue abrupt change parameter between the second pair of pixels is determined, and so on, until two hue abrupt change points in a vertical line are determined.

[0040] For a given vertical line, after identifying one hue variation point, it is determined whether two hue variation points have been identified for that vertical line. If not, i.e., the two hue variation points for the vertical line have not yet been identified, the calculation of the color bar transition parameters and confirmation of the hue variation points for the next pixel pair continues until the two hue variation points for the vertical line are identified. Then, the first-order differential scanning of the vertical line is stopped and the pixel coordinates of the two hue variation points are locked. The first-order differential scanning of the pixels of the next vertical line is then performed. The first-order differential scanning of the pixels of all vertical lines is performed in sequence until the two hue variation points of all vertical lines are obtained.

[0041] Two tonal variation points distributed vertically are extracted from each vertical line to obtain two sets of tonal variation points, one above and one below. For example, in the 16 vertical lines in the example above, the pixel coordinates of 32 tonal variation points are finally extracted.

[0042] S33, based on all the determined tonal variation points, perform least squares fitting to obtain two straight lines; that is, perform least squares fitting based on the pixel coordinates of the upper set of tonal variation points (16 points) among all the obtained tonal variation points to obtain one straight line, and perform least squares fitting based on the pixel coordinates of the lower set of tonal variation points (16 points) among all the obtained tonal variation points to obtain another straight line.

[0043] These two straight lines can be identified as the two sides of the aerial track in the image. The specific method of obtaining a straight line by least squares fitting of a set of discrete points is a known technique and is not an innovation of this invention, so it will not be elaborated here.

[0044] S34, the region located between two straight lines in the H component image is defined as the aerial track region. The aerial track region can be cropped from the H component image to obtain an aerial track region image, for example, with a size of 2048×800 pixels. Subsequently, corresponding recognition can be performed based on the cropped aerial track region image, for example, for the identification of dust, cracks, oil stains, etc., on the aerial track. Example 2

[0045] This embodiment discloses a system for determining aerial orbit regions in an image, including: Image acquisition unit, used to acquire RGB track images to be processed; A conversion unit is used to convert the RGB orbital image into a single-channel H component image; The region of interest determination unit is used to find hue variation points in the H component image by utilizing the first derivative abrupt changes in the hue of a set of vertical lines on the H component image, and to determine the air orbit region based on the hue variation points. Example 3

[0046] This embodiment discloses a server, including a memory and a processor. The memory stores a program that can be executed by the processor. When the program is executed, it implements the method for determining the aerial orbit region in an image as described above. Example 4

[0047] This embodiment discloses a method for detecting track dust, including a method for determining the aerial track region in the image above. After cropping the aerial track region image, dust detection can be performed based on the aerial track region image, as detailed below: S4, based on the difference box dimension algorithm, the cropped aerial orbit region image is processed into a binary image; S5, the binary image is processed by run-length encoding, and morphological closing operation is performed in the compressed domain based on the result of run-length encoding, and candidate dust images are cropped from the aerial orbit region image based on the result of morphological closing operation. S6, input the candidate dust image into the target detection model to obtain the identification result of dust on the surface of the air track.

[0048] Correspondingly, as shown in the appendix Figure 5 As shown, step S4 includes the following steps: S41, divide the aerial orbit region image with size L×N pixels (such as 2048×800 pixels above) into multiple continuous macroblocks with size M×M pixels; M is, for example, 32, 64, 128, 256, etc., which is not limited here.

[0049] S42, determine the local fractal dimension corresponding to each macroblock, specifically according to the following process: S421, a macroblock is divided into a set of s×s pixel continuous grids according to multiple scales s, and the scale factor r=s / M corresponding to each scale; for example, s takes four scales, namely 2, 4, 8, and 16.

[0050] Each grid (i,j), where i is the number of rows and j is the number of columns, is divided into three-dimensional grayscale boxes of height h in the vertical direction, satisfying the algebraic relationship G / h=(M / s), where G is the total grayscale level of the current grid, which is usually 256.

[0051] S422, For each grid divided at each scale, calculate the total number of gray boxes corresponding to the grid. The total number of gray boxes corresponding to one grid is the number of boxes required to cover the gray-level extreme difference of the grid. The height h of each gray box is G / (M / s).

[0052] Specifically, for a grid, iterate through all the pixels it contains and find the maximum and minimum gray values ​​among all pixels in the grid; determine the gray box layers where the maximum and minimum gray values ​​fall. For example, assuming G=256, M=256, s=16, then h=256 / (256 / 16)=16, the maximum gray value is 130, and the minimum gray value is 110. Then the gray box layer where the maximum gray value falls is l=ceil(130 / 16)=9, and the gray box layer where the minimum gray value falls is k=ceil(110 / 16)=7.

[0053] Then, the total number of grayscale boxes corresponding to a grid can be determined using the following formula: n r (i, j) = l - k + 1; Where, n r (i, j) represents the total number of gray boxes corresponding to the grid in the i-th row and j-th column; l represents the number of gray box layers in which the maximum gray value of all pixels in the i-th row and j-th column falls; and k represents the number of gray box layers in which the minimum gray value of all pixels in the i-th row and j-th column falls.

[0054] S423, sum the total number of gray boxes corresponding to all grids divided at each scale to obtain the total number of image boxes Nr corresponding to each scale factor r, and determine a data pair based on each scale factor and its corresponding total number of image boxes.

[0055] For example, when M=256 and s=2 pixels, r=1 / 128, the number of grids at a scale of 2 pixels is 16384. After determining the total number of gray boxes corresponding to each grid, the sum of the total number of gray boxes in these 16384 grids gives the total number of image boxes N corresponding to the scale factor of 1 / 128. 1 / 128， At this point, we obtain a data pair (x) 1 / 128 ,y 1 / 128 = (log(1 / 128), log(N)) 1 / 128 )).

[0056] Following the example above, when s = 4 pixels, 8 pixels, and 16 pixels, the corresponding scale factors r are 1 / 64, 1 / 32, and 1 / 16, respectively. The total number of image boxes corresponding to these scale factors is determined to be N. 1 / 64 N 1 / 32 N 1 / 16 The final three data pairs are as follows: (x) 1 / 64 ,y 1 / 64 = (log(1 / 64), log(N)) 1 / 64 )); (x) 1 / 32 ,y 1 / 32 = (log(1 / 32), log(N)) 1 / 32 )); (x) 1 / 16 ,y 1 / 16 = (log(1 / 16), log(N)) 1 / 16 )).

[0057] S424, based on the multiple sets of data obtained in S423, perform least squares fitting, and use the slope of the fitted regression equation as the local fractal dimension D of the macroblock.

[0058] Because clean, regular metal surface textures have extremely high self-similarity and roughness uniformity, their corresponding local fractal dimension remains at a very low stable reference threshold (close to the theoretical Euclidean plane dimension of 2.0); while scattered dust particles or dust clusters, due to their chaotic micro-geometric spatial structure and obvious spatial discontinuities, will have a sharp increase in the fractal dimension of their corresponding grids, thus forming significant distinguishable features.

[0059] Therefore, in S43, the binary image can be obtained by assigning a value to each pixel of each macroblock based on the local fractal dimension D and the fractal dimension threshold of each macroblock.

[0060] Specifically, the local fractal dimension of each macroblock is compared with the fractal dimension threshold, which can be set as needed, for example, the fractal dimension threshold is 2.40; If it is determined that the local fractal dimension D of a macroblock is greater than or equal to the fractal dimension threshold, then each pixel of the macroblock is assigned a logic high level (1), which represents dust. If it is determined that the local fractal dimension of a macroblock is less than the fractal dimension threshold, then each pixel of the macroblock is assigned a logic low level (0), which represents a smooth grain region belonging to a clean aluminum alloy texture. Finally, the image of the aerial orbit region is converted into a binary image consisting of 0s and 1s.

[0061] After binarization, the image area corresponding to dust often appears as discrete, isolated clusters of pixels, and the contour edges are very prone to carrying burrs, holes and local micro-discontinuities caused by discrete scattering.

[0062] Traditional interactive image segmentation algorithms based on GraphCut or standard GrabCut, while offering some noise suppression, typically require constructing complex Gibbs energy functions and introducing a mixed negative entropy Gibbs energy term as a penalty function to eliminate redundant Gaussian components. In practice, these iterative segmentation algorithms require multiple Gibbs optimization iterations (2-7 times) of the energy term per frame to converge, resulting in processing times of up to hundreds of milliseconds per frame. This can lead to catastrophic frame loss when the crane travels at a speed of 2 m / s, failing to meet the requirements for real-time, low-latency online monitoring.

[0063] This invention abandons the iterative logic of pixel-by-pixel optimization and adopts morphological binary closed-loop operation logic optimized by run-length encoding (RLE). The discrete dust regions selected by S3 are converted into a sparse array composed of "row number," "starting pixel column number," and "continuous pixel length." Dilation and erosion operations are performed within the compressed data domain using the commutative and distributive laws of Minkowski addition. This design avoids pixel-by-pixel matrix convolution traversal, eliminating edge burrs and filling in fine breaks in the binarized image in a very short time, merging discrete reflective points into a connected, closed candidate region for the measured contour.

[0064] Correspondingly, as shown in the appendix Figure 6 As shown, step S5 includes the following steps: S51, The binary graph is compressed using run-length encoding to generate an RLE run-length data stream.

[0065] Specifically, the dust pixel segments that are consecutive in each row of the binary image and have the same logical value (1) are compiled into an RLE process, and the row number, starting pixel column number and logical pixel length are recorded.

[0066] Assume that the binary image is a 10×10 pixel binary image, where 1 represents a bright dust pixel and 0 represents a normal orbit pixel, and its image geometry set A is as follows: Line 0:0000000000 Line 1: 0000000000 Line 2: 0000000000 Line 3: 0011110000 Line 4: 0010011000 Line 5: 0011011000 Line 6:0001110000 Line 7: 0000000000 Line 8: 0000000000 Line 9: 0000000000 The RLE process flow after RLE compression consists of 7 RLE processes, as follows: [(3,2,4),(4,2,1),(4,5,2),(5,2,2),(5,5,2),(6,3,3)].

[0067] S52, perform morphological dilation operation on the RLE flow obtained in S51 within the compression domain.

[0068] Specifically, according to the algebraic definition of Minkowski addition, the spatial dilation of the image geometric set A and the structuring element B is transformed into the union of the following interval translations: ; Furthermore, by utilizing the inherent commutative law of Minkowski addition and the following distributive law, high-speed translation, intersection, and merging of RLE flow paths can be performed directly in the algebraic domain: ; in, `\r` is the morphological dilation operator, `\r` is the union operation, which combines all translated results together; `\r` represents the relative coordinates or position vector of each pixel in structuring element B relative to the structuring element's own reference point (usually the center point); A b This means that each RLE run is compressed from the image geometry set A and translated by an offset b (i.e., all pixels in A are moved by an offset b), that is, each RLE run is translated as a whole by the offset b of the structuring element B, and then the union is calculated (overlapping intervals are merged); A1 and A2 represent two RLE runs in the RLE run stream after the image geometry set A is compressed by RLE.

[0069] For example: Suppose that structural element B is a cross-shaped structural element: 0 1 0 1 1 1 0 1 0 When the center point is used as the reference point for structural element B, with the center point coordinates (1,1), the relative coordinates of the points in structural element B with a value of 1 are determined as follows: B 中 ={(-1,0),(0,-1),(0,0),(0,1),(1,0)},b∈B 中 ; For example, for the RLE trip (3,2,4) in the example above, adding the five offsets mentioned above respectively yields the following new RLE trip: +(-1,0) results in the new RLE route (2,2,4); +(0,-1), resulting in the new RLE route (3,1,4); +(0,0) results in the new RLE route (3,2,4); +(0,1), resulting in the new RLE route (3,3,4); +(1,0) gives the new RLE route (4,2,4).

[0070] Repeat the same operation on the other RLE processes to obtain multiple new RLE processes.

[0071] All the new RLE trips obtained are merged to get the expanded RLE trip flow D. 膨 This completes the morphological dilation operation of the RLE flow obtained from S41 processing in the compressed domain: For example: [(2,2,4),(3,1,6),(4,1,7),(5,1,7),(6,2,6),(7,3,5)].

[0072] S53, RLE stroke flow D obtained by morphological dilation 膨 Perform morphological erosion operations within the compressed domain.

[0073] The mathematical definition of corrosion is as follows: A⊖B={x|B+x⊆A}; That is, the set of all points x that satisfy the condition that "after B is translated to position x, it is completely contained within A".

[0074] Convert erosion into intersection operation: A⊖B=∩_{b∈B}(Ab) This formula means that for each b∈B, A is translated as a whole by -b, and then the intersection of all translation results is found.

[0075] Specifically, we take the inverse vector -b of the offset vector of the structuring element B, and translate each RLE stroke after dilation according to the inverse vector -b. We then find the intersection of all translation results to perform morphological erosion operation in the compressed domain and complete the morphological closing operation in the compressed domain.

[0076] For example, taking the structuring element B from the example above, the inverse vector -b ∈ -B 中 ={(1,0),(0,1),(0,0),(0,-1),(-1,0)} For the expanded RLE stroke flow D obtained from S42 膨 Each RLE stroke is translated according to each inverse vector -b, as follows: 1. D 膨 -(1,0) (shifts down 1 row), the result is as follows: (2,2,4)→(3,2,4); (3,1,6)→(4,1,6); (4,1,7)→(5,1,7); (5,1,7)→(6,1,7); (6,2,6)→(7,2,6); (7,3,3)→(8,3,3); 2. D 膨 -(0,1) (shift right by 1 column), the result is as follows: (2,2,4)→(2,3,4); (3,1,6)→(3,2,6); (4,1,7)→(4,2,7); (5,1,7)→(5,2,7); (6,2,6)→(6,3,6); (7,3,3)→(7,4,3); 3. D 膨 -(0,0), the result is as follows: (2,2,4),(3,1,6),(4,1,7),(5,1,7),(6,2,6),(7,3,3); 4. D 膨 -(0,-1) (shift left by 1 column), the result is as follows: (2,2,4)→(2,1,4); (3,1,6)→(3,0,6); (4,1,7)→(4,0,7); (5,1,7)→(5,0,7); (6,2,6)→(6,1,6); (7,3,3)→(7,2,3); 5. D 膨 -(-1,0) (shifts up 1 row), the result is as follows: (2,2,4)→(1,2,4); (3,1,6)→(2,1,6); (4,1,7)→(3,1,7); (5,1,7)→(4,1,7); (6,2,6)→(5,2,6); (7,3,3)→(6,3,3); Find the intersection of these 5 translation results (find the intersection row by row and column by column). When finding the intersection, for each row number, take the portion that is covered by the interval of all 5 results in that row.

[0077] For example, in line 4: D 膨 -(1,0)Row 4: (4,1,6)→Columns 1~6; D 膨 -(0,1) Row 4: (4,2,7) → Columns 2 to 7; D-(0,0) Row 4: (4,1,7) → Columns 1-7; D 膨 -(0,-1) Row 4: (4,0,7) → Columns 0 to 7; D 膨 -(-1,0) Row 4: (4,1,7) → Columns 1-7 The intersection of the fourth row = column 1~6 ∩ column 2~7 ∩ column 1~7 ∩ column 0~7 ∩ column 1~7 = column 2~6, ultimately yielding the new RLE route (4,2,5). Process all rows in the same way to obtain the final erosion result: The RLE runflow obtained from the morphological erosion operation includes 5 RLE runs, as follows: [(3,2,4),(4,2,5),(5,2,5),(6,3,3)].

[0078] S54, candidate dust images are cropped from the aerial orbit region image based on the RLE data stream obtained by morphological erosion operation. Specifically, known methods can be used to perform connected component labeling, candidate dust boundary tracking, and determination of the vertex pixel coordinates of the bounding rectangle of the boundary on the RLE data stream obtained by morphological erosion operation. For example, the final RLE run stream can be scanned line by line, and the column intervals of the current RLE run and the previous RLE run can be compared to see if they overlap. If they do, they are determined to be a connected component with the same connected component label; otherwise, a new connected component label is assigned to the current RLE run. Based on the connected component labeling results, a four-neighbor check (top, bottom, left, right) is performed on each run to determine whether a pixel is a boundary, and all RLE run boundaries are extracted one by one. The specific boundary judgment conditions and processes of the four-neighbor check are well known in the art and are not innovative in this invention, so they will not be elaborated here.

[0079] This embodiment uses the RLE route (3,2,4) as an example to illustrate how to perform a four-neighbor check, which covers the column interval [2,5]. Based on the boundary judgment conditions, the following boundary pixels are obtained: The boundary points of the RLE route (3,2,4) are summarized below: (3,2),(3,3),(3,4),(3,5),(3,2),(3,5); After deduplication, the final boundary points are as follows: (3,2),(3,3),(3,4),(3,5).

[0080] After merging and deduplicating the boundary points of all RLE trips with the same connected component label, the set of contour points of the corresponding connected component is obtained, thus completing the boundary tracking of candidate dust.

[0081] Subsequently, the row and column extreme values ​​of all contour points of a connected region can be counted, and the pixel coordinates of the vertices of the bounding rectangle of the connected region can be determined based on the row and column extreme values.

[0082] Finally, candidate dust images can be cropped from the aerial orbit region image based on the vertex pixel coordinates of each determined circumscribed rectangle.

[0083] In step S6, the complete candidate dust image obtained from step S5 is cropped from the aerial orbit region image in real time and seamlessly fed into the target detection model for processing. The target detection model can be a known YOLO series model, MobileNet-SSD model, NanoDet model, etc.

[0084] To further adapt to efficient inference in embedded devices within semiconductor cleanrooms, the target detection model employs a lightweight target detection network optimized through network pruning. Specifically, it can utilize the well-known YOLOv8-tiny or YOLOv11 network architectures released by Ultralytics. This lightweight target detection network is dedicated to extracting deep, multi-dimensional semantic features from candidate dust images using deeper multi-scale convolutional kernels and efficient RoI pooling layers. The target detection model is trained using a supervised learning strategy, and its training dataset originates from a sample library constructed after preprocessing historical aerial orbital region image data. This sample library covers the following three types of image patches: positive samples with accurately labeled real dust locations, difficult-to-distinguish negative samples containing interference such as track mechanical scratches and metal oxidation points, and clean track background images without any target interference (used for data augmentation and suppressing model overfitting to improve the false alarm prevention capability during the inference stage). The specific training process of the target detection model (including loss function design, optimizer selection, and hyperparameter tuning strategies) are all well-known technologies in the field and are not innovative in this invention, and will not be elaborated upon here.

[0085] In the high-level semantic feature map of the lightweight object detection network, traditional algorithms mainly rely on aspect ratio to distinguish between dust and scratches. This method is highly susceptible to changes in the object's shape and lacks reliability. To address this issue, our lightweight object detection network abandons the single size judgment in the high-level semantic feature extraction stage and instead uses topological descriptors as a new discrimination criterion. Topological descriptors are a set of mathematical measures that describe the global structural attributes of objects in candidate dust images, such as the number of connected components, the number of holes, the Euler number, the number of skeleton branch points, and the distribution of endpoints, which can accurately characterize the essential shape of the object.

[0086] For example, the inherent mechanical scratches on the surface of the overhead crane track (aluminum alloy) caused by years of friction exhibit highly smooth geometric edges and long-distance longitudinal linear continuity in high-level semantic features. Their topological descriptors are highly compact, characterized by: connected components C=1, no holes H=0, very few skeleton branch points, and acyclic single-chain extension. In contrast, real micron-level outdoor dust (dust particles or particle clusters, etc.) in a cleanroom exhibits random, irregular, clumped geometric topology in the multidimensional feature space, accompanied by extremely high-frequency microscopic gray-scale fluctuations. Their topological descriptors are highly fragmented or complex, characterized by: connected components C>1, significantly increased skeleton branch points, and crisscrossing branches. Through this essential morphological distinction, the network can easily eliminate interference from the inherent mechanical scratches on the track surface.

[0087] The classification layer of the lightweight target detection network uses deep texture features to distinguish between true and false targets. During the forward propagation inference process, these high-level multidimensional feature vectors are extracted in real time. The network automatically identifies and removes normal mechanical scratches that are geometrically linear and continuous in the longitudinal direction, as well as local endogenous metal oxidation points. This allows the network to distinguish between true and false targets and accurately remove endogenous false positive interference items such as track mechanical scratches and metal oxidation points with similar shapes.

[0088] The output layer of the lightweight target detection network outputs detection results including information on whether the target object in the candidate dust image is dust (type label, confidence level, etc.). When the detection result indicates that the target object in the candidate dust image is dust, the server uploads the detection result and the location information of the unmanned aerial track vehicle at the time of acquisition of the RGB track image corresponding to the detection result to the scheduling system of the overhead crane system. When the scheduling system receives the detection result indicating the presence of dust, the scheduling system can instruct the track cleaning overhead crane to clean the track segment within a set distance (which can be customized as needed) before and after the location of the unmanned aerial track vehicle at the time of acquisition of the RGB track image, or instruct the track cleaning overhead crane to perform a comprehensive cleaning of all aerial tracks, or the scheduling system can issue an alarm to remind staff to handle the situation. The specific structure of the track cleaning overhead crane is known technology, for example, it can be the structure disclosed in patent documents such as application publication number CN121755485A, which will not be elaborated here.

[0089] This invention has many other embodiments, and all technical solutions formed by equivalent transformation or equivalent transformation fall within the protection scope of this invention.

Claims

1. A method for determining the aerial orbit region in an image, characterized in that, Includes the following steps: S1, Obtain the RGB track image to be processed; S2, Based on the RGB track image conversion, a single-channel H component image is obtained; S3, determine a set of tonal variation points on a set of vertical lines in the H component image and determine the air track region from the H component image based on the determined set of tonal variation points.

2. The method for determining the aerial orbit region in an image according to claim 1, characterized in that: The RGB image is acquired by an image acquisition component, which includes a dark field light source located on the side of the aerial track. The angle between the normal of the light-emitting end face of the dark field light source and the projection line of the normal onto the surface of the aerial track does not exceed 30°.

3. The method for determining the aerial orbit region in an image according to claim 2, characterized in that: The dark field light source is an LED linear array light source.

4. The method for determining the aerial orbit region in an image according to claim 1, characterized in that: The image acquisition component is mounted on the unmanned aerial vehicle. The image acquisition component also includes a camera with a lens that is vertically downward and focused on the upper surface of the aerial track. The trigger pin of the camera is connected to the pulse output terminal of the encoder of the unmanned aerial vehicle's walking mechanism. The encoder controls the pulse output terminal to output pulses according to a set frequency to trigger the camera to acquire images.

5. The method for determining the aerial orbit region in an image according to claim 1, characterized in that: Each pixel of the RGB track image is converted into a hue component value according to the following formula; Where H is the hue component value, R is the red component value of a pixel, G is the green component value of a pixel, and B is the blue component value of a pixel.

6. The method for determining the aerial orbit region in an image according to any one of claims 1-5, characterized in that: S3 includes: S31, extract n vertical lines that are uniformly distributed along the horizontal direction of the H component image from the H component image, with each vertical line extending along the vertical direction of the H component image to the edge of the H component image; S32, along the vertical axis of the H component image, determine the tone change parameters between pixel pairs sequentially distributed on each vertical axis in a top-to-bottom or bottom-to-top order, and determine two tone change points on each vertical axis based on the tone change parameters between pixel pairs; each pixel pair is two adjacent pixels on the vertical axis. S33, Two straight lines are obtained by least-squares fitting based on all determined hue variation points; S34, the region located between the two straight lines in the H component image is determined as the aerial orbit region.

7. The method for determining the aerial orbit region in an image according to claim 6, characterized in that: In step S32, the tone transition parameter between each pixel pair is determined according to the following formula: ； Where t is the hue abrupt change parameter, h(x) p ,y i Let y be the ordinate of the Pth vertical line. i The hue component values ​​of the pixels; h(x) p ,y i-1 Let y be the ordinate of the Pth vertical line. i-1 The hue component values ​​of the pixels; Δ th This is the threshold for hue abrupt change.

8. The method for determining the aerial orbit region in an image according to claim 6, characterized in that: When the tone change parameter between a pair of pixels is determined to be 1, the pixel with the larger ordinate in the pair is designated as the tone change point. When the tone change parameter between a pixel pair is determined to be 0, the tone change parameter between the next pixel pair is determined. For a given vertical line, after identifying two hue variation points on the vertical line, the first-order differential scan of the vertical line is stopped, and the first-order differential scan of the pixels of the next vertical line is performed.

9. A system for determining aerial orbit regions in an image, characterized in that, include: Image acquisition unit, used to acquire RGB track images to be processed; A conversion unit is used to convert the RGB orbital image into a single-channel H component image; The region of interest determination unit is used to find hue variation points in the H component image by utilizing the first derivative abrupt changes in the hue of a set of vertical lines on the H component image, and to determine the air orbit region based on the hue variation points.

10. A server, comprising a memory and a processor, wherein the memory stores a program executable by the processor, characterized in that: When the program is executed, it implements the method for determining the aerial orbit region in the image as described in any one of claims 1-8.

Images