A real-time detection method for specular surface defects using structured light fields based on graphics processors

Abstract

A real-time detection method for structured light field specular surface defects based on a graphics processor includes the following steps: A1: The CPU launches at least two threads in parallel, captures video frame data from a light field camera shooting the object under inspection, and writes the video frame data from memory to the GPU global cache; A2: The GPU and CPU asynchronously and in parallel launch microlens feature calculation, segments the original light field image according to pre-input calibration information, and uses GPU parallel calculation to obtain the statistical eigenvalues ​​of the residual image under the microlens; A3: Utilizing the spatial and temporal correlations of the light field video, initiates light field correlation fusion processing, performs weighted fusion of the eigenvalues ​​under each microlens in the frame with the eigenvalues ​​of the nearest neighbor microlens, and adds them to the final thermal map using bilinear interpolation; A4: Updates the structured light field video defect detection results corresponding to the object under inspection. This invention addresses the high time complexity challenges brought about by high data dimensionality and large data volume in the field of light field defect detection.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital video processing, and in particular to a real-time detection method for high-gloss surface defects in a structured light field based on a graphics processor. Background Art

[0002] Machine vision defect detection uses images captured by optical sensors as input, extracts surface features using computer vision methods, and then identifies physical defects. Compared to traditional manual recognition, this technology offers high accuracy and speed, and holds broad application prospects in industrial quality inspection. However, traditional image recognition and stereo vision technologies rely heavily on image feature points and texture information, often failing to detect textureless metal objects. While structured light-based inspection technologies can reconstruct three-dimensional surface information, these systems require multiple exposures at the expense of time complexity or require multiple structured light systems for multiple spatial exposures, failing to meet the real-time requirements of industrial inspection environments. Light field imaging, through a unique optical path design, simultaneously captures four-dimensional light information, encompassing both spatial and angular dimensions, enabling single-exposure, multi-angle stereoscopic imaging. However, light field data is high-dimensional and complex to process, and research on its application in industrial inspection is still in its infancy.

[0003] Existing research on 3D reconstruction of specular surfaces based on structured light fields is limited by the high dimensionality of light field data, resulting in reconstruction times measured in hours for typical scenes. Traditional light field image processing relies on sub-aperture views, which can take minutes to render. Existing structured light field defect detection technologies are unable to overcome the high dimensionality, large data volume, and complex processing of light field data, making real-time online processing difficult.

[0004] Therefore, it is of great significance to develop a real-time method for detecting surface defects of high-gloss objects using structured light fields.

[0005] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The main purpose of the present invention is to provide a real-time detection method for high-gloss surface defects in structured light fields based on a graphics processor, so as to solve the challenges in the field of light field defect detection such as high time complexity caused by high data dimension and large data volume, and difficulty in achieving real-time online processing.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A real-time detection method for high-gloss surface defects using a structured light field based on a graphics processor comprises the following steps:

[0009] A1: The central processing unit (CPU) starts at least two threads in parallel to capture video frame data from the light field camera that captures the object under test and write the video frame data from the memory to the global cache of the graphics processing unit (GPU).

[0010] A2: The GPU and CPU asynchronously and in parallel start microlens feature calculations, segmenting the original light field image according to pre-input calibration information. Each grid, block, and thread of the GPU is used to calculate the feature values ​​of each frame of the light field image, each microlens image, and multiple pixels with a set position interval. The GPU also uses parallel calculations to obtain the statistical feature values ​​of the residual image under the microlens.

[0011] A3: Leveraging the spatial and temporal correlations of light field videos, we perform light field correlation fusion processing. Each thread of the GPU processes the fusion of each microlens' eigenvalues ​​in the final heat map. Each thread performs a weighted fusion of the eigenvalues ​​of each microlens within the frame with the eigenvalues ​​of the nearest neighboring microlenses and adds them to the final heat map using bilinear interpolation.

[0012] A4: Update the structured light field video defect detection results corresponding to the inspected object; preferably, provide real-time light field video and heat map feedback; preferably, also dynamically update the label corresponding to the inspected object to guide downstream workstation operations.

[0013] Further:

[0014] In step A1, the at least two threads include thread 1 and thread 2. Thread 1 is the acquisition card communication thread, responsible for communicating between the computer and the light field camera acquisition card, executing the acquisition card's corresponding image acquisition function, and transmitting the real-time video data stream to the computer memory via the data link. Thread 2 is the graphics processor communication thread, responsible for communicating with the graphics processor and transmitting the first video frame in the memory to the graphics processor's global cache via the data link when the graphics processor is in an inactive state.

[0015] Thread 1 continuously accesses the light field camera during its life cycle, and writes a new image frame into the memory when the light field camera captures it. Thread 2 continuously accesses the memory and the graphics processor during its life cycle, and when the graphics processor completes the previous round of computing operations and enters an inactive state and there are unprocessed image frames in the memory, they are removed from the memory and sent to the graphics processor global cache. The graphics processor performs serial logical operations internally, and obeys the scheduling of thread 2 on the host side.

[0016] In step A2, the GPU starts to perform calculations after receiving the data transmitted by thread 2 in step A1. Preferably, multiple threads are started in parallel to reduce and implement the calculation of eigenvalues ​​within the microlens.

[0017] Step A2 includes:

[0018] A21: Allocate computing and storage resources based on the light field image size.

[0019] A22: Complete the eigenvalue calculation in each microlens image within the block and write the calculation results back to the global cache.

[0020] Preferably:

[0021] Step A21 includes:

[0022] The GPU's hardware structure is divided into three levels: Grid, Block, and Thread. The storage structure, in descending order of latency, includes a global cache, a shared cache, and registers. Each frame of the original light field image is stored in the global cache, corresponding to the activation of a Grid. Each microlens image within the frame is stored in the shared cache, corresponding to a Block. The intermediate values ​​calculated within each microlens are stored in registers, corresponding to a Thread. The specific calculation operations are as follows:

[0023] Assume that the total number of microlenses in the original light field image processed by the system is n, and the width and height of each microlens image are w and h respectively. Then, n blocks are started, and n spaces are allocated in the global cache to store the calculation results in each microlens; w*h (or any multiple of this value downsampled) threads are started in each block.

[0024] Step A22 includes:

[0025] Each thread reads one or more pixel grayscale values ​​corresponding to the thread index (including downsampling factor information) in the microlens image corresponding to its block index into a register. Depending on the feature descriptor used, calculation operations such as averaging and squaring are performed on the grayscale values. During the operation, operations such as reduction can be used to maximize the performance of the graphics processor. The operations within each thread are described as follows:

[0026]

[0027] In the above formula:

[0028] G m —The value before the update of the mth position in the GPU global cache;

[0029] G′ m —The updated value of the mth position in the GPU global cache;

[0030] I m —mth microlens image;

[0031] i—Thread index;

[0032] w—width of the microlens image;

[0033] After completion, thread 0 accumulates the data to the location corresponding to the Block index in the global cache through shared memory to complete the calculation of the statistics.

[0034] Step A3 includes:

[0035] A31: Perform microlens spatial correlation fusion;

[0036] A32: Calculate the coordinates of the microlens time-domain fusion heat map;

[0037] A33: Perform bilinear interpolation on the resulting heat map.

[0038] Step A31 includes:

[0039] For each microlens image feature value obtained in step A2, a weighted sum is performed on it and its 18 nearest spatially neighboring microlenses. The 18 nearest-neighbor microlenses are arranged in a hexagonal array. The closest 6, next-nearest 6, and next-nearest 6 microlenses have the same distance from the central microlens and are divided into three groups. The specific fusion operation is as follows:

[0040]

[0041] In the above formula:

[0042] C m —The eigenvalue of the mth microlens calculated by A2;

[0043] σ n —The weighting coefficient corresponding to the nth group of microlenses of the nearest neighbor;

[0044] C′ m —Eigenvalue of the mth microlens after fusion.

[0045] Step A32 includes:

[0046] For each microlens fused feature value obtained in step A31, the position of the feature value in the final heat map is calculated based on the current frame index and the microlens position. The specific operation is as follows:

[0047] C t (x,y)=C(x,y)+tk-k0 (3)

[0048] In the above formula:

[0049] C(x,y)—Current microlens center position;

[0050] t—frame index number;

[0051] k, k0—correction coefficient;

[0052] C t (x,y)—The position of the current microlens center in the final heat map.

[0053] Step A33 includes:

[0054] For the position of the current microlens center in the final heat map obtained in step A32, bilinear interpolation is used to fuse the feature values ​​extracted from the microlens image into the final heat map. The specific operation is as follows:

[0055]

[0056]

[0057] Where R—final heat map;

[0058] x floor ,y floor —Integer downsampling result of the position of the current microlens center in the final heat map;

[0059] C′ m —Eigenvalue of the mth microlens after fusion.

[0060] In step A4, it is determined whether there are pixels in the heat map whose grayscale values ​​reach the grayscale value threshold, and a corresponding label is output based on the determination result; the updated heat map, real-time light field video frame, and label are drawn in the user view.

[0061] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for real-time detection of structured light field highlight surface defects.

[0062] The present invention has the following beneficial effects:

[0063] This paper proposes a real-time detection method for specular surface defects using a structured light field based on a graphics processing unit (GPU). This method efficiently implements a structured light field defect detection algorithm based on residual images on a heterogeneous parallel computer system. The GPU combines the distribution characteristics of light field data for processing, using grids, blocks, and threads to process light field image frames, microlens images, and pixels respectively. Different threads are used to aggregate the calculated microlens image eigenvalues, achieving computational resource allocation that matches the light field image structure. This method addresses the challenges of high data dimensionality and large data volumes, resulting in high time complexity and difficulty in achieving real-time online processing in the field of light field defect detection. Its main advantages include:

[0064] 1) Synchronously start multiple threads on the computer's central processing unit (CPU), separating the light field camera acquisition card data reading and CPU-GPU data exchange into two parallel threads, thereby hiding the system's computational latency.

[0065] 2) Perform microlens image feature extraction calculations on the GPU side, utilizing the differences in data dependencies inside and outside the microlens image to allocate computing resources. Thread blocks are aligned with the light field image data structure, improving GPU memory access efficiency and overall program execution efficiency on the GPU side.

[0066] 3) After completing single-frame detection, the results are fused into the final heatmap using bilinear interpolation, and the correlation between light field video data in the temporal and spatial domains is used to improve the heatmap detection effect;

[0067] 4) Provide visual feedback to end users based on the real-time update of the result heat map, and refresh the inspection label corresponding to the current workpiece in real time to guide the downstream workstations to further work. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of a method for real-time detection of highlight surface defects in a structured light field based on a graphics processor according to an embodiment of the present invention.

[0069] Figure 2 Schematic diagram of the connection and thread allocation of system components according to an embodiment of the present invention.

[0070] Figure 3 This is a diagram of the real-time visual feedback content composition of an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0072] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and coupling or communication.

[0073] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0074] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0075] The embodiment of the present invention proposes a real-time detection method for high-gloss surface defects in a structured light field based on a graphics processor, such as Figure 1 As shown, in some embodiments, the method includes the following steps:

[0076] A1: The central processing unit (CPU) starts at least two threads in parallel to capture video frame data from the light field camera that captures the object under test and write the video frame data from the memory to the global cache of the graphics processing unit (GPU).

[0077] A2: The GPU and CPU asynchronously and in parallel start microlens feature calculations, segmenting the original light field image according to pre-input calibration information. Each grid, block, and thread of the GPU is used to calculate the feature values ​​of each frame of the light field image, each microlens image, and multiple pixels with a set position interval. The GPU also uses parallel calculations to obtain the statistical feature values ​​of the residual image under the microlens.

[0078] A3: Leveraging the spatial and temporal correlations of light field videos, we perform light field correlation fusion processing. Each thread of the GPU processes the fusion of each microlens' eigenvalues ​​in the final heat map. Each thread performs a weighted fusion of the eigenvalues ​​of each microlens within the frame with the eigenvalues ​​of the nearest neighboring microlenses and adds them to the final heat map using bilinear interpolation.

[0079] A4: Dynamically updates the structured light field video defect detection results corresponding to the current workpiece, providing users with real-time light field video and heat map feedback. At the same time, it dynamically updates the label corresponding to the current workpiece to guide downstream workstations to perform sorting and other operations.

[0080] When performing the above steps in a specific embodiment, the following methods may be used. It should be noted that the specific methods used in the implementation process are only for illustration, and the scope of the present invention includes but is not limited to the following methods listed in the embodiments.

[0081] A1: The computer's central processing unit launches two threads in parallel to capture video frame data from the light field camera and write the video frame data from the memory to the graphics processor's global cache.

[0082] In actual operation, the acquisition process of video frame data can be directly read from the camera, or it can be read from the camera in real time through an acquisition card, or read from other storage media. The video frame data is acquired by the computer in real time in an automated manner.

[0083] Specifically, the relative relationships among the light field camera, computer CPU, memory, and graphics processor are shown in the attached abstract. Figure 2 The core of this method is to use the computer's central processing unit to launch at least two threads. Thread 1 is the acquisition card communication thread, responsible for communicating between the computer and the light field camera acquisition card, executing the acquisition card's corresponding image acquisition function, and transmitting the real-time video data stream to the computer's memory via a data link. Thread 2 is the graphics processor communication thread, responsible for communicating with the graphics processor and, when the graphics processor is inactive, transferring the first video frame in memory to the graphics processor's global cache via a data link. Threads 3-n are threads corresponding to executing other tasks (such as robotic arm movement and peripheral device communication).

[0084] In the program's execution logic, thread 1 continuously accesses the light field camera during its lifecycle, and writes new image frames into memory when the light field camera captures them. Thread 2 continuously accesses memory and the graphics processor during its lifecycle, and when the graphics processor completes the previous round of computing operations and enters an inactive state and there are unprocessed image frames in the memory, they are removed from the memory and sent to the graphics processor's global cache. The graphics processor performs serial logical operations internally and obeys the scheduling of thread 2 on the host side.

[0085] A2: The GPU and CPU asynchronously and in parallel start the microlens feature calculation, segment the original light field image according to the pre-input calibration information, and use the GPU parallel calculation to obtain the statistical feature values ​​of the residual image under the microlens.

[0086] When the graphics processor receives the data transmitted by thread 2 in A1, it starts to perform calculations.

[0087] Preferably, multiple threads are started in parallel to reduce and implement the calculation of eigenvalues ​​within the microlens.

[0088] Specifically, step A2 includes:

[0089] A21: Allocate computing and storage resources based on the light field image size.

[0090] A22: Complete the eigenvalue calculation in each microlens image within the block and write the calculation results back to the global cache.

[0091] In terms of hardware structure, the GPU is divided into three levels: Grid, Block, and Thread. In terms of storage structure, in descending order of latency, the GPU includes global memory, shared memory, and registers. In light field images, the data interdependence between pixels within each microlens is high, and feature calculations within the same microlens rely on all pixels within it. However, inter-pixel interdependence between different microlenses is low. This characteristic is exploited to allocate computing resources.

[0092] In a preferred embodiment, step A21 includes: each frame of the original light field image is stored in a global cache, corresponding to the start of a grid; each microlens image in the frame is stored in a shared cache, corresponding to a block; and the intermediate value calculated in each microlens is stored in a register, corresponding to a thread.

[0093] The specific calculation operations are as follows:

[0094] Assume that the total number of microlenses in the original light field image processed by the system is n, and the width and height of each microlens image are w and h respectively. Then, n blocks are started, and n spaces are allocated in the global cache to store the calculation results in each microlens; w*h (or any multiple of this value downsampled) threads are started in each block.

[0095] In a preferred embodiment, step A22 includes:

[0096] Each thread reads one or more pixel grayscale values ​​corresponding to the thread index (including downsampling factor information) in the microlens image corresponding to its block index into a register. Depending on the feature descriptor used, calculation operations such as averaging and squaring are performed on the grayscale values. During the operation, operations such as reduction can be used to maximize the performance of the graphics processor. The operations within each thread are described as follows:

[0097]

[0098] In the above formula:

[0099] G m —The value before the update of the mth position in the GPU global cache;

[0100] G′ m —The updated value of the mth position in the GPU global cache;

[0101] I m —mth microlens image;

[0102] i—Thread index;

[0103] w—width of the microlens image;

[0104] After completion, thread 0 accumulates the data to the location corresponding to the Block index in the global cache through shared memory to complete the calculation of the statistics.

[0105] A3: Utilize the spatial and temporal correlation of light field videos to perform light field correlation fusion processing. The eigenvalues ​​of each microlens in the frame are weightedly fused with the eigenvalues ​​of the nearest neighboring microlens and added to the final result heat map using bilinear interpolation.

[0106] A31: Microlens spatial correlation fusion;

[0107] Specifically, for each microlens image feature value obtained in step A2, a weighted sum is performed on it and its 18 nearest spatially neighboring microlenses. The 18 nearest neighbor microlenses are arranged in a hexagonal array. The closest 6, next closest 6, and next closest 6 microlenses have the same distance from the center microlens and are divided into three groups. The specific fusion operation is as follows:

[0108]

[0109] In the above formula:

[0110] C m —The eigenvalue of the mth microlens calculated by A2;

[0111] σ n —The weighting coefficient corresponding to the nth group of microlenses of the nearest neighbor;

[0112] C′ m —Eigenvalue of the mth microlens after fusion.

[0113] A32: Microlens time-domain fusion heat map coordinate calculation;

[0114] Specifically, for the fused feature value of each microlens obtained in step A31, the position of the feature value in the final heat map is calculated based on the current frame index and the microlens position. The specific operation is as follows:

[0115] C t (x,y)=C(x,y)+tk-k0 (3)

[0116] In the above formula:

[0117] C(x,y)—Current microlens center position;

[0118] t—frame index number;

[0119] k, k0—correction coefficient;

[0120] C t (x,y)—The position of the current microlens center in the final heat map.

[0121] A33: Bilinear interpolation of result heatmap;

[0122] Specifically, for the position of the current microlens center in the final heat map obtained in step A32, bilinear interpolation is used to fuse the feature values ​​extracted from the microlens image into the final heat map. The specific operations are as follows:

[0123]

[0124]

[0125] In the above formula:

[0126] R—Final heatmap

[0127] x floor ,y floor —Integer downsampling result of the position of the current microlens center in the final heat map;

[0128] C′ m —Eigenvalue of the mth microlens after fusion.

[0129] The meanings of other symbols are the same as above.

[0130] A4: Dynamically updates the structured light field video defect detection results corresponding to the current workpiece, providing users with real-time light field video and heat map feedback. At the same time, it dynamically updates the label corresponding to the current workpiece to guide downstream workstations to perform sorting and other operations.

[0131] Specifically, after the thermal map is updated in step A3, it is determined whether there are pixels in the thermal map whose grayscale value reaches 255 (the maximum grayscale value of an 8-bit image). If so, the label "NG" is output; otherwise, the label "OK" is output. The host uses the central processing unit to call the image drawing API to draw the updated thermal map, real-time light field video frame, and label in the user view. The user view structure is shown in the attached summary. Figure 3 shown.

[0132] The real-time detection method for specular surface defects using a structured light field, based on a graphics processing unit (GPU), proposed in this invention efficiently implements a residual image-based structured light field defect detection algorithm on a heterogeneous parallel computer system. In a preferred embodiment, the GPU performs corresponding calculations based on the distribution characteristics of light field data, addressing challenges in the field of light field defect detection, such as high data dimensionality and large data volumes, resulting in high time complexity and difficulty in achieving real-time online processing. Its main advantages include:

[0133] By synchronously starting multiple threads on the computer's central processing unit (CPU), the data reading from the light field camera acquisition card and the data exchange between the CPU and GPU are separated into two parallel threads, thereby hiding the system's computational latency.

[0134] Microlens image feature extraction is performed on the GPU side, and the difference in data dependencies inside and outside the microlens image is used to allocate computing resources. The thread blocks are aligned with the light field image data structure, which improves the GPU's memory access efficiency and enhances the overall execution efficiency of the program on the GPU side.

[0135] After completing the single-frame detection, the results are fused into the final heat map using bilinear interpolation, and the correlation between light field video data in the time domain and spatial domain is used to improve the heat map detection effect.

[0136] Based on the real-time update of the result heat map, the visual feedback provided to the end user is refreshed in real time, and the corresponding inspection label of the current workpiece is refreshed in real time to guide the downstream workstations to further operate.

[0137] The present invention utilizes the difference in data dependency inside and outside the microlens image to allocate computing resources on the GPU side, aligns thread blocks with the light field image data structure, improves the GPU's memory access efficiency, and enhances the overall execution efficiency of the program on the GPU side.

[0138] The present invention provides terminal industrial users with visual feedback based on real-time updates of result heat maps, refreshes the detection labels corresponding to the current workpiece in real time, and guides further operations of downstream workstations, which has important application value.

[0139] The present invention can achieve real-time online light field defect detection. The proposed GPU-based acceleration method can increase the speed of the defect detection algorithm to sub-millisecond level, enabling detection within input delays and truly serving production applications.

[0140] The present invention can be widely applied in aerospace, automotive, and other precision manufacturing fields, enabling the detection of defects such as pits and convex hulls on high-gloss surfaces found in various industrial production processes. Taking the automotive industry as an example, the present invention can effectively enhance the level of automated quality inspection on automotive production lines, effectively improving work efficiency.

[0141] An embodiment of the present invention further provides a control device comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the structured light field highlight surface defect real-time detection method when executing the computer program.

[0142] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for real-time detection of high-gloss surface defects in a structured light field is implemented.

[0143] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0144] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0146] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0148] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0149] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0150] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0151] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0152] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0153] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A real-time detection method for high-gloss surface defects using a structured light field based on a graphics processor, characterized in that: The steps include: A1: The central processing unit (CPU) starts at least two threads in parallel to capture video frame data from the light field camera that captures the object under test and write the video frame data from the memory to the global cache of the graphics processing unit (GPU). A2: The GPU and CPU asynchronously and in parallel start microlens feature calculations, segmenting the original light field image according to pre-input calibration information. Each grid, block, and thread of the GPU is used to calculate the feature values ​​of each frame of the light field image, each microlens image, and multiple pixels with a set position interval. The GPU also uses parallel calculations to obtain the statistical feature values ​​of the residual image under the microlens. A3: Leveraging the spatial and temporal correlations of light field videos, we perform light field correlation fusion processing. Each thread of the GPU processes the fusion of each microlens' eigenvalues ​​in the final heat map. Each thread performs a weighted fusion of the eigenvalues ​​of each microlens within the frame with the eigenvalues ​​of the nearest neighboring microlenses and adds them to the final heat map using bilinear interpolation. Step A3 specifically includes: A31: performing microlens spatial correlation fusion, wherein each microlens eigenvalue obtained in step A2 is weightedly fused with the eigenvalue of the nearest neighboring microlens; A32: Calculate the coordinates of the microlens time-domain fusion heat map. For each microlens fusion feature value obtained in step A31, calculate the position of the feature value in the final heat map based on the current frame index and the microlens position. A33: performing bilinear interpolation on the resulting heat map, wherein for the position of the current microlens center in the final heat map obtained in step A32, bilinear interpolation is used to fuse the feature values ​​extracted from the microlens image into the final heat map; A4: Update the structured light field video defect detection results corresponding to the inspected object.

2. The method for real-time detection of surface defects using structured light fields according to claim 1, wherein: In step A1, the at least two threads include thread 1 and thread 2. Thread 1 is the acquisition card communication thread, responsible for communicating between the computer and the light field camera acquisition card, executing the acquisition card's corresponding image acquisition function, and transmitting the real-time video data stream to the computer memory via the data link. Thread 2 is the graphics processor communication thread, responsible for communicating with the graphics processor and transmitting the first video frame in the memory to the graphics processor's global cache via the data link when the graphics processor is in an inactive state.

3. The method for real-time detection of surface defects using structured light fields according to claim 2, wherein: Thread 1 continuously accesses the light field camera during its life cycle, and writes a new image frame into the memory when the light field camera captures it. Thread 2 continuously accesses the memory and the graphics processor during its life cycle, and when the graphics processor completes the previous round of computing operations and enters an inactive state and there are unprocessed image frames in the memory, the unprocessed image frames are removed from the memory and sent to the global cache of the graphics processor. The graphics processor performs serial logic operations internally and obeys the scheduling of thread 2 on the host side.

4. The method for real-time detection of surface defects using structured light fields according to any one of claims 2 to 3, wherein: In step A2, after receiving the data transmitted by thread 2 in step A1, the graphics processor starts to perform calculations; at the same time, multiple threads are started in parallel to reduce and calculate the eigenvalues ​​within the microlens; Step A2 includes: A21: Allocate computing and storage resources based on the light field image size. A22: Complete the eigenvalue calculation for each microlens image within the block and write the calculation results back to the global cache; Step A21 includes: The GPU's hardware structure is divided into three levels: Grid, Block, and Thread. The storage structure, in descending order of latency, includes a global cache, a shared cache, and registers. Each frame of the original light field image is stored in the global cache, corresponding to the activation of a Grid. Each microlens image within the frame is stored in the shared cache, corresponding to a Block. The intermediate values ​​calculated within each microlens are stored in registers, corresponding to a Thread. The specific calculation operations are as follows: Assume that the total number of microlenses in the original light field image processed by the system is n, and the width and height of each microlens image are w and h respectively. Then, n blocks are started, and n spaces are allocated in the global cache to store the calculation results of each microlens; w*h threads or any multiple of w*h threads are started in each block to downsample; Step A22 includes: Each thread reads one or more pixel grayscale values ​​corresponding to the thread index in the microlens image corresponding to its block index into a register. Depending on the feature descriptor used, calculation operations such as averaging and squaring are performed on the grayscale values. Reduction operations can be used during the operation to maximize the performance of the graphics processor. The operations within each thread are described as follows: In the above formula: G m —The value before the update of the mth position in the GPU global cache; G′ m —The updated value of the mth position in the GPU global cache; I m —mth microlens image; i—Thread index; w—width of the microlens image; After completion, thread 0 accumulates the data to the location corresponding to the Block index in the global cache through shared memory to complete the calculation of the statistics.

5. The method for real-time detection of surface defects using structured light fields according to any one of claims 1 to 3, wherein: Step A31 includes: For each microlens image feature value obtained in step A2, a weighted sum is performed on it and its 18 nearest spatially neighboring microlenses. The 18 nearest-neighbor microlenses are arranged in a hexagonal array. The closest 6, next-nearest 6, and next-nearest 6 microlenses have the same distance from the central microlens and are divided into three groups. The specific fusion operation is as follows: In the above formula: C m —The eigenvalue of the mth microlens calculated by A2; σ n —The weighting coefficient corresponding to the nth group of microlenses of the nearest neighbor; C′ m —Eigenvalue of the mth microlens after fusion.

6. The method for real-time detection of surface defects using structured light fields according to claim 5, wherein: Step A32 includes: For each microlens fused feature value obtained in step A31, the position of the feature value in the final heat map is calculated based on the current frame index and the microlens position. The specific operation is as follows: C t (x,y)=C(x,y)+tk-k0 (3) In the above formula: C(x,y)—Current microlens center position, x and y represent the coordinates of the position; t—frame index number; k, k0—correction coefficient; C t (x,y)—The position of the current microlens center in the final heat map.

7. The method for real-time detection of surface defects using structured light fields according to claim 6, wherein: Step A33 includes: For the position of the current microlens center in the final heat map obtained in step A32, bilinear interpolation is used to fuse the feature values ​​extracted from the microlens image into the final heat map. The specific operation is as follows: Where R—final heat map; x floor ,y floor —Integer downsampling result of the position of the current microlens center in the final heat map; C′ m —Eigenvalue of the mth microlens after fusion.

8. The method for real-time detection of surface defects using structured light fields according to any one of claims 1 to 3, wherein: In step A4, real-time light field video and thermal map feedback are also provided to dynamically update the labels corresponding to the inspected objects to guide downstream workstation operations. Specifically, it includes: determining whether there are pixels in the thermal map whose grayscale values ​​reach the grayscale value threshold, and outputting the corresponding labels based on the judgment results; and drawing the updated thermal map, real-time light field video frame, and labels in the user view.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the real-time detection method for highlight surface defects in a structured light field is implemented.

Images