Multi-camera fused workpiece surface defect detection method and related equipment

Abstract

The invention provides a multi-camera fusion workpiece surface defect detection method and related equipment, and the method comprises the steps: employing a plurality of industrial cameras to carry out the video collection of the surface of a rotary body machined part, so as to generate the video data of the surface of the rotary body machined part; decoding the video data to obtain each frame of image in the video; cutting and splicing the obtained frame images to form an overall or local unfolded image of the workpiece surface; and performing image processing analysis on the expanded image to detect defects on the surface of the workpiece. In combination with flexible configuration of a plurality of industrial cameras, a high-speed video acquisition technology is adopted, and optical flexibility which is incomparable with that of a common line scanning camera is realized. The configuration not only adapts to various complex industrial scenes, but also has the characteristic of large depth of field, and can ensure that clear images can be obtained under different working distances. And the method has very strong universality for detecting the surface defects of the outer side wall of a metal rotary body machined part with a curved surface and high reflectivity.

Description

Technical Field

[0001] The present application relates to the field of machine vision detection technology, and in particular to a multi-camera fusion workpiece surface defect detection method and related equipment. Background Art

[0002] In the current industrial inspection field, machine vision technology is widely used in product inspection. It mainly uses industrial cameras to shoot the inspected products, which are usually saved in the computer in the form of pictures. The image signal captured by the camera is first converted into a digital signal that can be processed by the computer, and then transmitted to the computer's memory through a high-speed bus. On this basis, the machine vision software will conduct in-depth analysis of these digital images to determine the quality of the product and whether it meets the preset standards, so as to achieve the purpose of image inspection. Therefore, the imaging effect plays a vital role in machine vision inspection.

[0003] When using an area array camera for imaging, if the inspected product is a metal object with a specific high-reflective curved surface, the changes in the curvature and reflective characteristics of the metal surface may cause uneven distribution of image brightness, increasing the difficulty of image processing. The working principle of line scan cameras is different from that of common area array cameras. Area array cameras can obtain a complete two-dimensional image in a single exposure process, while line scan cameras only shoot one line on the object at a time (the width of this line is usually one pixel). During the shooting process, the line scan camera will gradually combine these lines by scanning line by line, and finally form a complete two-dimensional image. Although line scan cameras have certain advantages in imaging highly reflective curved objects, in actual applications, they have extremely high requirements for the installation position, and any slight position deviation (more than 0.05mm) may have a significant impact on the imaging effect.

[0004] In view of the limitations of area array cameras in processing images of highly reflective curved metal objects and the strict requirements of line scan cameras on the installation position, the present application provides a multi-camera fusion workpiece surface defect detection method, which provides a new solution for workpiece inspection. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present application provides a multi-camera fusion workpiece surface defect detection method and related equipment to at least solve the problems in the prior art of imaging difficulties and defect detection limitations of high-reflectivity metal rotating bodies.

[0006] In order to achieve the above objectives and other advantages, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a multi-camera fusion workpiece surface defect detection method, which is applied to defect detection of a rotating workpiece with a curved surface shape, high light reflectivity, and made of a metal material, and the method comprises:

[0008] Using a plurality of industrial cameras to collect video of the surface of the rotating workpiece to generate video data of the surface of the rotating workpiece;

[0009] Decoding the video data to obtain each frame of image in the video;

[0010] The acquired frame images are cropped and spliced ​​to form an expanded image of the entire or partial workpiece surface;

[0011] Image processing and analysis are performed on the expanded image to detect defects on the surface of the workpiece.

[0012] According to a multi-camera fusion workpiece surface defect detection method provided by the present application, the step of using a plurality of industrial cameras to collect video of the surface of the rotating workpiece to generate video data of the surface of the rotating workpiece includes:

[0013] A plurality of industrial cameras are arranged above or around the rotating workpiece, each of the industrial cameras being configured to take a video of a specific local position of the rotating workpiece;

[0014] Starting the industrial cameras to simultaneously capture video of each specific local position of the rotating workpiece, ensuring that the shooting range of each camera covers an independent area on the surface of the rotating workpiece with overlapping edges;

[0015] The local video data captured by each of the industrial cameras are spliced ​​into video data of the surface of the rotating workpiece.

[0016] According to a multi-camera fusion workpiece surface defect detection method provided by the present application, the surface of the rotating workpiece includes a plurality of rotating surfaces of different diameters, and the diameters of the rotating surfaces corresponding to the specific local positions photographed by each of the industrial cameras are different;

[0017] The step of arranging a plurality of industrial cameras above or around the rotating workpiece, each of the industrial cameras being configured to take a video of a specific local position of the rotating workpiece, comprises:

[0018] Determining the diameters of different rotational surfaces on the surface of the rotating workpiece;

[0019] A maximum processing frame rate is set for the rotating surface with the largest diameter, and the processing frame rate of the corresponding industrial camera is set according to the diameter size with the maximum processing frame rate as a reference standard, wherein a rotating surface with a larger diameter corresponds to a higher processing frame rate, and a rotating surface with a smaller diameter corresponds to a lower processing frame rate;

[0020] The set different processing frame rates are configured in the parameters of the corresponding industrial cameras, so that each of the industrial cameras can shoot the corresponding rotating surface on the rotating body workpiece according to the set processing frame rate.

[0021] According to a multi-camera fusion workpiece surface defect detection method provided by the present application, the step of decoding the video data to obtain each frame image in the video includes:

[0022] Receiving a video code stream after encoding and compressing the video data, wherein the video code stream includes video parameter information and compressed image data, wherein the parameter information includes: video parameters and encoding format;

[0023] Parsing the video code stream based on the encoding format and the video parameters, and acquiring image data in the video frame by frame;

[0024] Traversing each frame of image data, and determining the frame image to be extracted according to the set frame extraction parameters, wherein the frame extraction parameters include: a start frame, an end frame, a frame extraction interval, and a frame number;

[0025] Each of the selected frame images is saved separately as an image file.

[0026] According to a multi-camera fusion workpiece surface defect detection method provided by the present application, the step of parsing the video stream based on the encoding format and the video parameters and acquiring image data in the video frame by frame includes:

[0027] For the image frame data in the video code stream, using the transform coding technology in the video encoding process, performing inverse quantization and inverse transformation processing in the video decoding process, mapping the quantized coefficients back to the original or nearly original transform coefficient values, and converting the transform coefficient values ​​back to the spatial domain to restore to the uncompressed original image data;

[0028] At the same time, in the video decoding process, the predictive coding technology in the video encoding process is used to perform intra-frame prediction on the correlation between adjacent pixels in the same frame image, and inter-frame prediction on the motion information between adjacent frame images, so as to reconstruct the image data using the inverse process of the predictive coding technology;

[0029] The video decoding process is performed frame by frame to obtain and reconstruct the image data in the video until the entire video code stream is decoded.

[0030] According to a multi-camera fusion workpiece surface defect detection method provided by the present application, the step of cropping and splicing the acquired frame images to form an expanded image of the entire or partial workpiece surface includes:

[0031] According to the workpiece part where defects need to be detected, the acquired frame image is cropped according to preset cropping parameters;

[0032] The cropped multiple frame images are spliced ​​according to a predetermined splicing order and rule to form an expanded image of the entire or partial workpiece surface;

[0033] The expanded image of the workpiece surface after splicing is saved as an image file.

[0034] A multi-camera fusion workpiece surface defect detection method provided by the present application also includes: using multi-core CPU and CUDA architecture parallel processing technology to perform real-time image stitching of local video data of multiple industrial cameras to form video data of the surface of a rotating workpiece, and decoding and stitching the video data to form an expanded image of the workpiece surface.

[0035] In a second aspect, the present application provides an electronic device, the electronic device comprising:

[0036] One or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes any one of the multi-camera fusion workpiece surface defect detection methods described above.

[0037] In a third aspect, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements a multi-camera fusion workpiece surface defect detection method as described in any one of the above.

[0038] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements a multi-camera fusion workpiece surface defect detection method as described in any one of the above.

[0039] The present application provides a multi-camera fusion workpiece surface defect detection method and related equipment, which are applied to the defect detection of a rotating body workpiece with a curved surface shape, high reflectivity to light, and made of metal material. The method includes: using several industrial cameras to collect video of the surface of the rotating body workpiece to generate video data of the surface of the rotating body workpiece; decoding the video data to obtain each frame image in the video; cutting and splicing the acquired frame images to form an expanded image of the entire or partial workpiece surface; performing image processing and analysis on the expanded image to detect defects on the workpiece surface. The present application combines the flexible configuration of multiple industrial cameras to collect video of the surface of the rotating body workpiece, which can ensure that image information of the workpiece surface is obtained from multiple angles and positions. The high-speed video acquisition technology is used to achieve optical flexibility that ordinary line scan cameras cannot match. This configuration is not only suitable for various complex industrial scenes, but also has a large depth of field characteristic, which can ensure that clear images can be obtained at different working distances. It has strong versatility for the detection of the outer wall of metal rotating body workpieces with curved surfaces and high reflectivity. By configuring the number of camera stations and adjusting the focal length, surface defect detection of rotating workpieces of any length can be easily achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a schematic diagram of the process of a multi-camera fusion workpiece surface defect detection method provided in an embodiment of the present application;

[0042] Figure 2 It is a schematic diagram of the dimensions of a rotating body processing part provided in an embodiment of the present application;

[0043] Figure 3 It is an original image of a certain part of a rotating body workpiece and a corresponding surface development image provided in an embodiment of the present application;

[0044] Figure 4 It is an original image of another part of the rotating body workpiece provided in the embodiment of the present application and a corresponding surface development image;

[0045] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail with the accompanying drawings.

[0047] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in this application should be the usual meanings understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in this application are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0048] The surface reflectivity of rotating workpieces made of metal materials is strong, which can easily cause reflection and glare when shooting with an area array camera, thus affecting the clarity and contrast of the image. In addition, changes in the surface curvature and reflective properties of metal bodies can lead to uneven distribution of image brightness, increasing the difficulty of image processing. Therefore, when processing ordinary images of highly reflective metal bodies, area array cameras have limitations in terms of limited imaging quality, limited measurement accuracy, limited applicable scenarios, cost and technical difficulty. Therefore, the present application provides a multi-camera fusion workpiece surface defect detection method and related equipment, and takes corresponding measures to optimize the image acquisition and processing process based on these characteristics to improve the accuracy and efficiency of defect detection.

[0049] Reference Figure 1 As shown, the embodiment of the present application provides a multi-camera fusion workpiece surface defect detection method, which is applied to defect detection of a rotating body workpiece with a curved surface shape, high reflectivity to light, and made of a metal material. The method includes:

[0050] Step S1: using a plurality of industrial cameras to capture video of the surface of the rotating workpiece to generate video data of the surface of the rotating workpiece.

[0051] In this embodiment, step S1 specifically includes:

[0052] Step S101: a plurality of industrial cameras are arranged above or around the rotating workpiece, and each industrial camera is configured to take a video of a specific local position of the rotating workpiece.

[0053] For example, Figure 2 The design drawing of the rotating body workpiece shown is a typical rotating body shape, which rotates continuously around a fixed axis. The rotating body workpiece has multiple rotating surfaces of different diameters. That is, the peripheral surface of the workpiece is not of uniform diameter, but presents different diameter sizes at different positions according to design requirements. At one end of the workpiece (for example, the left end), there is a rotating surface with a larger diameter, and as the peripheral surface extends to the other end (for example, the right end), the diameter gradually decreases. At the other end of the workpiece, there is a rotating surface with a smaller diameter.

[0054] Since the rotating workpiece has multiple rotating surfaces of different diameters, the shooting height of the industrial camera needs to be adjusted according to the different diameters. When the camera shoots a rotating surface with a larger diameter, in order to fully capture the image of the area, the camera needs to be adjusted to a higher shooting height to expand the field of view. On the contrary, when shooting a rotating surface with a smaller diameter, the camera can appropriately lower the shooting height.

[0055] In this embodiment, step S101 specifically includes:

[0056] Step S1011: determining the diameters of different rotational surfaces on the surface of the rotating workpiece;

[0057] Step S1012: setting a maximum processing frame rate for the rotating surface with the largest diameter, taking the maximum processing frame rate as a reference standard, and setting the processing frame rate of the corresponding industrial camera according to the diameter size, wherein a rotating surface with a larger diameter corresponds to a higher processing frame rate, and a rotating surface with a smaller diameter corresponds to a lower processing frame rate;

[0058] Step S1013: configuring the set different processing frame rates in the parameters of the corresponding industrial cameras, so that each industrial camera can shoot the corresponding rotating surface on the rotating workpiece according to the set processing frame rate.

[0059] Specifically, the different diameters of each rotating surface are determined according to the diameters of each rotating surface marked on the design drawings. After determining the diameters of each rotating surface, it is necessary to select a suitable maximum processing frame rate. This frame rate is usually determined based on the shooting requirements of the rotating surface with the largest diameter to ensure that the image of the rotating surface can be captured clearly and continuously. With the maximum processing frame rate as a reference standard, the processing frame rate of the corresponding industrial camera is set proportionally according to the diameter size of other rotating surfaces.

[0060] For example, three to four cameras are used. Figure 2 The outer surface of the rotating workpiece shown in the figure is imaged, and the image is cropped into 1280×30 pixels. This cropping operation significantly improves the processing frame rate of the camera, and the processing frame rate can be increased from the standard 216.25fps to 1000fps.

[0061] In this working environment, the camera's field of view is set to 96mm × 76.8mm. Based on this field of view and the cropped pixel size, the calculated single pixel accuracy is: 96mm / 1280pixel = 0.075mm / pixel, which means that each pixel in the image corresponds to 0.075mm on the actual object.

[0062] In order to capture the video, the camera is then adjusted to the video capture mode. Taking the first camera as an example, the diameter of the product captured by the camera is φ87.8mm (calculated approximately to φ88mm, such as Figure 2 At the middle mark ①), the circumference is calculated by the formula π×D, which is 276.32 mm. At this time, under the cropping effect, the video acquisition speed of the camera is: 1000fps×0.075mm / pixel=75mm / s.

[0063] In order to ensure that the collected video is synchronized with the product rolling, the rotation speed of the outer surface needs to be accurately set. This rotation speed is obtained by multiplying the collection speed of 75 mm per second by 60 (converting the speed unit from mm per second to mm per minute), and then dividing it by the circumference of the product 276.32 mm. The calculation result of the set outer surface rotation speed is: 60×collection speed / circumference=60×75 / 276.32=16.285r / min (minute).

[0064] After setting the above rotation speed, the video recording time can be further determined. This time is calculated by dividing the circumference of the product, 276.32 mm, by the acquisition speed of 75 mm per second. The result is 276.32 / 75 = 3.684 seconds. Based on the setting of the first camera, the frame rate settings of the other cameras can be inferred (the frame rate setting can be set in the camera software, or the camera configuration interface can be called using the programming interface to complete the setting) to ensure that all cameras are collected under the same rotation speed conditions, thereby ensuring the synchronization and accuracy of the entire system. According to Figure 2 The half-surface radius of the rotating surface at mark ① is 44 mm and the half-surface radius of the rotating surface at mark ② is 22 mm. The camera recording parameters are shown in Table 1 below.

[0065] Table 1 Recording parameters of rotating surfaces with different diameters

[0066]

[0067] Step S102: Start the industrial camera to simultaneously capture video of each specific local position of the rotating workpiece, ensuring that the shooting range of each camera covers an independent area on the surface of the rotating workpiece with overlapping edges.

[0068] Specifically, each camera is equipped with a corresponding lens to capture different areas of the outer surface of the workpiece. The light source provides the necessary lighting to ensure that the captured image is clear. The rotating workpiece is clamped on the rolling mechanism, and the belt drives the product to rotate. As the product rotates, its outer surface passes through the camera's shooting area in turn. Multiple cameras are installed in a fixed position, and the lenses are aimed at the rotating workpiece. When the workpiece rotates into the camera's shooting range, the camera captures the image of the area. Since there are multiple cameras, they can capture different parts of the outer surface of the workpiece respectively. According to the time the camera needs to record, the speed of the rolling belt is adjusted to accurately control the rotation speed of the workpiece surface. The camera shooting can be triggered synchronously according to the movement of the belt to ensure that the workpiece surface is photographed at the best position. The position and intensity of the light source can be adjusted according to the shooting requirements to ensure that clear images can be obtained at different shooting angles. This helps to reduce shadows and reflections and improve the contrast and detail performance of the image.

[0069] It should be noted that the shooting range of each camera needs to be carefully designed to ensure that they cover independent areas of the workpiece surface and that there is a certain amount of edge overlap between these areas. The edge overlap is to achieve seamless connection and avoid information omission or misalignment in the subsequent video data splicing process.

[0070] For example, the industrial camera can be a high-frame-rate industrial camera, whose frame rate can be as high as 1000fps. The industrial camera adopts the transmission speed of USB3.0, which is greatly improved compared with USB2.0 and GigE camera (Gigabit Ethernet, Gigabit Ethernet industrial camera). With a transmission bandwidth of 5Gbps, combined with a high-speed sensor, the data transmission time can be as low as 1.2 milliseconds under the shooting conditions of 300,000 pixels and 800 frames per second (FPS), and even under a high-resolution camera of 20 million pixels, a transmission speed of up to 18 frames per second can be achieved. Benefiting from the ample transmission bandwidth, USB3.0 technology enables a computer to be compatible with and connect to multiple cameras while maintaining a high frame rate, which undoubtedly improves the flexibility and scalability of the system. In addition, it performs well in terms of CPU occupancy. It adopts direct memory access (DMA) transmission mode, which means that the data transmission process will hardly occupy CPU resources, thereby ensuring the efficient operation of the system and the rational use of resources.

[0071] Step S103: stitching the local video data captured by each industrial camera into video data of the surface of the rotating workpiece.

[0072] Specifically, this is a key step in multi-camera fusion. Through image processing algorithms, the local video data captured by each camera are stitched together to form a video data that can cover the entire or partial surface of the workpiece. In this process, the information of the overlapping edges will be used to calibrate and adjust the positions between different video frames to ensure that the stitched video data is continuous in time and space. Before video stitching, the camera needs to be calibrated to obtain the internal and external parameters of the camera. These parameters are used for subsequent image registration and fusion. Ensure that the video streams from different cameras are synchronized in time to avoid misalignment or breakage after stitching.

[0073] Exemplarily, a video stream is obtained from each camera, and each frame of the image is preprocessed, including denoising, resizing, etc., and a feature detection algorithm (such as SIFT, SURF or ORB) is used to extract key points and descriptors in each frame of the image, and then the best matching point is found through a feature matching algorithm (such as FLANN or BFMatcher). The homography matrix is ​​calculated based on the matching points, and the image is perspectively transformed using the homography matrix so that the images taken by different cameras are aligned to the same coordinate system. The aligned images are fused to generate a seamlessly spliced ​​video screen. The video splicing process generally involves multi-band fusion, image smoothing and other technologies to reduce the visibility of the seams. Therefore, through the above steps and the application of key algorithms, the local video data captured by each camera can be spliced ​​together to form a video data covering the entire or local surface of the workpiece.

[0074] Step S2: Decode the video data to obtain each frame of the video.

[0075] In this embodiment, step S2 specifically includes:

[0076] Step S201: receiving a video code stream obtained by encoding and compressing video data, wherein the video code stream includes video parameter information and compressed image data, and the parameter information includes: video parameters and encoding format.

[0077] Specifically, the video bitstream is the video data after encoding and compression, which contains the video parameter information and the actual image data. The parameter information is crucial for the subsequent decoding process, which tells the decoder how to correctly parse and restore the video data. These parameter information usually include video parameters (such as resolution, frame rate, etc.) and encoding formats (such as H.264, H.265, etc.).

[0078] Step S202: parsing the video code stream based on the encoding format and video parameters, and acquiring image data in the video frame by frame.

[0079] In this embodiment, step S202 specifically includes:

[0080] For the image frame data in the video code stream, the transform coding technology in the video encoding process is used to perform inverse quantization and inverse transformation in the video decoding process, the quantized coefficients are mapped back to the original or nearly original transform coefficient values, and the transform coefficient values ​​are converted back to the spatial domain to restore to the uncompressed original image data;

[0081] At the same time, in the video decoding process, the predictive coding technology in the video encoding process is used to perform intra-frame prediction on the correlation between adjacent pixels in the same frame image, and inter-frame prediction on the motion information between adjacent frame images, so as to reconstruct the image data using the inverse process of the predictive coding technology;

[0082] The video decoding process is performed frame by frame to obtain and reconstruct the image data in the video until the entire video code stream is decoded.

[0083] Specifically, in the video encoding process, transform coding technology transforms and encodes the pixel values ​​in the image or video frame, such as discrete cosine transform (DCT) or discrete wavelet transform (DWT), and quantizes the transformed coefficients to reduce the amount of data. In the decoding process, these quantized coefficients need to be dequantized first, that is, the quantized coefficient values ​​are mapped back to the original or nearly original transform coefficient values. The transform coefficient values ​​obtained after dequantization are still in the transform domain (such as the frequency domain), and these coefficient values ​​need to be converted back to the spatial domain (i.e., the pixel domain) through inverse transform processing to restore them to the uncompressed original image data. The inverse transform processing usually corresponds to the transform algorithm used in the encoding process, such as the inverse transform of discrete cosine transform, the inverse transform of discrete wavelet transform, etc.

[0084] In order to remove the temporal and spatial redundancy in the video, video coding uses predictive coding technology. In the video decoding process, the predictive coding technology used in the video coding process is used for intra-frame prediction and inter-frame prediction. For intra-frame prediction, there is usually a strong correlation between adjacent pixels within the same frame image. The decoder will use this correlation to perform intra-frame prediction, that is, predict the current pixel value based on the decoded adjacent pixel values. By calculating the residual between the predicted value and the true value and decoding this residual, the accurate value of the current pixel can be obtained. For inter-frame prediction, there is motion information between adjacent frame images due to the temporal continuity of the video sequence. The decoder will use inter-frame prediction technology to predict the pixel value of the current frame by finding the motion vector between adjacent frames. This usually involves the process of motion estimation and compensation, that is, generating the predicted image of the current frame based on the decoded adjacent frame images and motion vectors, and then decoding the residual to obtain the accurate image of the current frame.

[0085] The decoder will perform the above decoding process frame by frame according to the frame order in the video bitstream. For each frame of image data, inverse quantization, inverse transformation, intra-frame prediction and inter-frame prediction are performed in sequence to restore the uncompressed original image data. The restored original image data is not compressed or lost, so more detailed information and details are retained, making the image clearer and more realistic. Using the original image data for defect detection can reduce the impact of noise and distortion introduced during image compression or processing on the detection results.

[0086] Step S203: traverse each frame of image data, and determine the frame image to be extracted according to the set frame extraction parameters, where the frame extraction parameters include: a start frame, an end frame, a frame extraction interval, and a frame number;

[0087] Step S204: Save each selected frame image separately as an image file.

[0088] The decoder will determine the frame image to be extracted based on the set frame extraction parameters. The frame extraction parameters include the start frame, end frame, frame extraction interval, and number of frames. By traversing each frame image data and filtering according to these parameters, the decoder can accurately extract the frame image required by the user. Finally, the decoder will save each selected frame image separately as an image file. These image files are used for subsequent splicing processing. The saving format can be selected according to user needs, such as JPEG, PNG, etc.

[0089] Step S3: cropping and splicing the acquired frame images to form an expanded image of the entire or partial workpiece surface.

[0090] In this embodiment, step S3 specifically includes:

[0091] Step S301: cropping the acquired frame image according to preset cropping parameters according to the workpiece part where defects need to be detected;

[0092] Step S302: splicing the cropped multiple frame images according to a predetermined splicing order and rule to form an expanded image of the entire or partial workpiece surface;

[0093] Step S303: saving the expanded image of the spliced ​​workpiece surface as an image file.

[0094] Specifically, the part of the workpiece that needs to be inspected for defects can be identified first, such as the entire workpiece itself or a local part of the workpiece. According to the determined workpiece part, the corresponding cropping parameters are preset, such as the size, shape, and position of the cropping area. The acquired frame images can be batch processed using image processing tools.

[0095] Before stitching, the stitching order and rules between the cropped multiple frame images need to be determined. If the frame images are extracted in the time order of video recording, then the stitching also needs to be carried out in this time order to maintain the continuity of the surface changes of the workpiece. The stitching rules are usually determined by the shape, texture, defect characteristics and detection requirements of the workpiece surface. If the workpiece surface is flat, the stitching rules may focus on ensuring the precise alignment of adjacent images in the horizontal or vertical direction. For curved workpieces, the stitching rules need to consider the perspective deformation and surface fitting of the image to ensure that the stitched image remains visually coherent and accurate. The consistency and repeatability of the texture can help determine the overlapping area between adjacent images and serve as a reference for stitching. Defect characteristics are also one of the key factors to be considered in the stitching rules. For example, if the defect appears as a small crack or scratch, the stitching rules should avoid excessive smoothing or image fusion in the defect area.

[0096] If necessary, the stitched image can be further processed and adjusted, such as color correction, brightness adjustment, contrast enhancement, etc., to improve image quality and detection effect. After stitching and adjustment, the final workpiece surface expansion image is saved as an image file. Select the appropriate image format to save according to the image resolution and quality parameters. The stitched workpiece surface expansion image is as follows: Figure 3 , Figure 4 As shown in the figure, Figure a shows the original image of a part of the rotating workpiece, and Figure b shows the two-dimensional view of the surface of the part after it is unfolded.

[0097] pass Figure 3 , Figure 4 It can be seen that the images generated by a series of algorithms have uniform lighting and clear defect contrast, providing reliable guarantee for defect detection and identification.

[0098] Step S4: Perform image processing and analysis on the expanded image to detect defects on the workpiece surface. The color image can be converted into a grayscale image to simplify the calculation and highlight the defect features in the image. The image segmentation algorithm is used to separate the defects from the background by setting a suitable threshold for segmentation based on the grayscale difference between the defects and the background in the image. Based on the extracted features, a classifier (such as a support vector machine, a neural network, etc.) is used to detect and identify defects in the image. According to the characteristics and types of the defects, they can be classified into different defect categories, such as cracks, scratches, pits, etc. Therefore, by performing image processing and analysis on the expanded image, the defects on the workpiece surface are automatically detected. This replaces the traditional manual detection method, improves detection efficiency and accuracy, and reduces detection costs.

[0099] In this embodiment, it also includes: using parallel processing technology of multi-core CPU and CUDA architecture to perform real-time image stitching of local video data of multiple industrial cameras to form video data of the surface of the rotating workpiece, and decoding and stitching the video data to form an expanded image of the workpiece surface.

[0100] Specifically, multi-core CPUs can handle multiple tasks at the same time and improve processing speed through parallel computing. CUDA (Compute Unified Device Architecture) is a parallel computing architecture launched by NVIDIA, which greatly improves computing performance by utilizing the processing power of graphics processing units (GPUs). By combining the parallel processing technology of multi-core CPUs and CUDA architecture, the video fusion and image processing algorithms of multiple groups of workstations can be highly parallelized, and parallel computing can significantly speed up the image stitching speed to achieve real-time image stitching of local video data from multiple industrial cameras, providing an efficient and accurate solution for surface quality inspection of rotating parts.

[0101] In summary, a multi-camera fusion workpiece surface defect detection method provided in the embodiment of the present application is applied to the defect detection of a rotating body workpiece with a curved shape, high reflectivity to light, and made of metal material. The method includes: using several industrial cameras to collect video of the surface of the rotating body workpiece to generate video data of the surface of the rotating body workpiece; decoding the video data to obtain each frame image in the video; cutting and splicing the acquired frame images to form an expanded image of the entire or partial workpiece surface; performing image processing and analysis on the expanded image to detect defects on the workpiece surface. The present application combines the flexible configuration of multiple industrial cameras to collect video of the surface of the rotating body workpiece, which can ensure that image information of the workpiece surface is obtained from multiple angles and positions. The high-speed video acquisition technology is used to achieve optical flexibility that ordinary line scan cameras cannot match. This configuration is not only suitable for various complex industrial scenes, but also has a large depth of field characteristic, which can ensure that clear images can be obtained at different working distances. It has strong versatility for the detection of the outer wall of metal rotating body workpieces with curved surfaces and high reflectivity. By configuring the number of workstations and adjusting the focal length, surface defect detection of rotating workpieces of any length can be easily achieved.

[0102] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0103] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0104] The electronic device includes: one or more processors; and a memory storing computer program instructions, and when the computer program instructions are executed, the processor executes the multi-camera fusion workpiece surface defect detection method provided by any one or more of the above embodiments. Figure 5 An exemplary structural diagram of the electronic device is disclosed. Figure 5 As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0105] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0106] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0107] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0108] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the multi-camera fusion workpiece surface defect detection method provided in any one or more of the above embodiments is implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0109] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0110] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] It should be noted that more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Computer readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0113] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using a dedicated integrated circuit (ASIC, Application-Specific Integrated Circuit), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0115] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL, Digital Subscriber Line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive, solid state disk, SSD), etc.

[0116] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A multi-camera fusion workpiece surface defect detection method, characterized in that: Applied to defect detection of rotating workpieces with curved surface shape, high light reflectivity and made of metal material, the method includes: Using a plurality of industrial cameras to collect video of the surface of the rotating workpiece to generate video data of the surface of the rotating workpiece; Decoding the video data to obtain each frame of image in the video; The acquired frame images are cropped and spliced ​​to form an expanded image of the entire or partial workpiece surface; Image processing and analysis are performed on the expanded image to detect defects on the surface of the workpiece.

2. The multi-camera fusion workpiece surface defect detection method according to claim 1, characterized in that: The step of using a plurality of industrial cameras to collect video of the surface of the rotating workpiece to generate video data of the surface of the rotating workpiece includes: A plurality of industrial cameras are arranged above or around the rotating workpiece, each of the industrial cameras being configured to take a video of a specific local position of the rotating workpiece; Starting the industrial cameras to simultaneously capture video of each specific local position of the rotating workpiece, ensuring that the shooting range of each camera covers an independent area on the surface of the rotating workpiece with overlapping edges; The local video data captured by each of the industrial cameras are spliced ​​into video data of the surface of the rotating workpiece.

3. The multi-camera fusion workpiece surface defect detection method according to claim 2, characterized in that: The surface of the rotating body workpiece includes a plurality of rotating surfaces with different diameters, and the diameters of the rotating surfaces corresponding to the specific local positions photographed by the industrial cameras are different; The step of arranging a plurality of industrial cameras above or around the rotating workpiece, each of the industrial cameras being configured to take a video of a specific local position of the rotating workpiece, comprises: Determining the diameters of different rotational surfaces on the surface of the rotating workpiece; A maximum processing frame rate is set for the rotating surface with the largest diameter, and the processing frame rate of the corresponding industrial camera is set according to the diameter size with the maximum processing frame rate as a reference standard, wherein a rotating surface with a larger diameter corresponds to a higher processing frame rate, and a rotating surface with a smaller diameter corresponds to a lower processing frame rate; The set different processing frame rates are configured in the parameters of the corresponding industrial cameras, so that each of the industrial cameras can shoot the corresponding rotating surface on the rotating body workpiece according to the set processing frame rate.

4. The multi-camera fusion workpiece surface defect detection method according to claim 1, characterized in that: The step of decoding the video data to obtain each frame of image in the video includes: Receiving a video code stream after encoding and compressing the video data, wherein the video code stream includes video parameter information and compressed image data, wherein the parameter information includes: video parameters and encoding format; Parsing the video code stream based on the encoding format and the video parameters, and acquiring image data in the video frame by frame; Traversing each frame of image data, and determining the frame image to be extracted according to the set frame extraction parameters, wherein the frame extraction parameters include: a start frame, an end frame, a frame extraction interval, and a frame number; Each of the selected frame images is saved separately as an image file.

5. The multi-camera fusion workpiece surface defect detection method according to claim 4 is characterized in that: The step of parsing the video code stream based on the encoding format and the video parameters to obtain image data in the video frame by frame includes: For the image frame data in the video code stream, using the transform coding technology in the video encoding process, performing inverse quantization and inverse transformation processing in the video decoding process, mapping the quantized coefficients back to the original or nearly original transform coefficient values, and converting the transform coefficient values ​​back to the spatial domain to restore to the uncompressed original image data; At the same time, in the video decoding process, the predictive coding technology in the video encoding process is used to perform intra-frame prediction on the correlation between adjacent pixels in the same frame image, and inter-frame prediction on the motion information between adjacent frame images, so as to reconstruct the image data using the inverse process of the predictive coding technology; The video decoding process is performed frame by frame to obtain and reconstruct the image data in the video until the entire video code stream is decoded.

6. The multi-camera fusion workpiece surface defect detection method according to claim 1, characterized in that: The step of cutting and splicing the acquired frame images to form an expanded image of the entire or partial workpiece surface includes: According to the workpiece part where defects need to be detected, the acquired frame image is cropped according to preset cropping parameters; The cropped multiple frame images are spliced ​​according to a predetermined splicing order and rule to form an expanded image of the entire or partial workpiece surface; The expanded image of the workpiece surface after splicing is saved as an image file.

7. The multi-camera fusion workpiece surface defect detection method according to claim 1, characterized in that: Also includes: By using the parallel processing technology of multi-core CPU and CUDA architecture, the local video data of multiple industrial cameras are stitched in real time to form the video data of the surface of the rotating workpiece, and the video data is decoded and stitched to form the expanded image of the workpiece surface.

8. An electronic device, characterized in that: The electronic device comprises: One or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the multi-camera fusion workpiece surface defect detection method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by the processor, the multi-camera fusion workpiece surface defect detection method as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-camera fusion workpiece surface defect detection method as described in any one of claims 1 to 7 is implemented.

Images