Defect position determination method and device based on camera calibration, equipment and medium

Abstract

Provided in the embodiments of the present disclosure are a defect position determination method, device, equipment and medium based on camera calibration, which comprises: calibrating each camera to obtain camera intrinsic parameters and camera extrinsic parameters; wherein the calibration mode of the camera extrinsic parameters is a non-overlapping field of view calibration mode; determining a first defect position in an image of a metallurgical product collected by each camera based on a captured image of the metallurgical product to be detected under a field of view of each camera, obtaining a second defect position of each first defect position in a reference space coordinate system based on a position mapping relationship; obtaining an initial point position of an initial point of the metallurgical product in the reference space coordinate system based on a front-back frame difference when the metallurgical product enters the field of view of the camera and the position coordinate conversion relationship, and calculating an offset of the second defect position relative to the initial point position. The embodiments of the present disclosure realize comprehensive, efficient and accurate defect positioning on the surface of the metallurgical product.

Description

Technical Field

[0001] This disclosure relates to the field of industrial inspection technology, and in particular to a method, apparatus, equipment and medium for determining defect location based on camera calibration. Background Technology

[0002] Due to the current limitations of smelting technology, surface defects often occur in round continuous casting billets during production. If these defects are not detected in time, a large number of scrap products will be produced, which will seriously affect the economic benefits of enterprises. Therefore, it is necessary to locate surface defects in round continuous casting billets. However, considering that manual observation is inefficient and costly, this approach is not feasible.

[0003] While machine vision can enable automated monitoring of the surface quality of circular continuously cast billets, efficiently, accurately, automatically, and cost-effectively detecting the location of defects on the surface of circular continuously cast billets, the defect analysis typically performed using monocular or binocular vision is not suitable for larger circular continuously cast billets, making it difficult to perform comprehensive defect detection on the surface of circular continuously cast billets. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this disclosure is to provide a method, apparatus, device and medium for determining defect location based on camera calibration, so as to solve the problems in the related art.

[0005] The first aspect of this disclosure provides a defect location determination method based on camera calibration, applied to an image acquisition system including multiple cameras, for determining the location of surface defects in metallurgical products; the method includes: calibrating each camera to obtain camera intrinsic parameters and camera extrinsic parameters; wherein the camera extrinsic parameters are calibrated using a non-overlapping field-of-view calibration method, and one camera serves as a reference camera, and the camera extrinsic parameters of each other camera correspond to the coordinate transformation from their respective camera coordinate systems to the reference space coordinate system of the reference camera; the camera intrinsic parameters and camera extrinsic parameters form a positional mapping relationship from the pixel coordinate systems of each other camera to the reference space coordinate system; based on the acquired images of the metallurgical product to be detected moving into the field of view of each camera, determining a first defect location in the images of the metallurgical product acquired by each camera, and obtaining a second defect location in the reference space coordinate system for each of the first defect locations based on the positional mapping relationship; based on the difference between the preceding and following frames when the metallurgical product enters the field of view and the positional coordinate transformation relationship, obtaining the initial point location of the metallurgical product in the reference space coordinate system, and calculating the offset of the second defect location relative to the initial point location.

[0006] In an embodiment of the first aspect, the plurality of cameras are spaced apart along a linear direction.

[0007] In an embodiment of the first aspect, the plurality of cameras do not share a common field of view with each other.

[0008] In an embodiment of the first aspect, calibrating each camera to obtain camera intrinsic and extrinsic parameters includes: moving a calibration plate displaying a feature pattern sequentially along the linear direction from a reference camera through the field of view of each camera; when the feature pattern appears in the field of view of a camera, calibrating the camera intrinsic parameters of the camera based on the image captured by the camera for the feature pattern; when the feature pattern appears in the field of view of a preceding camera and a following camera, calibrating the camera extrinsic parameters of the following camera using a non-overlapping field of view calibration method based on the first image and the second image obtained by the preceding and following cameras respectively capturing the feature pattern.

[0009] In the first aspect of the embodiment, the step of calibrating the camera extrinsic parameters of the subsequent camera using a non-overlapping field-of-view calibration method based on the first image and the second image obtained from the feature patterns acquired by the preceding camera and the subsequent camera respectively includes: determining a first extrinsic parameter matrix for the camera coordinate system coordinate transformation from the second feature pattern portion to the first feature pattern portion based on the physical positional relationship between the first feature pattern portion in the first image and the second feature pattern portion in the second image; and calculating the spatial coordinate system of the first feature pattern portion based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, as well as the camera intrinsic parameters of the preceding camera. The second extrinsic parameter matrix for coordinate transformation from the camera coordinate system of the preceding camera; the third extrinsic parameter matrix for coordinate transformation from the camera coordinate system of the following camera to the spatial coordinate system of the second feature pattern based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, and the camera intrinsic parameters of the following camera; the local extrinsic parameters for coordinate transformation from the camera coordinate system of the following camera to the camera coordinate system of the preceding camera based on the first extrinsic parameter matrix, the second extrinsic parameter matrix, and the third extrinsic parameter matrix; and the camera extrinsic parameters of the following camera based on the local extrinsic parameters of each camera between the following camera and the reference camera.

[0010] In an embodiment of the first aspect, the defect location determination method based on camera calibration further includes: establishing an optimization objective function based on the error of reprojecting feature points using the calculated local extrinsic parameters; and optimizing the camera extrinsic parameter matrix among all cameras using the Levenberg-Marquardt algorithm based on the optimization objective function.

[0011] In an embodiment of the first aspect, obtaining the initial point position of the metallurgical product in the reference spatial coordinate system based on the difference between the preceding and following frames when the metallurgical product enters the camera's field of view and the position coordinate transformation relationship includes: obtaining a corresponding difference image based on the difference in grayscale values ​​between the preceding and following frames when the metallurgical product enters the camera's field of view; performing thresholding and connectivity analysis on the difference image to obtain the pixel coordinates of the starting point of the metallurgical product in the camera's field of view of a target camera; and transforming the pixel coordinates to the reference spatial coordinate system based on the calibrated camera intrinsic and extrinsic parameters of the target camera to obtain the initial point position.

[0012] A second aspect of this disclosure provides a defect location determination device based on camera calibration, applied to an image acquisition system including multiple cameras, for determining the location of surface defects in metallurgical products. The device includes a calibration module for calibrating each camera to obtain camera intrinsic and extrinsic parameters. The extrinsic parameters are calibrated using a non-overlapping field-of-view calibration method, with one camera serving as a reference camera. The extrinsic parameters of each of the other cameras correspond to a coordinate transformation from their respective camera coordinate systems to the reference space coordinate system of the reference camera. The camera intrinsic and extrinsic parameters form the pixel coordinate systems of the other cameras to the reference space coordinate system. The system includes a position mapping relationship between coordinate systems; a defect location acquisition module, used to determine the first defect location in the image of the metallurgical product acquired by each camera based on the image acquired when the metallurgical product moves into the field of view of each camera, and to obtain the second defect location of each first defect location in the reference space coordinate system based on the position mapping relationship; and a defect localization module, used to obtain the initial point location of the metallurgical product in the reference space coordinate system based on the difference between the preceding and following frames when the metallurgical product enters the field of view of the camera and the position coordinate transformation relationship, and to calculate the offset of the second defect location relative to the initial point location.

[0013] A third aspect of this disclosure provides a computer device, comprising: a memory and a processor; the memory storing program instructions, and the processor executing the program instructions to perform a defect location determination method based on camera calibration as described in any one aspect.

[0014] The fourth aspect of this disclosure provides a computer-readable storage medium storing program instructions that are executed to perform a camera-calibrated defect location determination method as described in any one of the first aspects.

[0015] As described above, this disclosure provides a method, apparatus, device, and medium for determining defect locations based on camera calibration. The method includes: calibrating each camera to obtain camera intrinsic and extrinsic parameters; wherein the extrinsic parameters are calibrated using a non-overlapping field-of-view calibration method; determining a first defect location in the images of the metallurgical product captured by each camera based on the images acquired when the metallurgical product moves into the field of view of each camera; obtaining a second defect location in a reference spatial coordinate system for each of the first defect locations based on the position mapping relationship; obtaining an initial point location of the metallurgical product in the reference spatial coordinate system based on the frame difference when the metallurgical product enters the camera's field of view and the position coordinate transformation relationship; and calculating the offset of the second defect location relative to the initial point location. This disclosure enables comprehensive, efficient, and accurate defect localization on the surface of metallurgical products. Attached Figure Description

[0016] Figure 1 A schematic diagram of the structure of the image acquisition system in an embodiment of this disclosure is shown.

[0017] Figure 2 A flowchart illustrating the defect location determination method based on camera calibration in an embodiment of this disclosure is shown.

[0018] Figure 3 This illustration shows a schematic diagram of the principle of calibrating the camera intrinsic and extrinsic parameters of each camera using a calibration plate in one embodiment of the present disclosure.

[0019] Figure 4 A flowchart illustrating a non-overlapping field-of-view calibration method according to an embodiment of this disclosure is shown.

[0020] Figure 5 A schematic diagram showing the field-of-view segmentation feature patterns of camera 2 and camera 1 in one embodiment of this disclosure.

[0021] Figure 6 A schematic diagram of a metallurgical product is shown in one embodiment of this disclosure.

[0022] Figure 7 A schematic diagram illustrating defect detection results in one embodiment of this disclosure.

[0023] Figure 8 A schematic diagram of the module for determining the defect location based on camera calibration in an embodiment of this disclosure is shown.

[0024] Figure 9 A schematic diagram of the structure of a computer device in an embodiment of this disclosure is shown. Detailed Implementation

[0025] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the information disclosed herein. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this disclosure can be modified or changed according to different viewpoints and application modules without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0026] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily implement it. This disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

[0027] In this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic represented in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in any one or a group of embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples represented in this disclosure, as well as the features of those different embodiments or examples.

[0028] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this disclosure, "a set" means two or more, unless otherwise explicitly specified.

[0029] For the purpose of clarity, devices unrelated to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0030] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0031] While the terms first, second, etc., are used in some examples herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, module, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0032] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this disclosure. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in this specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0033] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the message of the present disclosure, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0034] In the metallurgical industry, metallurgical products such as circular continuously cast billets require surface defect detection. These defects include, but are not limited to, surface cracks, burns, joints, pinholes, slag inclusions, scratches, slag grooves, ellipticity discrepancies, shrinkage cavities, and inconsistencies in steel composition. However, current detection methods still rely on manual labor, which suffers from low efficiency, subjectivity leading to low accuracy, and a high risk of omissions. While common machine vision methods such as monocular and binocular vision can improve detection efficiency, metallurgical products like circular continuously cast billets are typically large, making comprehensive surface inspection difficult.

[0035] In view of the above problems, this disclosure provides a camera-calibrated defect localization method to effectively solve these problems. The camera-calibrated defect localization method is applied to an image acquisition system including multiple cameras to determine the location of surface defects in metallurgical products.

[0036] like Figure 1 The diagram shown illustrates the structure of the image acquisition system in an embodiment of this disclosure.

[0037] The image acquisition system comprises multiple cameras for acquiring images of the surface of the metallurgical product through different fields of view, thereby obtaining an image set covering various parts of the surface of the metallurgical product. Exemplarily, the cameras can be arranged at linear intervals, the shape of which is adapted to the shape of the metallurgical product. For example, the metallurgical product in the illustration could be, for instance, a circular continuously cast billet with a straight length direction; therefore, the cameras in the illustration are arranged along a straight line consistent with its length direction.

[0038] In the camera imaging model, each camera has a field of view and camera intrinsic parameters, and can establish its own camera coordinate system centered on itself. Each camera's intrinsic parameters include an intrinsic parameter matrix and distortion parameters, which can be used for coordinate transformations between its captured image coordinate system, pixel coordinate system, and its camera coordinate system. Coordinate transformations between camera coordinate systems use camera extrinsic parameters, including translation and rotation parameter matrices. Both the camera extrinsic and intrinsic parameters can be determined through calibration.

[0039] In some embodiments, the multiple cameras may not share a common field of view but may have adjacent fields of view, that is, the captured images may not overlap but may be stitched together. During calibration, the extrinsic parameter matrix of camera coordinate system transformation between cameras is obtained by non-overlapping field of view calibration method.

[0040] In some embodiments, Figure 1 The circular continuously cast billet in the image is rolling, for example, moving along the depth direction shown in the image. It can roll into or out of the field of view of each camera.

[0041] Understandably, the number of cameras corresponds to the size of the metallurgical product, with the goal of capturing images that can essentially cover the entire surface of the product. The figure exemplarily shows 14 cameras, namely cameras 1 to 14, each with its own camera coordinate system. Specifically, since the camera coordinate systems of cameras 1 to 14 are independent of each other, the measured coordinates are not uniform. Therefore, it is necessary to solve the coordinate transformation relationship between the various camera coordinate systems and unify the obtained coordinates to a reference space coordinate system. Thus, one camera can be selected as the reference camera, and its camera coordinate system is the reference space coordinate system. For example, camera 1 is used as the reference camera, and its camera coordinate system is considered the reference space coordinate system. Therefore, the camera extrinsic parameters of the other cameras, namely cameras 2 to 14, are the camera extrinsic parameters resulting from the coordinate transformation from their respective camera coordinate systems to camera 1's camera coordinate system. Of course, in other embodiments, the reference space coordinate system can also be the world coordinate system, and is not limited to this.

[0042] The camera intrinsic and extrinsic parameters form the positional mapping relationship between the pixel coordinate systems of other cameras and the reference spatial coordinate system.

[0043] like Figure 2 The diagram shows a flowchart illustrating the defect location determination method based on camera calibration in an embodiment of this disclosure.

[0044] exist Figure 2 The defect location determination method based on camera calibration includes:

[0045] Step S201: Calibrate each camera to obtain camera intrinsic and camera extrinsic parameters.

[0046] In some embodiments, a calibration plate displaying a feature pattern can be moved sequentially along the linear direction from the reference camera through the field of view of each camera. When the feature pattern appears in the field of view of a camera, the camera intrinsic parameters of that camera are calibrated based on the image captured by that camera for the feature pattern. When the feature pattern appears in the field of view of both the preceding and following cameras, the camera extrinsic parameters of the following camera are calibrated using a non-overlapping field-of-view calibration method, based on the first and second images obtained by the preceding and following cameras respectively capturing the feature pattern.

[0047] Please refer to them together. Figure 1 and Figure 3 Please provide a detailed explanation. Figure 3 This illustration shows a schematic diagram of the principle of calibrating the camera intrinsic and extrinsic parameters of each camera using a calibration plate in one embodiment of the present disclosure.

[0048] The calibration board is moved from camera 1 to camera 14. Single-target calibration is performed as the board enters the field of view of each camera (camera 1-14) to obtain the intrinsic parameters and distortion coefficients of each camera. During the movement, extrinsic parameter calibration is performed on cameras 1-14 to obtain the extrinsic parameters of cameras 2-14 to the reference coordinate system (i.e., the camera coordinate system of camera 1). Since cameras 1-14 may not share a common field of view, the extrinsic parameters need to be calibrated using a non-overlapping field-of-view method. For example, when the calibration board enters the field of view of camera 1 (the preceding camera), single-target calibration of the intrinsic parameters of camera 1 begins, and sufficient images are acquired to complete the single-target calibration. Then, the calibration board moves towards the field of view of camera 2 (the following camera) until part of the calibration board appears in the field of view of camera 1 and another part appears in the field of view of camera 2. At this point, non-overlapping field-of-view calibration of the extrinsic parameters of cameras 1 and 2 begins. At this point, the single-target calibration of cameras 1 and 2 is complete, and the non-overlapping field-of-view calibration of camera 2 to the reference spatial coordinate system is complete. Similarly, the same operation is performed on cameras 3 to 14, thus completing the single-target calibration of cameras 1 to 14 and the non-overlapping field-of-view calibration of camera 2 to 14 to the reference spatial coordinate system.

[0049] In some embodiments, the calibration plate feature pattern includes multiple feature points, each of which may have unique distinguishing feature information. For example, the feature pattern includes an array of feature graphics, which may have the same shape and size but different content to serve as distinguishing feature information. The feature graphics may be, for example, centrally symmetrical shapes (such as squares or circles), where the grayscale of the center point can be clearly distinguished from the surrounding points to be used as a feature point. The feature pattern acquired by the camera contains multiple feature points, the reference space coordinates of which are known in a reference space coordinate system (e.g., the world coordinate system). The pixel coordinates of each feature point and its corresponding reference space coordinates form a coordinate pair. In the camera imaging model, the positional mapping relationship between the pixel coordinates and the reference space coordinates in each coordinate pair is determined by the camera intrinsic and extrinsic parameters, thereby constructing multiple equations based on multiple coordinate pairs to solve for the intrinsic parameter matrix.

[0050] Understandably, during the image acquisition process for camera extrinsic calibration, sufficient feature points of the acquired feature patterns must be obtained to ensure the formation of a feature point surface. This guarantees the accuracy of calculating camera extrinsic parameters when extracting local feature points in the non-overlapping field-of-view calibration. Similarly, acquiring sufficient images ensures the successful resolution of camera extrinsic parameter calibration results in the non-overlapping field-of-view mode. Furthermore, during camera extrinsic parameter calibration, the camera intrinsic parameters of camera 2 have already been obtained through calibration and can be used for camera extrinsic parameter calibration in the non-overlapping field-of-view mode.

[0051] To further explain the calculation principle of camera intrinsics, the coordinate pair consisting of the pixel coordinates and the corresponding reference space coordinates of each feature point is input into the camera model to obtain:

[0052] (1)

[0053] in, For the first The pixel coordinates (or more precise sub-pixel coordinates) of each feature point. The world coordinates of the feature graph are... The number of feature graphics, As a scaling factor, and The rotation matrix and translation vector of the target from the world coordinate system to the camera coordinate system are called the camera extrinsic parameters, or simply extrinsic parameters. Let be the intrinsic parameter matrix of the camera, denoted as ,for example Figure 3 Cameras 1 to 14 1~ 14.

[0054] (2)

[0055] in, The coordinates of the principal point in the image. and These are the image pixel coordinate systems. shaft and The scale factor of the axis, for shaft and The non-perpendicularity factor of the axis.

[0056] Considering the distortion inherent in camera lenses, radial and tangential distortions are taken into account. Points are set... Ideal imaging point The coordinates in the image coordinate system are , point actual imaging point The coordinates in the image coordinate system are Image distortion is corrected based on the following distortion model.

[0057] (3)

[0058] in, It is the radial distortion coefficient of the lens. This represents the tangential distortion coefficient of the lens. .

[0059] Based on the imaging model and distortion model, calibration board images in different poses are processed to obtain the corresponding point pairs. Substituting these points into the model, the camera intrinsic parameters of a single camera, i.e., the intrinsic parameter matrix, can be initially solved. and distortion coefficient .

[0060] When a feature pattern appears in the field of view of two adjacent cameras, the camera extrinsic parameters for coordinate transformation between the camera coordinate systems of the adjacent cameras can be calculated based on the physical positional relationship between multiple feature points in the feature pattern captured by each camera, the pixel coordinates of the feature points, and the calculated camera intrinsic parameters of the cameras.

[0061] like Figure 4 The diagram shown illustrates a flowchart of a non-overlapping field-of-view calibration method according to an embodiment of this disclosure.

[0062] The first image and the second image, obtained by the front camera and the rear camera respectively capturing the feature pattern, wherein the first image contains a first feature pattern portion of the feature pattern, and the second image contains a second feature pattern portion of the feature pattern. For example... Figure 5 As shown, the front camera and the rear camera are configured such that their fields of view cover the left and right portions of the feature pattern, respectively.

[0063] Figure 4 The process includes:

[0064] Step S401: Based on the physical positional relationship between the first feature pattern portion in the first image and the second feature pattern portion in the second image, determine the first extrinsic parameter matrix for the camera coordinate system coordinate transformation from the second feature pattern portion to the first feature pattern portion.

[0065] The following explanation will take the calibration of camera extrinsic parameters from camera coordinate system 2 to camera coordinate system 1 as an example.

[0066] Specifically, such as Figure 3 As shown, there is no common field of view between camera 1 and camera 2, making it impossible to directly obtain the camera extrinsic matrix from the camera 2 coordinate system to the camera 1 coordinate system through non-overlapping field of view calibration. Therefore, when the calibration plate is placed between camera 1 and camera 2, its feature pattern is simultaneously divided into two interconnected regions by the fields of view of both camera 1 and camera 2, denoted as […]. , , arrive Camera extrinsic matrix That is, the first extrinsic parameter matrix, using , The physical connection relationships between them are calibrated, for example A certain feature point 1 is rotated x degrees and translated y millimeters in space. A certain feature point can be obtained based on actual measurements of the calibration plate.

[0067] Step S402: Based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, as well as the camera intrinsic parameters of the previous camera, calculate the second extrinsic parameter matrix for the coordinate transformation from the spatial coordinate system of the first feature pattern portion to the camera coordinate system of the previous camera.

[0068] Step S403: Based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, as well as the camera intrinsic parameters of the subsequent camera, calculate the third extrinsic parameter matrix for the coordinate transformation from the camera coordinate system of the subsequent camera to the spatial coordinate system of the second feature pattern portion.

[0069] Continuing from the example in step S401, camera 1 calibrates a calibration plate located within its field of view. Partial single-target targeting was performed to obtain Camera extrinsic matrix to camera 1 coordinate system That is, the second extrinsic parameter matrix; at the same time, camera 2 sets up a calibration board located within its field of view. Partial single-target targeting was performed to obtain Camera extrinsic matrix to camera 2 coordinate system That is, the third extrinsic parameter matrix.

[0070] Specifically, camera 1 and camera 2 simultaneously capture an image of the calibration board, which is divided by the fields of view of camera 1 and camera 2 into the aforementioned sections. and part, Partially shared One feature point, Partially shared The feature points can be accurately extracted using the method in step S111. and Pixel coordinates of all concentric circle center feature points , , respectively represented as , Its corresponding world coordinates , , respectively represented as , The correspondence is as follows:

[0071] (9)

[0072] (10)

[0073] Let homography matrix , By solving the linear equations of the corresponding point pairs in (9) and (10), two homography matrices can be obtained. Then, by performing singular value decomposition on them respectively, the calibration plate can be obtained. Camera extrinsic matrix of part to camera 1 and calibration plate Camera extrinsic matrix of part to camera 2 .

[0074] Step S404: Calculate the local extrinsic parameters for the coordinate transformation from the camera coordinate system of the latter camera to the camera coordinate system of the former camera based on the first extrinsic parameter matrix, the second extrinsic parameter matrix, and the third extrinsic parameter matrix.

[0075] set up Upper feature point exist Homogeneous coordinates in the coordinate system are ,exist Homogeneous coordinates in the coordinate system are The homogeneous coordinates in the camera 2 coordinate system are: The homogeneous coordinates in the camera 1 coordinate system are The transformation relationships between the above homogeneous coordinates are as follows:

[0076] (11)

[0077] The connection relationship between the two parts of the calibration plate is obtained. , The local extrinsic parameters from camera 2 coordinate system to camera 1 coordinate system can be obtained from the single-target calibration process described above. Thus, the camera extrinsic parameter matrix calibration between the camera 1 coordinate system and the camera 2 coordinate system is completed. Similarly, by moving the calibration plate, the camera extrinsic parameter matrices from the camera 2 coordinate system to the camera 14 coordinate system and back to the camera 1 coordinate system can be obtained sequentially using the non-overlapping field-of-view camera extrinsic parameter calibration method described above.

[0078] Step S405: Calculate the camera extrinsic parameters of the next camera based on the local extrinsic parameters of each camera between the next camera and the reference camera.

[0079] For example, see formula (16), the product of each local extrinsic parameter from the next camera to the reference camera constitutes the camera extrinsic parameter for the coordinate transformation of the camera coordinate system from the next camera to the reference camera.

[0080] In some embodiments, since the orthogonal constraint property of the rotation matrix is ​​not considered in the linear solution of the camera extrinsic parameters, the solution result is easily affected by noise. Therefore, the camera extrinsic parameters obtained from the linear solution are used as initial values, and the optimization objective function is established using the reprojection error. Then, the Levenberg-Marquardt algorithm is used to optimize the camera extrinsic parameter matrix among all cameras. From equations (9), (10), and (11), the following expression can be obtained:

[0081] (12)

[0082] (13)

[0083] in , The pixel coordinates of the feature points obtained by reprojecting them from the feature points in the encoding coordinate system through the camera intrinsic and extrinsic parameter matrices can be used to obtain the optimization objective function as follows:

[0084] (14)

[0085] The optimized camera extrinsic matrix can be obtained by applying the Levenberg-Marquardt optimization algorithm. Similarly, cameras 3 to 14 use the same method to optimize the rotation matrix R and translation vector T obtained from the calibration, thereby obtaining the final calibration result.

[0086] Following step S201, step S202 is executed: based on the images acquired by the metallurgical product to be detected moving into the field of view of each camera, the first defect position in the image of the metallurgical product acquired by each camera is determined, and the second defect position of each first defect position in the reference space coordinate system is obtained based on the position mapping relationship.

[0087] Specifically, with Figure 1 For example, 14 cameras acquire first images of portions of the metallurgical product within their field of view in real time, and perform defect detection on each first image, i.e., defect detection on each part of the metallurgical product. The defect detection can be achieved using a deep learning model (such as CNN, R-CNN, or an improved model) pre-trained with defect image data, or through other image analysis algorithms. If the input image contains a corresponding defect, the model can accurately predict the pixel coordinates of the defect location, i.e., the location of the first defect. Let the defect location be at the camera... Within the field of view, after processing by a neural network, the pixel coordinates of the corresponding defect location can be obtained, denoted as... In the camera In the coordinate system, the camera coordinates corresponding to its pixel coordinates are: The camera obtained by combining the aforementioned methods for single-target positioning intrinsic parameter matrix The coordinate transformation formula from pixel coordinates to camera coordinates can be obtained as follows:

[0088] (15)

[0089] At this point, the location of the defect at the camera can be determined. Camera coordinates in coordinate system The camera extrinsic parameter matrix obtained in step S12 from the camera 2 coordinate system to the camera 14 coordinate system to the camera 1 coordinate system (i.e., the reference space coordinate system) can be used to transform the coordinates of the defect position in all camera coordinate systems to the coordinates of the second defect position in the reference space coordinate system. The formula is as follows:

[0090] (16)

[0091] Step S203: Based on the difference between the preceding and following frames when the metallurgical product enters the camera's field of view and the position coordinate transformation relationship, obtain the initial point position of the metallurgical product in the reference space coordinate system, and calculate the offset of the second defect position relative to the initial point position.

[0092] In some embodiments, when the metallurgical product enters the camera's field of view, a large proportion of pixels with high grayscale corresponding to the surface of the metallurgical product appear in the image, causing a significant difference in grayscale values ​​between the preceding and following frames captured by the camera. This allows for the acquisition of a corresponding difference image, i.e., an image obtained by subtracting the corresponding pixel values ​​at the same pixel position in the preceding and following frames. Further, the difference image undergoes thresholding (e.g., binary thresholding) and connectivity analysis (extracting the starting point of interest) to obtain the pixel coordinates of the starting point of the metallurgical product within the camera's field of view of a target camera. Based on the camera intrinsic and extrinsic parameters of the target camera calibrated in the previous steps, the pixel coordinates are then transformed to a reference space coordinate system to obtain the initial point position.

[0093] For example, let the camera coordinates of the starting point in the reference space coordinate system be... Suppose that the metallurgical product has several defect locations, and the camera coordinates of the k-th defect location in the reference space coordinate system are: The offset of the defect location relative to the starting point can be obtained according to the coordinate distance calculation formula. As shown below:

[0094] (17)

[0095] The offset represents the actual physical length from the defect location to the starting point.

[0096] To illustrate the above principles more clearly, several examples from actual experiments will be used below.

[0097] Implementation Case 1:

[0098] A batch of metallurgical products are awaiting testing, such as Figure 6 As shown, one of the metallurgical products is 9m long and has a defect located within the field of view of camera 2. A defect location determination device is located at a preset height on the metallurgical product. This device is used to automatically locate the defect. The circular continuous casting billet is placed on an inclined track and is manually pried downwards by a worker to slowly roll into the field of view of the automatic defect location determination device. This device contains 14 cameras with non-overlapping fields of view, simultaneously acquiring images of the metallurgical product within its field of view. The defect detection model then predicts and outputs the coordinates of the center pixel of the first defect within the field of view of camera 2. ,like Figure 7 As shown.

[0099] The intrinsic parameter matrices of cameras 1 through 14 were obtained through single-target calibration, where the intrinsic parameters of camera 1 are... Camera 2's internal parameters The camera coordinates of the defect location in the camera 2 coordinate system can be calculated using equation (15). The camera extrinsic matrix from camera 2 coordinate system to camera 1 coordinate system can be obtained through camera extrinsic parameter calibration using a non-overlapping field of view method. The camera coordinates of the defect location in the reference space coordinate system can be calculated using equation (16). The pixel coordinates of the starting point of the metallurgical product obtained by step S203 are... It can also convert and obtain the camera coordinates of the metallurgical product's starting point in the reference space coordinate system. This allows us to calculate the offset of the defect location relative to the starting point. This ultimately completes the automatic location of the defect.

[0100] like Figure 8 The diagram illustrates a defect location determination device based on camera calibration in an embodiment of this disclosure. Since the functional modules or sub-modules of the defect location determination device based on camera calibration in this embodiment are based on the same principles as the corresponding steps or sub-steps of the defect location determination method based on camera calibration in the above exemplary method embodiments, the specific implementation in this embodiment can refer to the previous content, and the same technical content will not be repeated.

[0101] The defect location determination device 800 includes:

[0102] The calibration module 801 is used to calibrate each camera to obtain camera intrinsic parameters and camera extrinsic parameters; wherein the calibration method of the camera extrinsic parameters is a non-overlapping field of view calibration method, and one of the cameras is used as a reference camera, and the camera extrinsic parameters of each other camera correspond to the coordinate transformation from their respective camera coordinate system to the reference space coordinate system of the reference camera; the camera intrinsic parameters and camera extrinsic parameters form a position mapping relationship from the pixel coordinate system of each other camera to the reference space coordinate system.

[0103] The defect location acquisition module 802 is used to determine the first defect location in the image of the metallurgical product acquired by each camera based on the image acquired by the metallurgical product moving into the field of view of each camera, and to obtain the second defect location of each first defect location in the reference space coordinate system based on the location mapping relationship.

[0104] The defect location module 803 is used to obtain the initial point position of the metallurgical product in the reference space coordinate system based on the difference between the preceding and following frames when the metallurgical product enters the camera's field of view and the position coordinate transformation relationship, and to calculate the offset of the second defect position relative to the initial point position.

[0105] It should be noted that, in Figure 8 The various functional modules in the embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a program instruction product. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0106] and, Figure 8 The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments shown above are merely illustrative. For example, the module division is only a logical functional division, and in actual implementation, there may be other division methods. For example, a group of modules or modules may be combined or dynamically integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces, and the indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0107] in addition, Figure 8The functional modules and sub-modules in the embodiments can be dynamically integrated within a single processing unit, or each module can exist physically independently, or two or more modules can be dynamically integrated within a single unit. These dynamic units can be implemented in hardware or as software functional modules. If these dynamic units are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0108] It should be specifically noted that the flowchart representations of the embodiments described above in this disclosure can be understood as representing modules, segments, or portions of code comprising one or more sets of executable instructions for implementing specific logical functions or processes. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0109] For example, Figure 2 The order of the steps in the method embodiments may vary in specific scenarios and is not limited to the above representation.

[0110] like Figure 9 The diagram shows a schematic diagram of the circuit structure of a computer device according to an embodiment of the present disclosure.

[0111] The computer device 900 includes a bus 901, a processor 1102, and a memory 903. The processor 1102 and the memory 903 can communicate with each other via the bus 901. The memory 903 can store program instructions (such as system or application software). The processor 1102 implements the steps of the defect location determination method in this embodiment by running the program instructions in the memory 903, thereby realizing the function of the defect location determination device in the previous embodiment.

[0112] Bus 901 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, although only one thick line is used in the diagram, this does not indicate that there is only one bus or one type of bus.

[0113] In some embodiments, the processor 1102 may be implemented as a central processing unit (CPU), a microprocessor unit (MCU), a system-on-chip (System-on-Chip), or a field-programmable array (FPGA). The memory 903 may include volatile memory for temporary data storage during program execution, such as random access memory (RAM).

[0114] The memory 903 may also include non-volatile memory for data storage, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state disk (SSD).

[0115] In some embodiments, the computer device 900 may further include a communicator 904. The communicator 904 is used for communication with external devices. In specific examples, the communicator 904 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 904 may include one or more of, such as a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, Nearfield Communication (NFC) technology, Infrared (IR) technology, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc.

[0116] This disclosure may also provide a computer-readable storage medium storing program instructions, which, when executed, perform the aforementioned method embodiments (e.g., ...). Figure 2 The process steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is downloaded via a network, originally stored in a remote recording medium or a non-transitory machine-readable medium, and will be stored in a local recording medium. Thus, the method represented herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as ASIC or FPGA).

[0117] In summary, this disclosure provides a method, apparatus, device, and medium for determining defect locations based on camera calibration. The method includes: acquiring first images of various parts of a metallurgical product to be inspected, wherein the combination of the first images covers the entire surface of the metallurgical product; inputting each of the first images to at least one defect detection model to obtain and record defect detection results; wherein the defect detection results include defect type information of each part and global location information of the corresponding defect on the metallurgical product; the defect detection model is a trained deep neural network model; and displaying the defect type information in at least the defect detection results in real time through a human-computer interaction interface. The solution in this disclosure uses the combination of first images to completely detect the surface of the metallurgical product and selects a defect detection model based on a deep neural network model to accurately identify defects.

[0118] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A method for determining defect location based on camera calibration, characterized in that, An image acquisition system comprising multiple cameras is used to determine the location of surface defects in metallurgical products; the method includes: Each camera is calibrated to obtain camera intrinsic and camera extrinsic parameters; wherein the camera extrinsic parameters are calibrated using a non-overlapping field-of-view calibration method, and one of the cameras is used as a reference camera, and the camera extrinsic parameters of each other camera correspond to the coordinate transformation from their respective camera coordinate system to the reference space coordinate system of the reference camera; the camera intrinsic and camera extrinsic parameters form a positional mapping relationship from the pixel coordinate system of each other camera to the reference space coordinate system; Based on the images captured by the metallurgical product to be detected moving into the field of view of each camera, the first defect position in the image of the metallurgical product captured by each camera is determined, and the second defect position of each first defect position in the reference space coordinate system is obtained based on the position mapping relationship. Based on the differences between the preceding and following frames when the metallurgical product enters the camera's field of view and the coordinate transformation relationship, the initial point position of the metallurgical product in the reference spatial coordinate system is obtained. Specific steps include: obtaining a corresponding difference image based on the grayscale value differences between the preceding and following frames when the metallurgical product enters the camera's field of view; performing thresholding and connectivity analysis on the difference image to obtain the pixel coordinates of the metallurgical product's starting point in the camera's field of view of a target camera; and transforming the pixel coordinates to the reference spatial coordinate system based on the calibrated camera intrinsic and extrinsic parameters of the target camera to obtain the initial point position. And calculate the offset of the second defect position relative to the initial point position.

2. The defect location determination method based on camera calibration according to claim 1, characterized in that, The multiple cameras are spaced apart along a linear direction.

3. The defect location determination method based on camera calibration according to claim 1, characterized in that, The multiple cameras do not share a common field of view.

4. The defect location determination method based on camera calibration according to claim 2, characterized in that, The calibration of each camera to obtain camera intrinsic and extrinsic parameters includes: The calibration plate, displaying a characteristic pattern, is moved sequentially along the linear direction from the reference camera through the field of view of each camera. When the feature pattern appears in the field of view of a camera, the camera intrinsic parameters of the camera are calibrated based on the image captured by the camera of the feature pattern. When the feature pattern appears in the field of view of the front camera and the rear camera, the camera extrinsic parameters of the rear camera are calibrated by non-overlapping field of view calibration based on the first image and the second image obtained by the front camera and the rear camera respectively acquiring the feature pattern.

5. The defect location determination method based on camera calibration according to claim 4, characterized in that, The step of calibrating the camera extrinsic parameters of the latter camera using a non-overlapping field-of-view calibration method, based on the first and second images obtained from the feature patterns captured by the preceding and following cameras respectively, includes: Based on the physical positional relationship between the first feature pattern portion in the first image and the second feature pattern portion in the second image, determine the first extrinsic parameter matrix for the camera coordinate system coordinate transformation from the second feature pattern portion to the first feature pattern portion; Based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, as well as the camera intrinsic parameters of the previous camera, calculate the second extrinsic parameter matrix for coordinate transformation from the spatial coordinate system of the first feature pattern portion to the camera coordinate system of the previous camera. Based on the pixel coordinates of multiple feature points in the first feature pattern portion and their corresponding coordinates in the world coordinate system, as well as the camera intrinsic parameters of the subsequent camera, calculate the third extrinsic parameter matrix for the coordinate transformation from the camera coordinate system of the subsequent camera to the spatial coordinate system of the second feature pattern portion. Based on the first extrinsic matrix, the second extrinsic matrix, and the third extrinsic matrix, the local extrinsic parameters for the coordinate transformation from the camera coordinate system of the latter camera to the camera coordinate system of the former camera are calculated. The camera extrinsic parameters of the latter camera are calculated based on the local extrinsic parameters of each camera between the latter camera and the reference camera.

6. The defect location determination method based on camera calibration according to claim 5, characterized in that, Also includes: An optimization objective function is established based on the error of reprojecting feature points using the calculated local extrinsic parameters. Based on the objective function, the camera extrinsic matrix among all cameras is optimized using the Levenberg-Marquardt algorithm.

7. A defect location determination device based on camera calibration, characterized in that, An image acquisition system comprising multiple cameras is used to determine the location of surface defects in metallurgical products; the device includes: A calibration module is used to calibrate each camera to obtain camera intrinsic parameters and camera extrinsic parameters; wherein the camera extrinsic parameters are calibrated using a non-overlapping field-of-view calibration method, and one camera is used as a reference camera, and the camera extrinsic parameters of each other camera correspond to the coordinate transformation from their respective camera coordinate system to the reference space coordinate system of the reference camera; the camera intrinsic parameters and camera extrinsic parameters form a positional mapping relationship from the pixel coordinate system of each other camera to the reference space coordinate system; The defect location acquisition module is used to determine the first defect location in the image of the metallurgical product acquired by each camera based on the image captured by the metallurgical product moving into the field of view of each camera, and to obtain the second defect location of each first defect location in the reference space coordinate system based on the location mapping relationship. The defect localization module is used to obtain the initial point position of the metallurgical product in the reference spatial coordinate system based on the difference between the preceding and following frames when the metallurgical product enters the camera's field of view and the position coordinate transformation relationship. Specific steps include: obtaining a corresponding difference image based on the difference in grayscale values ​​between the preceding and following frames when the metallurgical product enters the camera's field of view; performing thresholding and connectivity analysis on the difference image to obtain the pixel coordinates of the starting point of the metallurgical product in the camera's field of view of a target camera; and transforming the pixel coordinates to the reference spatial coordinate system based on the calibrated camera intrinsic and extrinsic parameters of the target camera to obtain the initial point position. And calculate the offset of the second defect position relative to the initial point position.

8. A computer device, characterized in that, include: Memory and processor; The memory stores program instructions, and the processor is used to run the program instructions to perform the camera-calibrated defect location determination method as described in any one of 1 to 6.

9. A computer-readable storage medium, characterized in that, The system stores program instructions that are executed to perform a camera-calibrated defect location determination method as described in any one of 1 to 6.

Images