Space-time registration method and system for metallurgical melt jet multi-modal image

Abstract

The invention belongs to the field of metallurgical melt jet flow detection and analysis, and discloses a time-space registration method and system for a metallurgical melt jet flow multi-modal image, so as to realize registration of the metallurgical melt jet flow multi-modal image in a complex environment. The method comprises the following steps: synchronously acquiring a metallurgical melt jet multi-modal image by adopting a binocular vision detection device; extracting a metallurgical melt jet flow contour in the multi-modal image, and respectively filtering texture details of the contour to obtain a smooth contour; analyzing the smooth contour of the multi-modal image to extract a contour feature point set, and performing rough matching on the multi-modal contour feature point set; constructing a feature point descriptor for the contour feature point set of the multi-modal image, and performing fine matching on the multi-modal contour feature point set based on the feature point descriptor; and mismatching among the contour feature point sets of the multi-modal image is eliminated, and final registration of the contour feature point sets is realized.

Description

Technical Field

[0001] The invention belongs to the field of metallurgical molten jet detection and analysis, and in particular relates to multi-modal visual image registration of metallurgical molten jet. Background Art

[0002] Metallurgical melt is a high-temperature molten fluid formed after metal ore undergoes a complex smelting process. Metallurgical melt jet is a high-speed, high-temperature molten metal flow formed by liquid metal passing through a jet nozzle under high temperature and high pressure. The multimodal visual image of the metallurgical melt jet is important information reflecting the production status of the metallurgical reaction vessel, and provides key feedback information for the closed-loop refinement of the metal smelting process.

[0003] Most of the existing detection methods for jet temperature, flow velocity, and stream morphology use information from a single modality for detection, and cannot achieve synchronous identification and acquisition of multi-source information. Image registration is an important step in the collaborative detection of multimodal images, and infrared images and visible light images are two commonly used multimodal images for visual detection. They can integrate the complementary and significant feature information of the two modal images, and can perform more comprehensive detection of the target scene. However, most of the existing multimodal image registration methods can only be used for simple and general scenes such as weak environmental interference, small viewing angle differences, and significant contour features. They cannot be directly applied to the spatiotemporal registration of multimodal images of metallurgical melt jets, which have the characteristics of large modal differences, high registration difficulty, and few image details. In summary, the existing metallurgical melt jet detection technology has the problem of being unable to synchronously obtain multimodal information, and the existing multimodal image registration technology also has the problem of difficulty in achieving multimodal image registration in complex scenes. Summary of the invention

[0004] The technical problem to be solved by the present invention is that it is difficult to synchronously obtain complete and clear multimodal metallurgical melt jet images in a complex industrial environment, and the dynamic changes of the jet stream are large and the texture features are complex, which makes it difficult to accurately align the multimodal images. The shortcomings and defects mentioned in the above background technology are overcome, and a method and system for spatiotemporal alignment of multimodal images of metallurgical melt jets are provided.

[0005] In order to solve the above technical problems, the present invention proposes a spatiotemporal registration method and system for multimodal images of metallurgical melt jet.

[0006] The present application provides a method for spatiotemporal registration of multimodal images of a metallurgical melt jet, comprising:

[0007] S1: using a binocular vision detection device to synchronously acquire a multimodal image of a metallurgical melt jet, wherein the image includes a first modal image and a second modal image;

[0008] S2: extracting the metallurgical melt jet contours in the first modal image and the second modal image, and filtering out the texture details of the contours to obtain smooth contours;

[0009] S3: analyzing the smooth contours of the first modality image and the second modality image to extract a contour feature point set, and performing rough matching on the contour feature point sets of the first modality image and the second modality image;

[0010] S4: determining a search range, searching the contour feature point set of the first modality image based on the search range to obtain a precise feature point set, constructing a feature point descriptor for the precise feature point set and the contour feature point set of the second modality image, and performing precise matching between the precise feature point set of the first modality image and the contour feature point set of the second modality image based on the feature point descriptor;

[0011] S5: Eliminate mismatches between the precise feature point set of the first modality image and the contour feature point set of the second modality image, and achieve final alignment of the contour feature point set.

[0012] Preferably, the S2 includes:

[0013] S21: extracting the metallurgical melt jet contour in the first modal image and the second modal image using an active contour method;

[0014] S22: Use a contour smoothing model based on variational regularization to filter out the texture details of the contour. The calculation of the smoothed contour is shown in formula (1):

[0015]

[0016] Among them, k, p are the coefficients of the optimization target, and ||·|| are Frobenius and L1 norms respectively, l is the optimized profile, m is the original profile, ω is the weight matrix, is a first-order difference filter, and · represents element-by-element multiplication.

[0017] Preferably, the step of extracting a set of contour feature points by analyzing the smooth contours of the first modality image and the second modality image in S3 includes:

[0018] The local curvature of the smooth contour of the first modality image and the second modality image is calculated to extract the contour feature point set. The calculation of the contour feature point set is shown in formula (2):

[0019]

[0020] Among them, l i,x , l i,y , l i,k represents the x, y coordinates and corresponding curvature of the i-th index of contour l, Respectively represent the first-order and second-order derivatives of the contour l, c, l cur They respectively represent the selection threshold of the contour curvature and the selected contour feature point set.

[0021] Preferably, the step S3 of roughly matching the contour feature point sets of the first modality image and the second modality image comprises:

[0022] Extract the corresponding contour feature point set l cur Finally, the nearest contour point is found as the corresponding point through the greedy algorithm to determine the rough matching relationship between the contour feature point sets of the first modality image and the second modality image. The calculation formula is:

[0023]

[0024] Among them, i and j represent the metallurgical melt jet contour feature point set l in the first modal image and the second modal image respectively. cur,col , l cur,inf The index value, l i,a represents the x or y coordinate of the ith index in the metallurgical melt jet profile l. The rough matching relationship between the profile feature point sets is obtained by formula (3):

[0025]

[0026] Preferably, the step of determining a search range in S4, searching for a contour feature point set of the first modality image based on the search range to obtain an accurate feature point set includes:

[0027] S41: Specify the search length as [kn,k+n] and the search width as Combine the search length with the search width to get the search range. The calculation of is shown in formula (5):

[0028]

[0029] Where k represents the index value of the kth contour point, n represents the search length near the kth contour point, and d i is the Euclidean distance between feature points, i and j represent the metallurgical melt jet contour feature point set l in the first modal image and the second modal image respectively. cur,col , l cur,inf The index value of

[0030] The contour feature point set of the first modality image is searched according to the search range. At the corresponding contour point Search around;

[0031] S42: Calculate the spatial gradient change of the pixel points within the search range, and use the points that reach the specified threshold as candidate points of the contour feature points. Determine the precise feature points based on the positions of the candidate points, and obtain all the precise feature points to form a precise feature point set.

[0032] Preferably, the step of constructing a feature point descriptor for the precise feature point set and the contour feature point set of the second modality image in S4, and performing precise matching on the precise feature point set of the first modality image and the contour feature point set of the second modality image, comprises:

[0033] S43: is the contour feature point set of the second modality image and the corresponding accurate feature point set of the first modality image Constructing descriptors by contour angles and Disc cur,col , use the mean filter algorithm to calculate the feature points The left and right contour vectors The calculation of is shown in formula (6):

[0034]

[0035] Among them, i and j represent the index values ​​of the contour feature point set respectively;

[0036] Normalize the contour vectors on both sides and merge them into the main direction vector calculate Compare the values ​​in the x and y dimensions to get the contour angle of the contour feature point The calculation of is shown in formula (7):

[0037]

[0038] S44: Constructing descriptors Then, perform similarity measurement on formula (7) to determine Each feature point in The corresponding precise feature points in The registration relationship between the contour feature points is determined as follows:

[0039]

[0040] Preferably, the step of eliminating mismatches between the precise feature point set of the first modality image and the contour feature point set of the second modality image in step S5 to achieve final alignment of the contour feature point sets includes:

[0041] S51: Using the nearest neighbor iteration method, construct the contour feature point set registration loss function based on adaptive weights as shown in formula (9):

[0042]

[0043] Among them, i and j represent the index of the contour feature point set. represents the contour feature point set of the second modality image, express The exact feature point set of each feature point in the first modality image The point set consisting of the corresponding precise feature points in , nInf represents the number of contour feature points after matching, A represents the rigid registration matrix, R * ,t * are the rotation matrix and translation matrix, ω represents the weight matrix, and the size of ω is nInf*1;

[0044] S52: Solve the rotation matrix R and translation matrix t in equation (9) respectively. The rotation matrix R * The calculation of is shown in formula (10):

[0045] R * =UV T (10)

[0046] Among them, U, V are orthogonal matrices decomposed by SVD;

[0047] Translation matrix t * The calculation of is shown in formula (11):

[0048] t * =gR * p (11)

[0049] Among them, g and p are the center points of the contour,

[0050] S53: Combining the similarity between feature points and the registration distance between feature points to achieve adaptive allocation of the weight matrix, each weight coefficient {ω i |i∈[1,n inf The calculation of ]} is shown in formula (12):

[0051]

[0052] Among them, n inf Indicates the number of matched contour feature points, s i,j Represents the minimum distance between two feature descriptors, d i,j represents the distance between the feature points of two images, It represents the distance between the feature points of the two images after normalization, iter represents the number of iterations, and k represents the index value of the kth contour point. Indicates the kth iter-1 A rotation matrix, Indicates the kth iter-1 A translation matrix.

[0053] As a general inventive concept, the present invention also provides a spatiotemporal registration system for metallurgical melt jet multimodal images, comprising: a binocular vision detection module, a hardware signal triggering module and a calculation processing module connected in sequence;

[0054] The binocular vision detection module is used to obtain a synchronized multi-modal image of the metallurgical melt jet based on a trigger signal, wherein the image includes a first modal image and a second modal image;

[0055] The hardware signal trigger module is used to send the trigger signal to the binocular vision detection module based on the instruction from the calculation processing module, and transmit the metallurgical melt jet multimodal image acquired by the binocular vision detection module to the calculation processing module;

[0056] The computing and processing module is used to send instructions to the hardware signal triggering module and perform the following steps:

[0057] Extract the metallurgical melt jet contours in the first modal image and the second modal image, filter out the texture details of the contours respectively to obtain a smooth contour; analyze the smooth contours of the first modal image and the second modal image to extract a contour feature point set, and roughly match the contour feature point sets of the first modal image and the second modal image; determine a search range, search the contour feature point set of the first modal image based on the search range to obtain a precise feature point set, construct a feature point descriptor for the precise feature point set and the contour feature point set of the second modal image, and precisely match the precise feature point set of the first modal image and the contour feature point set of the second modal image based on the feature point descriptor; eliminate mismatches between the precise feature point set of the first modal image and the contour feature point set of the second modal image, and achieve final alignment of the contour feature point sets.

[0058] Preferably, the hardware signal trigger module comprises: a voltage step signal trigger unit, a voltage stabilizing unit and a remote network communication unit; the voltage stabilizing unit is connected to the voltage step signal trigger unit and the remote network communication unit, and the remote network communication unit is connected to the voltage step signal trigger unit;

[0059] The voltage step signal trigger unit is used to generate an instantaneous voltage step trigger signal;

[0060] The voltage stabilizing unit is used to realize voltage step-down conversion among multiple voltages and to supply power to the voltage step signal triggering unit and the remote network communication unit.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention uses a hardware signal trigger circuit to achieve synchronous acquisition of multimodal images, extracts contours in images and smoothes the contours, further extracts contour feature point sets based on contour curvature analysis, and uses a greedy algorithm to preliminarily achieve coarse matching between feature point sets. On this basis, a feature point descriptor is constructed based on local feature similarity to achieve precise matching of contour point sets, and finally an adaptive weighted contour feature point set registration method is used to achieve spatiotemporal registration of multimodal images of metallurgical molten jets. The above method can synchronously acquire multimodal images with high dynamic changes, and overcomes the influence of noise interference and viewing angle differences, can accurately extract contour features of metallurgical molten jets, and achieve multimodal image registration of metallurgical molten jets under complex and harsh interference.

[0063] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a flow chart of a method for spatiotemporal registration of multimodal images of a metallurgical melt jet according to a preferred embodiment of the present invention;

[0066] Figure 2 is a schematic diagram of searching a specified range around a contour feature point of a first modality image according to a preferred embodiment of the present invention;

[0067] Figure 3 is a multimodal image registration effect diagram of a preferred embodiment of the present invention, Figure 3 (a) is the contour registration effect diagram. Figure 3 (b) is the image registration effect diagram. DETAILED DESCRIPTION

[0068] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively and meticulously below in conjunction with the accompanying drawings and preferred embodiments of the present invention, but the protection scope of the present invention is not limited to the following specific embodiments.

[0069] Unless otherwise defined, all professional terms used below have the same meanings as those generally understood by those skilled in the art. The professional terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention.

[0070] Unless otherwise specified, various raw materials, reagents, instruments and equipment used in the present invention can be purchased from the market or prepared by existing methods.

[0071] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0072] like Figure 1 As shown, this embodiment provides a spatiotemporal registration method for metallurgical melt jet multimodal images, comprising:

[0073] S1: using a binocular vision detection device to synchronously acquire a multimodal image of a metallurgical melt jet, wherein the image includes a visible light image and an infrared light image;

[0074] S2: extract the metallurgical melt jet contours in the visible light image and the infrared light image, and filter out the texture details of the contours to obtain smooth contours;

[0075] S3: Analyze the smooth contours of the visible light image and the infrared light image to extract a contour feature point set, and perform rough matching on the contour feature point set of the visible light image and the infrared light image;

[0076] S4: determining a search range, searching the contour feature point set of the first modality image based on the search range to obtain a precise feature point set, constructing a feature point descriptor for the precise feature point set and the contour feature point set of the second modality image, and performing precise matching between the precise feature point set of the first modality image and the contour feature point set of the second modality image based on the feature point descriptor;

[0077] S5: Eliminate mismatches between the precise feature point set of the visible light image and the contour feature point set of the infrared light image, and achieve final alignment of the contour feature point set.

[0078] It is worth noting that in actual industrial environments, the morphology of metallurgical melt jets is highly dynamic and has complex texture features. At the same time, the dust and water mist at the metallurgical melt smelting site are extremely disturbing, which seriously restricts the multimodal visual perception of the metallurgical melt jet process and makes it difficult to accurately align the metallurgical melt jet image. The above-mentioned spatiotemporal registration method of multimodal images of metallurgical melt jets takes into account the polymorphism of environmental interference at the metallurgical melt smelting site, and proposes a multimodal image spatiotemporal registration method that uses signal-triggered synchronous image acquisition and matching based on contour feature point sets. It synchronously acquires highly dynamically changing multimodal images, analyzes the differences in brightness and gradient of multimodal images, overcomes the influence of noise interference and perspective differences, extracts the contour features of the metallurgical melt jet, and realizes the multimodal image registration of metallurgical melt jets under complex and harsh interference.

[0079] As a preferred implementation of this embodiment, S2 includes:

[0080] S21: Extracting the metallurgical melt jet contour in visible light image and infrared light image using active contour method;

[0081] S22: Use a contour smoothing model based on variational regularization to filter out the texture details of the contour. The calculation of the smoothed contour is shown in formula (1):

[0082]

[0083] Among them, k, p are the coefficients of the optimization target, and ||·|| are Frobenius and L1 norms respectively, l is the optimized profile, m is the original profile, ω is the weight matrix, is a first-order difference filter, and · represents element-by-element multiplication.

[0084] The metallurgical melt jet contour extracted by the active contour method in the present invention has many fine texture details. These fine texture details will affect the extraction of contour feature points, causing the extraction of feature points to fall into small local extreme values, thereby affecting the extraction of contour feature points and reducing the registration accuracy of feature points. Therefore, it is necessary to smooth the contour, filter out the texture details of the contour, and improve the accuracy of multimodal image registration.

[0085] As a preferred implementation of this embodiment, the step of extracting a contour feature point set by analyzing the smooth contours of the visible light image and the infrared light image in S3 includes:

[0086] The local curvature of the smooth contour of the visible light image and the infrared light image is calculated to extract the contour feature point set. The calculation of the contour feature point set is shown in formula (2):

[0087]

[0088] Among them, l i,x , l i,y , l i,k represents the x, y coordinates and corresponding curvature of the i-th index of contour l, Respectively represent the first-order and second-order derivatives of the contour l, c, l cur They respectively represent the selection threshold of the contour curvature and the selected contour feature point set.

[0089] As a preferred implementation of this embodiment, the step S3 of roughly matching the contour feature point sets of the visible light image and the infrared light image includes:

[0090] Extract the corresponding contour feature point set l cur Finally, the nearest contour point is found as the corresponding point through the greedy algorithm to determine the rough matching relationship between the contour feature point sets of the visible light image and the infrared light image. The calculation formula is:

[0091]

[0092] Among them, i and j represent the metallurgical melt jet contour feature point set l in the visible light image and infrared light image respectively. cur ,col , l cur,inf The index value, l i,a represents the x or y coordinate of the ith index in the metallurgical melt jet profile l. The rough matching relationship between the profile feature point sets is obtained by formula (3):

[0093]

[0094] As a preferred implementation of this embodiment, the step of determining the search range in S4, searching the contour feature point set of the first modality image based on the search range to obtain the precise feature point set includes:

[0095] S41: Contour feature point set for visible light image At the corresponding contour point Search the specified range around, specify the search length as [kn,k+n], and the search width as Combine the search length with the search width to get the search range. The calculation of is shown in formula (5):

[0096]

[0097] Where k represents the index value of the kth contour point, n represents the search length near the kth contour point, and d i is the Euclidean distance between feature points, i and j represent the feature point set l of the metallurgical melt jet contour in the visible light image and infrared light image respectively. cur,col , lcur,inf The index value of

[0098] S42: Calculate the spatial gradient change of the pixel points within the search range, take the points that reach the specified threshold as candidate points of the contour feature points, further determine the precise feature points through Taylor expansion of the candidate point positions, and obtain all the precise feature points to form a precise feature point set

[0099] The present invention directly searches for an accurate feature point set on the spatial gradient of a visible light image, plays a role of secondary matching, and thus constructs a more accurate feature point registration relationship.

[0100] As a preferred implementation of this embodiment, the step S4 constructs a feature point descriptor for the precise feature point set and the contour feature point set of the infrared image, and performs precise matching on the precise feature point set of the visible light image and the contour feature point set of the infrared image, including:

[0101] S43: is the contour feature point set of the infrared image And the corresponding accurate feature point set of the visible light image Constructing descriptors by contour angles and Disc cur,col , use the mean filter algorithm to calculate the feature points The left and right contour vectors The calculation of is shown in formula (6):

[0102]

[0103] Among them, i and j represent the index values ​​of the contour feature point set respectively;

[0104] Normalize the contour vectors on both sides and merge them into the main direction vector calculate Compare the values ​​in the x and y dimensions to get the contour angle of the contour feature point The calculation of is shown in formula (7):

[0105]

[0106] The multimodal images obtained by the present invention have a small scale space variation range and insignificant scale differences. Therefore, considering the computational efficiency and algorithm performance, the present invention performs segmented normalization on the gradient amplitude to reduce the influence of imaging spectrum differences. Specifically, the first 20% of the strongest gradient amplitudes are normalized to 1, 20% to 40%, 40% to 60%, and 60% to 80% are normalized to 0.75, 0.5, and 0.25, and the last 20% of the gradient amplitudes are discarded.

[0107] S44: Constructing descriptors Then, perform similarity measurement on formula (7) to determine Each feature point in The corresponding precise feature points in The registration relationship between the contour feature points is determined as follows:

[0108]

[0109] As a preferred implementation of this embodiment, the step S5 of eliminating mismatches between the precise feature point set of the visible light image and the contour feature point set of the infrared light image to achieve final alignment of the contour feature point set includes:

[0110] S51: Using the nearest neighbor iteration method, construct the contour feature point set registration loss function based on adaptive weights as shown in formula (9):

[0111]

[0112] Among them, i and j represent the index of the contour feature point set. Represents the contour feature point set of the infrared image, express The exact feature point set of each feature point in the visible light image The point set consisting of the corresponding precise feature points in , nInf represents the number of contour feature points after matching, A represents the rigid registration matrix, R * ,t * are the rotation matrix and translation matrix, ω represents the weight matrix, and the size of ω is nInf*1;

[0113] S52: Rotation matrix R in equation (9) * and the translation matrix t * Solve them separately, the rotation matrix R * The calculation of is shown in formula (10):

[0114] R * =UV T (10)

[0115] Among them, U, V are orthogonal matrices decomposed by SVD;

[0116] Translation matrix t * The calculation of is shown in formula (11):

[0117] t * =gR * p (11)

[0118] Among them, g and p are the center points of the contour,

[0119] S53: Combining the similarity between feature points and the registration distance between feature points to achieve adaptive allocation of the weight matrix, each weight coefficient {ω i |i∈[1,n inf The calculation of ]} is shown in formula (12):

[0120]

[0121] Among them, n inf Indicates the number of matched contour feature points, s i,j Represents the minimum distance between two feature descriptors, d i,j represents the distance between the feature points of two images, It represents the distance between the feature points of the two images after normalization, iter represents the number of iterations, and k represents the index value of the kth contour point. Indicates the kth iter-1 A rotation matrix, Indicates the kth iter-1 A translation matrix.

[0122] like Figure 3 As shown, the spatiotemporal registration method of the multimodal image of the metallurgical melt jet in this embodiment can realize the accurate registration of the visible light image and the infrared image of the metallurgical melt jet.

[0123] The present application also provides a spatiotemporal registration system for metallurgical melt jet multimodal images, comprising: a binocular vision detection module, a hardware signal triggering module and a calculation processing module connected in sequence;

[0124] The binocular vision detection module is used to obtain a synchronized multi-modal image of the metallurgical melt jet based on a trigger signal, wherein the image includes a first modal image and a second modal image;

[0125] The hardware signal trigger module is used to send the trigger signal to the binocular vision detection module based on the instruction from the calculation processing module, and transmit the metallurgical melt jet multimodal image acquired by the binocular vision detection module to the calculation processing module;

[0126] The computing and processing module is used to send instructions to the hardware signal triggering module and perform the following steps:

[0127] Extract the metallurgical melt jet contours in the first modal image and the second modal image, filter out the texture details of the contours respectively to obtain a smooth contour; analyze the smooth contours of the first modal image and the second modal image to extract a contour feature point set, and roughly match the contour feature point sets of the first modal image and the second modal image; determine a search range, search the contour feature point set of the first modal image based on the search range to obtain a precise feature point set, construct a feature point descriptor for the precise feature point set and the contour feature point set of the second modal image, and precisely match the precise feature point set of the first modal image and the contour feature point set of the second modal image based on the feature point descriptor; eliminate mismatches between the precise feature point set of the first modal image and the contour feature point set of the second modal image, and achieve final alignment of the contour feature point sets.

[0128] As a preferred implementation of this embodiment, the hardware signal trigger module includes: a voltage step signal trigger unit, a voltage stabilization unit and a remote network communication unit;

[0129] The voltage step signal triggering unit is used to generate an instantaneous voltage step signal;

[0130] The voltage stabilizing unit is connected to the voltage step signal trigger unit and the remote network communication unit, and is used to realize voltage step-down conversion between multiple voltages and supply power to the voltage step signal trigger unit and the remote network communication unit;

[0131] The remote network communication unit is connected to the voltage step signal trigger unit and performs network communication with the outside.

[0132] In this embodiment, due to the highly dynamic characteristics of the jet stream morphology, it is difficult to synchronously acquire multimodal metallurgical melt jet images. Therefore, in order to synchronously acquire multimodal images of high-dynamic jet streams, the present application designs a multimodal image synchronous acquisition circuit based on signal triggering, which is mainly composed of three parts: a voltage step signal trigger, a voltage stabilizing circuit, and a remote network communication. Among them, the voltage step signal trigger is the core of the signal trigger circuit. Through the MOS tube combination control circuit, according to the set acquisition cycle, the instantaneous voltage step signal is accurately generated to achieve precise synchronous control of the signal trigger circuit. The voltage stabilizing circuit module realizes the step-down conversion between multiple voltages and the control of the output current. Network communication realizes synchronous image acquisition and real-time network communication. Through the hardware signal trigger circuit, the synchronous acquisition of multimodal images of high-dynamic jet streams is achieved.

[0133] The spatiotemporal registration system for metallurgical molten jet multimodal images can implement various embodiments of the spatiotemporal registration method for metallurgical molten jet multimodal images described above, and can achieve the same beneficial effects, which will not be described in detail here.

[0134] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for spatiotemporal registration of multimodal images of metallurgical melt jets, characterized in that: The following steps are involved: S1: using a binocular vision detection device to synchronously acquire a multimodal image of a metallurgical melt jet, wherein the image includes a first modal image and a second modal image; S2: extracting the metallurgical melt jet contours in the first modal image and the second modal image, and filtering out the texture details of the contours to obtain smooth contours; S3: analyzing the smooth contours of the first modality image and the second modality image to extract a contour feature point set, and performing rough matching on the contour feature point sets of the first modality image and the second modality image; S4: determining a search range, searching the contour feature point set of the first modality image based on the search range to obtain a precise feature point set, constructing a feature point descriptor for the precise feature point set and the contour feature point set of the second modality image, and performing precise matching between the precise feature point set of the first modality image and the contour feature point set of the second modality image based on the feature point descriptor; S5: Eliminate mismatches between the precise feature point set of the first modality image and the contour feature point set of the second modality image, and achieve final alignment of the contour feature point set.

2. The spatiotemporal registration method of metallurgical melt jet multimodal images according to claim 1, characterized in that: The S2 includes: S21: extracting the metallurgical melt jet contour in the first modal image and the second modal image using an active contour method; S22: Use a contour smoothing model based on variational regularization to filter out the texture details of the contour. The calculation of the smoothed contour is shown in formula (1): Among them, k, p are the coefficients of the optimization target, and ||·|| are Frobenius and L1 norms respectively, l is the optimized profile, m is the original profile, ω is the weight matrix, is a first-order difference filter, and · represents element-by-element multiplication.

3. The spatiotemporal registration method of metallurgical melt jet multimodal images according to claim 1, characterized in that: S3, analyzing the smooth contours of the first modality image and the second modality image to extract a contour feature point set, comprises: The local curvature of the smooth contour of the first modality image and the second modality image is calculated to extract the contour feature point set. The calculation of the contour feature point set is shown in formula (2): Among them, l i,x , l i,y , l i,k represents the x, y coordinates and corresponding curvature of the i-th index of contour l, Respectively represent the first-order and second-order derivatives of the contour l, c, l cur They respectively represent the selection threshold of the contour curvature and the selected contour feature point set.

4. The spatiotemporal registration method of metallurgical melt jet multimodal images according to claim 1, characterized in that: S3, roughly matching the contour feature point sets of the first modality image and the second modality image, includes: Extract the corresponding contour feature point set l cur Finally, the nearest contour point is found as the corresponding point through the greedy algorithm to determine the rough matching relationship between the contour feature point sets of the first modality image and the second modality image. The calculation formula is: Among them, i and j represent the metallurgical melt jet contour feature point set l in the first modal image and the second modal image respectively. cur ,col , l cur,inf The index value, l i,a represents the x or y coordinate of the ith index in the metallurgical melt jet profile l. The rough matching relationship between the profile feature point sets is obtained by formula (3):

5. The method for spatiotemporal registration of multimodal images of metallurgical melt jet according to claim 1, characterized in that: S4 determines the search range, searches for the contour feature point set of the first modality image based on the search range, and obtains an accurate feature point set, including: S41: Specify the search length as [kn,k+n] and the search width as Combine the search length with the search width to get the search range. The calculation of is shown in formula (5): Where k represents the index value of the kth contour point, n represents the search length near the kth contour point, and d i is the Euclidean distance between feature points, i and j represent the metallurgical melt jet contour feature point set l in the first modal image and the second modal image respectively. cur,col , l cur,inf The index value of The contour feature point set of the first modality image is searched according to the search range. At the corresponding contour point Search around; S42: Calculate the spatial gradient change of the pixel points within the search range, take the points that reach the specified threshold as candidate points of the contour feature points, determine the precise feature points according to the positions of the candidate points, and obtain all the precise feature points to form a precise feature point set 6. The registration method according to claim 1, characterized in that: S4 constructs a feature point descriptor for the precise feature point set and the contour feature point set of the second modality image, and performs precise matching on the precise feature point set of the first modality image and the contour feature point set of the second modality image, including: S43: is the contour feature point set of the second modality image and the corresponding accurate feature point set of the first modality image Constructing descriptors by contour angles and Disc cur,col , use the mean filter algorithm to calculate the feature points The left and right contour vectors The calculation of is shown in formula (6): Among them, i and j represent the index values ​​of the contour feature point set respectively; Normalize the contour vectors on both sides and merge them into the main direction vector calculate Compare the values ​​in the x and y dimensions to get the contour angle of the contour feature point The calculation of is shown in formula (7): S44: Constructing descriptors Then, perform similarity measurement on formula (7) to determine Each feature point in The corresponding precise feature points in The registration relationship between the contour feature points is determined as follows:

7. The spatiotemporal registration method of metallurgical melt jet multimodal images according to claim 1, characterized in that: S5, eliminating mismatches between the precise feature point set of the first modality image and the contour feature point set of the second modality image to achieve final alignment of the contour feature point set, includes: S51: Using the nearest neighbor iteration method, construct the contour feature point set registration loss function based on adaptive weights as shown in formula (9): Among them, i and j represent the index of the contour feature point set. represents the contour feature point set of the second modality image, express The exact feature point set of each feature point in the first modality image The point set consisting of the corresponding precise feature points in , nInf represents the number of contour feature points after matching, A represents the rigid registration matrix, R * ,t * are the rotation matrix and translation matrix, ω represents the weight matrix, and the size of ω is nInf*1; S52: Rotation matrix R in equation (9) * The translation matrix t* is solved separately, and the rotation matrix R * The calculation of is shown in formula (10): R * =UV T (10) Among them, U, V are orthogonal matrices decomposed by SVD; Translation matrix t * The calculation of is shown in formula (11): t * =g-R * p (11) Among them, g and p are the center points of the contour, S53: Combining the similarity between feature points and the registration distance between feature points to achieve adaptive allocation of the weight matrix, each weight coefficient {ω i |i∈[1,n inf The calculation of ]} is shown in formula (12): Among them, n inf Indicates the number of matched contour feature points, s i,j Represents the minimum distance between two feature descriptors, d i,j represents the distance between the feature points of two images, It represents the distance between the feature points of the two images after normalization, iter represents the number of iterations, and k represents the index value of the kth contour point. Indicates the kth iter-1 A rotation matrix, Indicates the kth iter-1 A translation matrix.

8. A spatiotemporal registration system for multimodal images of metallurgical melt jets, used to implement the multimodal image registration method according to any one of claims 1 to 7, the system comprising a binocular vision detection module, a hardware signal triggering module and a calculation processing module connected in sequence; The binocular vision detection module is used to obtain a synchronized multi-modal image of the metallurgical melt jet based on a trigger signal, wherein the image includes a first modal image and a second modal image; The hardware signal trigger module is used to send the trigger signal to the binocular vision detection module based on the instruction from the calculation processing module, and transmit the metallurgical melt jet multimodal image acquired by the binocular vision detection module to the calculation processing module; The computing and processing module is used to send instructions to the hardware signal triggering module and perform the following steps: Extract the metallurgical melt jet contours in the first modal image and the second modal image, filter out the texture details of the contours respectively to obtain a smooth contour; analyze the smooth contours of the first modal image and the second modal image to extract a contour feature point set, and roughly match the contour feature point sets of the first modal image and the second modal image; determine a search range, search the contour feature point set of the first modal image based on the search range to obtain a precise feature point set, construct a feature point descriptor for the precise feature point set and the contour feature point set of the second modal image, and precisely match the precise feature point set of the first modal image and the contour feature point set of the second modal image based on the feature point descriptor; eliminate mismatches between the precise feature point set of the first modal image and the contour feature point set of the second modal image, and achieve final alignment of the contour feature point sets.

9. The metallurgical melt jet multimodal image spatiotemporal registration system according to claim 8, characterized in that: The hardware signal trigger module includes: a voltage step signal trigger unit, a voltage stabilizing unit and a remote network communication unit; the voltage stabilizing unit is connected to the voltage step signal trigger unit and the remote network communication unit, and the remote network communication unit is connected to the voltage step signal trigger unit; The voltage step signal trigger unit is used to generate an instantaneous voltage step trigger signal; The voltage stabilizing unit is used to realize voltage step-down conversion among multiple voltages and to supply power to the voltage step signal triggering unit and the remote network communication unit.

Images