A method and system for spatiotemporal registration of metallurgical melt jet multi-modal images

Abstract

The present application belongs to the field of metallurgical melt jet detection and analysis, and discloses a kind of metallurgical melt jet multi-modal image space-time registration method and system, to realize the multi-modal image registration of metallurgical melt jet under complex environment.The method comprises: adopting binocular vision detection device to synchronously acquire metallurgical melt jet multi-modal image;Extract the profile of metallurgical melt jet in multi-modal image, respectively filter out the texture details of profile to obtain smooth profile;The smooth profile of multi-modal image is analyzed to extract the profile feature point set, and the profile feature point set of multi-modal is roughly matched;Feature point descriptor is constructed for the profile feature point set of multi-modal image, and the profile feature point set of multi-modal is finely matched based on feature point descriptor;Excluding the mis-match between the profile feature point set of multi-modal image, to realize the final registration of profile feature point set.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of metallurgical melt jet detection and analysis, and particularly relates to multi-modal visual image registration of metallurgical melt jets. BACKGROUND

[0002] Metallurgical melt is a high-temperature molten fluid formed after complex smelting of metal ores, and metallurgical melt jet is a high-speed, high-temperature molten metal flow formed by liquid metal under the action of high temperature and high pressure through a jet nozzle. Metallurgical melt jet multi-modal visual image is important information reflecting the production state of a metallurgical reaction vessel, and provides key feedback information for closed-loop refinement of the metal smelting process.

[0003] Existing detection methods for jet temperature, flow rate and stream shape mostly use single modal information for detection, and cannot realize synchronous identification and acquisition of multi-source information. Image registration is an important step for multi-modal image collaborative detection, and infrared image and visible light image are two commonly used multi-modal images for visual detection. The complementary and significant feature information of the two modal images can be integrated to detect the target scene more comprehensively. However, existing multi-modal image registration methods mostly can only be applied to simple and general scenes with weak environmental interference, small view angle difference and significant contour features, and cannot be directly applied to metallurgical melt jet multi-modal image space-time registration with large modal difference, high registration difficulty and few image details. In summary, the existing metallurgical melt jet detection technology cannot synchronously acquire multi-modal information, and the existing multi-modal image registration technology also cannot realize multi-modal image registration in complex scenes. SUMMARY

[0004] The technical problem to be solved by the present application is that it is difficult to synchronously acquire complete and clear multi-modal metallurgical melt jet images in a complex industrial environment, and the dynamic change of the jet stream is large, and the texture features are complex, which makes it difficult to accurately register the multi-modal images. The present application overcomes the deficiencies and defects mentioned in the above background technology, and provides a metallurgical melt jet multi-modal image space-time registration method and system.

[0005] To solve the above technical problems, the present application provides a metallurgical melt jet multi-modal image space-time registration method and system.

[0006] The present application provides a metallurgical melt jet multi-modal image space-time registration method, which comprises the following steps:

[0007] S1: synchronously acquiring metallurgical melt jet multi-modal images by using a binocular vision detection device, wherein the images comprise a first modal image and a second modal image;

[0008] S2: extracting the metallurgical melt jet contour in the first modal image and the second modal image, and filtering out the texture details of the contour to obtain a smooth contour, respectively;

[0009] S3: analyzing the smoothed contours of the first modality image and the second modality image to extract a contour feature point set, and performing coarse matching on the contour feature point sets of the first modality image and the second modality image;

[0010] S4: determining a search range, searching the contour feature point set of the first modality image based on the search range to obtain an accurate feature point set, constructing a feature point descriptor for the accurate feature point set and the contour feature point set of the second modality image, and performing fine matching on the accurate feature point set of the first modality image and the contour feature point set of the second modality image based on the feature point descriptor;

[0011] S5: excluding false matches between the accurate feature point set of the first modality image and the contour feature point set of the second modality image, and realizing final registration of the contour feature point set.

[0012] Preferably, the S2 comprises:

[0013] S21: extracting a metallurgical melt jet contour in the first modality image and the second modality image by using an active contour method;

[0014] S22: filtering out texture details of the contour by using a contour smoothing model based on variational regularization, and the calculation of the smoothed contour is shown in formula (1):

[0015]

[0016] wherein k, p are coefficients of the optimization target, and ||·|| are Frobenius and L1 norms, respectively, l is an optimized contour, m is an original contour, ω is a weight matrix, is a first-order difference filter, and · represents element-wise multiplication.

[0017] Preferably, the S3 comprises:

[0018] calculating local curvatures of the smoothed contours of the first modality image and the second modality image to extract the contour feature point set, and the calculation of the contour feature point set is shown in formula (2):

[0019]

[0020] wherein l i,x , l i,y , l i,k represent x, y coordinates and the corresponding curvature of the i-th index of the contour l, and represent first and second derivatives of the contour l, respectively, c, l cur represent a selected threshold value of the contour curvature and the selected contour feature point set, respectively.

[0021] Preferably, S3 performs coarse matching on the contour feature point sets of the first and second modal images, comprising:

[0022] extracting the corresponding contour feature point sets l cur After that, the nearest contour points are found as corresponding points by a greedy algorithm to determine the coarse matching relationship between the contour feature point sets of the first and second modal images, and the calculation formula is:

[0023]

[0024] wherein i and j represent the index values of the metallurgical melt jet contour feature point sets l cur,col and l cur,inf in the first and second modal images respectively, and l i,a represents the x or y coordinate of the i-th index in the metallurgical melt jet contour l. The coarse matching relationship between the contour feature point sets is obtained by formula (3):

[0025]

[0026] Preferably, S4 determines the search range, searches the contour feature point set of the first modal image based on the search range, and obtains the accurate feature point set, comprising:

[0027] S41: specifies the search length as [k-n, k+n] and the search width as The search range is obtained by combining the search length and the search width, The calculation of is shown in formula (5):

[0028]

[0029] wherein k represents the index value of the k-th contour point, n represents the search length near the k-th contour point, d i is the Euclidean distance between the feature points, i and j represent the index values of the metallurgical melt jet contour feature point sets l cur,col and l cur,inf in the first and second modal images respectively;

[0030] According to the search range, the contour feature point set of the first modal image is searched around the corresponding contour point ;

[0031] S42: calculates the spatial gradient variation of the pixel points in the search range, takes the points reaching the specified threshold as the candidate points of the contour feature points, determines the accurate feature points according to the positions of the candidate points, and obtains all the accurate feature points to form the accurate feature point set

[0032] Preferably, S4 is configured to construct a feature point descriptor for the set of precise feature points of the first modality image and the set of contour feature points of the second modality image, and to perform precise matching between the set of precise feature points of the first modality image and the set of contour feature points of the second modality image, comprising:

[0033] S43: constructing a set of contour feature points for the second modality image and a set of precise feature points for the corresponding first modality image constructing a descriptor by a contour angle and Disc cur,col calculating the feature points by using a mean filtering algorithm left and right contour vectors The calculation is shown in formula (6):

[0034]

[0035] wherein i and j represent index values of the set of contour feature points, respectively;

[0036] normalizing the left and right contour vectors to form a main direction vector calculating obtaining a contour angle of the contour feature point by a ratio in x and y dimensions The calculation is shown in formula (7):

[0037]

[0038] S44: constructing a descriptor After that, formula (7) is subjected to similarity measurement to determine a corresponding precise feature point of each feature point in and is recorded as determining a registration relationship between the contour feature points as shown in formula (8):

[0039]

[0040] Preferably, S5 is configured to exclude false matching between the set of precise feature points of the first modality image and the set of contour feature points of the second modality image, and to realize final registration of the set of contour feature points, comprising:

[0041] S51: constructing a contour feature point set registration loss function based on adaptive weight by using a nearest neighbor point iteration method as shown in formula (9):

[0042]

[0043] wherein i and j represent indices of the set of contour feature points, represents the set of contour feature points of the second modality image, representing the accurate feature point set of each feature point in the first modality image the point set composed of the corresponding accurate feature points, nInf represents the number of matched contour feature points, A represents the rigid registration matrix, R * , t * is a rotation matrix and a translation matrix, ω represents a weight matrix, and the size of ω is nInf*1;

[0044] S52: solve the rotation matrix R and the translation matrix t in formula (9) respectively, the rotation matrix R * is calculated as formula (10):

[0045] R * = UV T (10)

[0046] wherein, U, V are orthogonal matrices of SVD decomposition;

[0047] the translation matrix t * is calculated as formula (11):

[0048] t * = g-R * p (11)

[0049] wherein, g and p are the contour center points,

[0050] S53: combine the similarity between feature points and the registration distance between feature points to realize adaptive distribution of the weight matrix, and the calculation of each weight coefficient {ω i |i∈[1,n inf ]} is calculated as formula (12):

[0051]

[0052] wherein, n inf represents the number of matched contour feature points, s i,j represents the minimum value of the distance between the two feature descriptors, d i,j represents the distance between the two image feature points, represents the distance between the two image feature points after normalization, iter represents the iteration number, and k represents the index value of the kth contour point, represents the k iter-1 th rotation matrix, represents the k iter-1 th translation matrix.

[0053] As a general inventive concept, the present application also provides a system for spatiotemporal registration of metallurgical melt jet multi-modal images, comprising: a binocular vision detection module, a hardware signal triggering module and a computing processing module connected in sequence;

[0054] The binocular vision detection module is configured to acquire synchronized metallurgical melt jet multi-modal images based on a triggering signal, the images comprising a first modal image and a second modal image;

[0055] The hardware signal triggering module is configured to send the triggering signal to the binocular vision detection module based on instructions from the computing processing module, and transmit the metallurgical melt jet multi-modal images acquired by the binocular vision detection module to the computing processing module;

[0056] The computing processing module is configured to send instructions to the hardware signal triggering module, and perform the following steps:

[0057] extracting the metallurgical melt jet profile in the first modal image and the second modal image, respectively filtering out the texture details of the profile to obtain a smoothed profile; analyzing the smoothed profile of the first modal image and the second modal image to extract a set of profile feature points, and performing coarse matching on the set of profile feature points of the first modal image and the second modal image; determining a search range, searching the set of profile feature points of the first modal image based on the search range to obtain an accurate set of feature points, constructing a feature point descriptor for the accurate set of feature points and the set of profile feature points of the second modal image, and performing fine matching on the accurate set of feature points of the first modal image and the set of profile feature points of the second modal image based on the feature point descriptor; excluding false matches between the accurate set of feature points of the first modal image and the set of profile feature points of the second modal image, and realizing final registration of the set of profile feature points.

[0058] Preferably, the hardware signal triggering module comprises a voltage step signal triggering unit, a voltage stabilizing unit and a remote network communication unit; the voltage stabilizing unit is connected with the voltage step signal triggering unit and the remote network communication unit, and the remote network communication unit is connected with the voltage step signal triggering unit;

[0059] The voltage step signal triggering unit is configured to generate a transient voltage step triggering signal;

[0060] The voltage stabilizing unit is configured to realize voltage step-down conversion between multiple voltages, and supply power to the voltage step signal triggering unit and the remote network communication unit.

[0061] Compared with the prior art, the present application has the following advantages:

[0062] The present application realizes the synchronous acquisition of multi-modal images by adopting a hardware signal triggering circuit, extracts the outline in the images and performs smoothing processing on the outline, further extracts the feature point set of the outline based on the curvature analysis of the outline, preliminarily realizes the coarse matching between the feature point sets by adopting a greedy algorithm, constructs the feature point descriptor based on the local feature similarity on the basis to realize the fine matching of the outline point set, and finally realizes the space-time registration of the multi-modal images of the metallurgical melt jet by adopting the registration method of the outline feature point set with adaptive weight. The above method can synchronously acquire the multi-modal images with high dynamic change, and overcomes the influence of noise interference and the view angle difference, can accurately extract the outline features of the metallurgical melt jet, and realizes the multi-modal image registration of the metallurgical melt jet under the complex and poor interference.

[0063] The present application will be further described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0065] Figure 1 is a space-time registration method flow chart of the multi-modal images of the metallurgical melt jet of the preferred embodiment of the present application;

[0066] Figure 2 is a schematic diagram of searching the specified range around the outline feature points of the first modal image of the preferred embodiment of the present application;

[0067] Figure 3 is a multi-modal image registration effect diagram of the preferred embodiment of the present application, Figure 3 (a) is an outline registration effect diagram, Figure 3 (b) is an image registration effect diagram. DETAILED DESCRIPTION

[0068] In order to facilitate the understanding of the present application, the following will combine the drawings in the specification and the preferred embodiments to make a more comprehensive and detailed description of the present application, but the protection scope of the present application is not limited to the following specific embodiments.

[0069] Unless otherwise defined, all the professional terms used in the following have the same meaning as generally understood by those skilled in the art. The professional terms used in the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the protection scope of the present application.

[0070] Unless otherwise specifically explained, various materials, reagents, instruments and equipment that are used in the present application are commercially available or are prepared by known methods.

[0071] Unless otherwise defined, technical terms and scientific terms used in the present application shall be understood as having the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms "first", "second", and similar terms used in the present application do not denote any order, quantity, or importance, but are used to distinguish different components. Similarly, the terms "one" or "a" or similar terms do not denote a quantity restriction, but denote the existence of at least one. The terms "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships are also changed accordingly.

[0072] As shown in Figure 1 The embodiment provides a space-time registration method of metallurgical melt jet multi-modal images, which comprises the following steps:

[0073] S1: A binocular vision detection device is used to synchronously acquire metallurgical melt jet multi-modal images, and the images comprise visible light images and infrared light images;

[0074] S2: The profiles of the metallurgical melt jet in the visible light images and the infrared light images are extracted, and the texture details of the profiles are filtered to obtain smooth profiles;

[0075] S3: The smooth profiles of the visible light images and the infrared light images are analyzed to extract profile feature point sets, and the profile feature point sets of the visible light images and the infrared light images are coarsely matched;

[0076] S4: A search range is determined, the profile feature point set of the first modal image is searched based on the search range to obtain an accurate feature point set, a feature point descriptor is constructed for the accurate feature point set and the profile feature point set of the second modal image, and the accurate feature point set of the first modal image and the profile feature point set of the second modal image are precisely matched based on the feature point descriptor;

[0077] S5: The false matches between the accurate feature point set of the visible light image and the profile feature point set of the infrared light image are excluded, and the final registration of the profile feature point sets is realized.

[0078] It is worth mentioning that in the actual industrial environment, the shape of the metallurgical melt jet is highly dynamic and the texture characteristics are complex, and the dust and water mist in the metallurgical melt smelting site interfere greatly, which seriously restricts the multi-modal visual perception of the metallurgical melt jet process, and it is difficult to accurately register the metallurgical melt jet image. The above-mentioned space-time registration method of the multi-modal image of the metallurgical melt jet considers the multi-state interference of the environment of the metallurgical melt smelting site, proposes a space-time registration method of multi-modal images by using signal triggering to synchronously acquire images and based on contour feature point set, synchronously acquires multi-modal images with high dynamic change, analyzes the differences in brightness and gradient of the multi-modal images, overcomes the influence of noise interference and perspective difference, extracts the contour features of the metallurgical melt jet, and realizes the multi-modal image registration of the metallurgical melt jet under complex and severe interference.

[0079] As a preferred embodiment of the present embodiment, S2 comprises:

[0080] S21: extracting the metallurgical melt jet contour in the visible light image and the infrared light image by using the active contour method;

[0081] S22: filtering out the texture details of the contour by using a contour smoothing model based on variational regularization, and the calculation of the smoothed contour is shown in formula (1):

[0082]

[0083] Wherein, k, p are the coefficients of the optimization target, And ||·|| are the Frobenius and L1 norms, respectively, l is the optimized contour, m is the original contour, and ω is the weight matrix, Is a first-order difference filter, and · represents element-wise multiplication.

[0084] The metallurgical melt jet contour extracted by the active contour method has many small texture details, which will affect the extraction of the contour feature points, cause the extraction of the feature points to fall into a small local extremum, and further affect the extraction of the contour feature points, and reduce the registration accuracy of the feature points. Therefore, the contour needs to be smoothed to filter out the texture details of the contour and improve the accuracy of the multi-modal image registration.

[0085] As a preferred embodiment of the present embodiment, S3 comprises:

[0086] The local curvature of the smoothed contour of the visible light image and the infrared light image is calculated, and the contour feature point set is extracted, and the calculation of the contour feature point set is shown in formula (2):

[0087]

[0088] wherein, l i,x , l i,y , l i,k represents the x, y coordinates and the corresponding curvature of the i-th index of the profile l, represents the first and second derivatives of the profile l, respectively, c, l cur represents the selection threshold of the profile curvature and the selected profile feature point set, respectively.

[0089] As a preferred embodiment of the present embodiment, the S3 described rough matching of the profile feature point set of the visible light image and the infrared light image, comprising:

[0090] After extracting the corresponding profile feature point set l cur , the nearest profile point is found by a greedy algorithm as the corresponding point to determine the rough matching relationship between the profile feature point sets of the visible light image and the infrared light image, and the calculation formula is:

[0091]

[0092] wherein, i, j represent the index values of the metallurgical melt jet profile feature point sets l cur ,col , l cur,inf in the visible light image and the infrared light image, respectively, l i,a represents the x or y coordinate of the i-th index of the metallurgical melt jet profile l, and the rough matching relationship between the profile feature point sets is obtained by formula (3):

[0093]

[0094] As a preferred embodiment of the present embodiment, S4 described determining the search range, searching the profile feature point set of the first modal image based on the search range, and obtaining the accurate feature point set, comprising:

[0095] S41: searching the profile feature point set l of the visible light image in the specified range around the corresponding profile point , the specified search length is [k-n, k+n], and the search width is The search range is obtained by combining the search length and the search width, The calculation of is shown in formula (5):

[0096]

[0097] wherein k represents the index value of the k-th profile point, n represents the search length near the k-th profile point, d i is the Euclidean distance between the feature points, i, j represent the index values of the metallurgical melt jet profile feature point sets l cur,col , lcur,inf The index value;

[0098] S42: Calculate the spatial gradient changes of pixels within the search range, and select points that reach a specified threshold as candidate contour feature points. Further determine the precise feature points by Taylor expansion of the candidate point positions, and obtain all precise feature points to form a precise feature point set.

[0099] This invention directly searches for a precise set of feature points on the spatial gradient of a visible light image, which serves as a secondary matching mechanism, thereby constructing a more accurate feature point registration relationship.

[0100] In a preferred embodiment of this example, step S4 involves constructing a feature point descriptor for the precise feature point set and the contour feature point set of the infrared image, and performing a fine match between the precise feature point set of the visible light image and the contour feature point set of the infrared image, including:

[0101] S43: The set of contour feature points for the infrared image. and the corresponding precise feature point set of the visible light image Descriptors are constructed using contour angles. and Disc cur,col The mean filtering algorithm is used to calculate feature points. Contour vectors on the left and right sides The calculation is shown in equation (6):

[0102]

[0103] Where i and j represent the index values ​​of the contour feature point set, respectively;

[0104] The contour vectors on both sides are normalized and merged into a principal direction vector. calculate The contour angle of the contour feature point is obtained by comparing the values ​​in the x and y dimensions. The calculation is shown in equation (7):

[0105]

[0106] The multimodal images obtained by this invention have a small range of scale spatial variation and no obvious scale differences. Therefore, considering computational efficiency and algorithm performance, this invention reduces the impact of imaging spectrum differences by segmenting and normalizing the gradient magnitude. Specifically, the first 20% of the strongest gradient magnitudes are normalized to 1, while the 20%–40%, 40%–60%, and 60%–80% are normalized to 0.75, 0.5, and 0.25, respectively. The last 20% of gradient magnitudes are discarded.

[0107] S44: Constructing Descriptors Then, similarity measure is performed on the formula (7) to determine the corresponding accurate feature points in the accurate feature point set of the visible light image, and is recorded as The registration relationship between the contour feature points is determined as formula (8):

[0108]

[0109] As a preferred embodiment of the present embodiment, the step S5 of eliminating the mismatch between the accurate feature point set of the visible light image and the contour feature point set of the infrared light image to realize the final registration of the contour feature point set comprises:

[0110] S51: An iterative nearest neighbor method is used to construct a contour feature point set registration loss function based on adaptive weight as formula (9):

[0111]

[0112] wherein i and j represent the index of the contour feature point set, represents the contour feature point set of the infrared light image, represents each feature point in the accurate feature point set of the visible light image the corresponding accurate feature point in the accurate feature point set of the visible light image, and nInf represents the number of matched contour feature points, A represents a rigid registration matrix, R * , t * is a rotation matrix and a translation matrix, and ω represents a weight matrix, and the size of ω is nInf*1;

[0113] S52: The rotation matrix R * and the translation matrix t * in formula (9) are solved respectively, and the calculation of the rotation matrix R * is shown in formula (10):

[0114] R * = UV T (10)

[0115] wherein U and V are orthogonal matrices of SVD decomposition;

[0116] The calculation of the translation matrix t * is shown in formula (11):

[0117] t * = g-R * p (11)

[0118] wherein g and p are the contour center points,

[0119] S53: combining the similarity between feature points and the registration distance between feature points to realize adaptive allocation of the weight matrix, and the calculation of each weight coefficient {ω i |i∈[1,n inf ]} is shown in equation (12):

[0120]

[0121] Wherein, n inf represents the number of matched contour feature points, s i,j represents the minimum value of the distance between the two feature descriptors, d i,j represents the distance between the two image feature points, represents the distance between the two image feature points after normalization, iter represents the iteration number, and k represents the index value of the kth contour point, represents the k iter-1 th rotation matrix, represents the k iter-1 th translation matrix.

[0122] As Figure 3 shown, the space-time registration method of the metallurgical melt jet multi-modal image in the embodiment can realize accurate registration of the metallurgical melt jet visible light image and the infrared image.

[0123] The application also provides a space-time registration system for metallurgical melt jet multi-modal images, comprising: a binocular vision detection module, a hardware signal triggering module and a calculation processing module connected in sequence;

[0124] The binocular vision detection module is used to acquire synchronized metallurgical melt jet multi-modal images based on a trigger signal, and the images include a first modal image and a second modal image;

[0125] The hardware signal triggering module is used to send the trigger signal to the binocular vision detection module based on instructions from the calculation processing module, and transmit the metallurgical melt jet multi-modal images acquired by the binocular vision detection module to the calculation processing module;

[0126] The calculation processing module is used to send instructions to the hardware signal triggering module, and perform the following steps:

[0127] The metallurgical melt jet profile in the first modality image and the second modality image is extracted, and the texture details of the profile are filtered to obtain a smooth profile; the smooth profiles of the first modality image and the second modality image are analyzed to extract a profile feature point set, and the profile feature point sets of the first modality image and the second modality image are coarsely matched; a search range is determined, the profile feature point set of the first modality image is searched based on the search range to obtain an accurate feature point set, a feature point descriptor is constructed for the accurate feature point set and the profile feature point set of the second modality image, and the accurate feature point set of the first modality image and the profile feature point set of the second modality image are precisely matched based on the feature point descriptor; the false matches between the accurate feature point set of the first modality image and the profile feature point set of the second modality image are excluded, and the final registration of the profile feature point set is realized.

[0128] As a preferred embodiment of the present embodiment, the hardware signal triggering module comprises: a voltage step signal triggering unit, a voltage stabilizing unit and a remote network communication unit.

[0129] The voltage step signal triggering unit is configured to generate a transient voltage step signal.

[0130] The voltage stabilizing unit is connected with the voltage step signal triggering unit and the remote network communication unit, and is configured to realize voltage step-down conversion among multiple voltages and supply power to the voltage step signal triggering unit and the remote network communication unit.

[0131] The remote network communication unit is connected with the voltage step signal triggering unit and performs network communication with the outside.

[0132] In this embodiment, due to the high dynamic characteristics of the jet flow, it is difficult to synchronously acquire multi-modality metallurgical melt jet images. Therefore, in order to synchronously acquire high dynamic jet flow multi-modality images, the present application designs a multi-modality image synchronous acquisition circuit based on signal triggering, which mainly consists of a voltage step signal trigger, a voltage stabilizing circuit and a remote network communication unit. The voltage step signal trigger is the core of the signal triggering circuit, which generates a transient voltage step signal accurately according to the set acquisition period through the MOS tube combination control circuit, and realizes accurate synchronous control of the signal triggering circuit. The voltage stabilizing circuit module realizes voltage step-down conversion among multiple voltages and control of output current. The network communication realizes image synchronous acquisition and real-time network communication. Through the hardware signal triggering circuit, the synchronous acquisition of high dynamic jet flow multi-modality images is realized.

[0133] The metallurgical melt jet multi-modality image space-time registration system can realize each embodiment of the metallurgical melt jet multi-modality image space-time registration method described above and achieve the same beneficial effects, and thus will not be described here in detail.

[0134] The preferred embodiments of the present application have been described above in detail. It should be understood that modifications and variations to the present application can be affected by those skilled in the art without departing from the scope of the application. Accordingly, it is intended that all possible modifications and alterations be included within the scope of the present application as defined by the following claims.

Claims

1. A method for spatio-temporal registration of metallurgical melt jet multi-modal images, characterized in that, The method comprises the following steps: S1: synchronously acquiring metallurgical melt jet multi-modal images by using a binocular vision detection device, the images comprising a first modal image and a second modal image; S2: extracting a metallurgical melt jet profile in the first modal image and the second modal image, and filtering out texture details of the profile to obtain a smoothed profile; S3: analyzing the smoothed profiles of the first modal image and the second modal image to extract a profile feature point set, and performing coarse matching on the profile feature point sets of the first modal image and the second modal image; S4: determining a search range, searching the profile feature point set of the first modal image based on the search range to obtain an accurate feature point set, constructing a feature point descriptor for the accurate feature point set and the profile feature point set of the second modal image, and performing fine matching on the accurate feature point set of the first modal image and the profile feature point set of the second modal image based on the feature point descriptor; S5: excluding false matching between the accurate feature point set of the first modal image and the profile feature point set of the second modal image, and realizing final registration of the profile feature point set; S4: the determination of the search range and the search of the profile feature point set of the first modal image based on the search range to obtain the accurate feature point set, comprising: S41: specify the search length as [k-n, k+n] and the search width as The search range is obtained by combining the search length and the search width, The calculation is shown in formula (1): wherein k represents an index value of the kth contour point, n represents a search length around the kth contour point, d i is an Euclidean distance between feature points, i, j represent index values of the feature point set l cur,col of the first modality image and the feature point set l cur,inf of the second modality image, respectively. a set of contour feature points of the first modality image according to the search range searching around the corresponding contour point ​ S42: Calculate the spatial gradient change of the pixel points in the search range, take the points reaching the specified threshold as the candidate points of the contour feature points, determine the accurate feature points according to the candidate point positions through Taylor expansion, and obtain all the accurate feature points to form an accurate feature point set S4: the construction of the feature point descriptor for the accurate feature point set and the profile feature point set of the second modal image, and the fine matching on the accurate feature point set of the first modal image and the profile feature point set of the second modal image, comprising: S43: the contour feature point set of the second modality image and the accurate feature point set of the corresponding first modality image constructing the descriptor through the contour angle and Disc cur,col , calculating the feature point by using the mean filtering algorithm the contour vectors on the left and right sides The calculation is shown in formula (2): Wherein, i, j represent the index values of the profile feature point set respectively; The two side profile vectors are normalized and combined into a main direction vector The calculation The profile angle of the profile feature point is obtained by the ratio in the x and y dimensions The calculation is shown in equation (3): S44: Constructing descriptors Then, similarity measure is performed on formula (3) to determine The corresponding accurate feature point of each feature point in is recorded as The registration relationship between the contour feature points is determined as formula (4): S5: the exclusion of the false matching between the accurate feature point set of the first modal image and the profile feature point set of the second modal image, and the realization of the final registration of the profile feature point set, comprising: S51: constructing a profile feature point set registration loss function based on adaptive weight by using a nearest neighbor point iteration method, as formula (5): wherein i, j represent the index of the contour feature point set, a contour feature point set representing the second modality image, representing a point set composed of the corresponding accurate feature points of each feature point in the first modality image , nInf represents the number of matched contour feature points, A represents a rigid registration matrix, R * , t * is a rotation matrix and a translation matrix, and ω represents a weight matrix, the size of ω is nInf*1; S52: Solve for the rotation matrix R and the translation matrix t* in equation (5) * respectively, the rotation matrix R * is calculated as in equation (6): R * = UV T (6) Wherein, U, V are orthogonal matrices of SVD decomposition; Translation matrix t * The calculation of t is given by equation (7): t * = g - R * p (7) wherein g and p are the profile center points, S53: Combining the similarity between feature points and the registration distance between feature points to realize adaptive allocation of the weight matrix, and the calculation of each weight coefficient {ω i |i∈[1,n inf ]} is seen in formula (8): Where, n inf s represents the number of matched contour feature points. i,j d represents the minimum distance between two feature descriptors. i,j This represents the distance between feature points in two images. This represents the distance between feature points in the two images after normalization, where iter represents the iteration number, and k represents the index of the k-th contour point. Indicates the kth iter-1 Rotation matrices, Indicates the kth iter-1 Translation matrices.

2. The spatiotemporal registration method of metallurgical melt stream multi-modal images of claim 1, wherein, The S2 comprises: S21: extracting a metallurgical melt jet profile in the first modal image and the second modal image by using an active contour method; S22: filtering out texture details of the profile by using a contour smoothing model based on variational regularization, and the calculation of the smoothed profile is shown in formula (9): where k, p are the coefficients of the optimization objective, and ||·|| are the Frobenius and L1 norms, respectively, l is the optimized profile, m is the original profile, ω is the weight matrix, ∇ is the first-order difference filter, and · denotes element-wise multiplication.

3. The spatio-temporal registration method of metallurgical melt stream multi-modal images of claim 1, wherein, S3: the analysis of the smoothed profiles of the first modal image and the second modal image to extract a profile feature point set, comprising: calculating local curvatures of the smoothed profiles of the first modal image and the second modal image to extract a profile feature point set, and the calculation of the profile feature point set is shown in formula (10): wherein, l i,x , l i,y , l i,k denote the x, y coordinates and the corresponding curvature of the i-th index of the contour l, denote the first and second derivatives of the contour l, respectively, c, l cur denote the selection threshold of the contour curvature and the selected contour feature point set, respectively.

4. The spatio-temporal registration method of metallurgical melt stream multi-modal images of claim 1, wherein, S3: the coarse matching on the profile feature point sets of the first modal image and the second modal image, comprising: extracting a corresponding set of contour feature points l cur After that, the nearest contour point is found as the corresponding point by a greedy algorithm to determine the rough matching relationship between the contour feature point sets of the first and second modal images, and the calculation formula is: Wherein, i, j respectively represent the index values of the profile feature point sets l cur ,col 、 cur,inf of the metallurgical melt jet profile l i,a indicates the x or y coordinate of the i-th index of the metallurgical melt jet profile l, and the coarse matching relationship between the profile feature point sets is obtained by formula (11):

5. A time-space registration system for metallurgical melt jet multi-modal images, used for realizing the multi-modal image registration method according to any one of claims 1-4, the system comprising a binocular vision detection module, a hardware signal triggering module and a calculation processing module connected in sequence; The binocular vision detection module: used for acquiring synchronous metallurgical melt jet multi-modal images based on a triggering signal, the images comprising a first modal image and a second modal image; The hardware signal triggering module is configured to send the triggering signal to the binocular vision detection module based on the instruction from the computing processing module, and transmit the multi-modal image of the metallurgical melt jet obtained by the binocular vision detection module to the computing processing module. The computing processing module is configured to send the instruction to the hardware signal triggering module, and perform the following steps: extracting the metallurgical melt jet profile in the first modal image and the second modal image, respectively filtering out the texture details of the profile to obtain a smooth profile, analyzing the smooth profile of the first modal image and the second modal image to extract a set of profile feature points, and performing coarse matching on the set of profile feature points of the first modal image and the second modal image; determining a search range, searching the set of profile feature points of the first modal image based on the search range to obtain an accurate set of feature points, constructing a feature point descriptor for the accurate set of feature points and the set of profile feature points of the second modal image, and performing fine matching on the accurate set of feature points of the first modal image and the set of profile feature points of the second modal image based on the feature point descriptor; excluding the mis-matching between the accurate set of feature points of the first modal image and the set of profile feature points of the second modal image, and realizing the final registration of the set of profile feature points.

6. The metallurgical melt jet multi-modal image space-time registration system of claim 5, wherein, The hardware signal triggering module comprises a voltage step signal triggering unit, a voltage stabilizing unit and a remote network communication unit; the voltage stabilizing unit is connected with the voltage step signal triggering unit and the remote network communication unit, and the remote network communication unit is connected with the voltage step signal triggering unit; The voltage step signal triggering unit is configured to generate a transient voltage step triggering signal. The voltage stabilizing unit is configured to realize voltage step-down conversion between multiple voltages, and supply power to the voltage step signal triggering unit and the remote network communication unit.

Images