A method and system for taking images of a dim-light high-temperature industrial stereoscopic endoscope

Abstract

The present invention discloses a method and system for imaging with a dim-light high-temperature industrial stereo endoscope. By acquiring the original video streams collected by multiple optical imaging systems, performing high-brightness video synthesis on the original video streams to obtain a brightness-enhanced output video stream, and performing spatio-temporal matching on the key-frame images extracted during the process of the material surface descending movement within the cloth cycle to obtain the relative position estimation of the material surface - endoscope, thereby inversely calculating the spatial position of the virtual endoscope, constructing a virtual-real multi-camera array, and reconstructing the material surface inside the industrial furnace, a three-dimensional image of the blast furnace material surface is obtained, solving the technical problem of being difficult to obtain a three-dimensional material surface image with high definition and high brightness under extremely weak light conditions. By performing high-brightness synthesis on the video streams collected by multiple optical imaging systems and constructing a virtual-real multi-endoscope array to reconstruct the blast furnace material surface, the brightness of the output blast furnace material surface image is improved, and three-dimensional high-brightness imaging of the blast furnace material surface image in a dim-light environment is achieved.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of high-temperature industrial endoscopes, and particularly refers to a method and system for obtaining images of a dim-light high-temperature industrial stereo endoscope. Background Art

[0002] As the most critical equipment in iron and steel smelting, the blast furnace accounts for about 60% - 70% of the energy consumption and cost in the entire iron and steel production process, and its carbon dioxide emissions account for more than 90%. Therefore, controlling its energy consumption is the key to realizing green production in the iron and steel industry. The three-dimensional topography information of the blast furnace burden surface can effectively reflect the internal operation status and flow field distribution of the blast furnace, and is of great significance for guiding refined burden distribution, improving the gas flow distribution, ensuring the smooth operation of the blast furnace, and reducing smelting energy consumption. However, the internal environment of the blast furnace is harsh, with a temperature reaching over 800°C, the overall is airtight and lightless, the internal burden layer has a strong light absorption effect, making the internal illuminance reach the dim-light level (less than 0.0001 Lux), and it is filled with a large amount of industrial gas and dust, which greatly limits the imaging range and performance of detection equipment. Obtaining high-definition and high-brightness three-dimensional topography information of the burden surface and realizing three-dimensional imaging in such an environment have always been extremely challenging problems. Currently, three-dimensional imaging is mainly achieved by infrared imaging systems, laser imaging systems, and radar scanning systems. Among them, the infrared imaging system obtains multi-segment spectral information through multiple infrared cameras, and obtains the depth information of an object based on the binocular stereo imaging principle. However, infrared cameras are easily affected by high-concentration dust and water vapor in the blast furnace, and the image quality and clarity obtained are relatively low, making it difficult to provide effective burden surface topography information; the laser imaging system measures information such as the distance position of a target through a laser scanner, processes the data information, and finally outputs a three-dimensional image through a three-dimensional image synthesis device. However, the probe of the laser scanner is easily blocked by dust in the blast furnace, affecting its measurement accuracy and having low practical value; the radar scanning system measures data at different positions of an object through multi-dimensional radar components, and uses the data for mathematical modeling to achieve three-dimensional imaging and obtain the coordinates and height information of each point of the object. However, this system requires opening multiple detection holes at the top of the blast furnace, which is likely to affect the airtightness of the blast furnace, and the detection accuracy is easily interfered by dust.

[0003] Patent Publication No. CN209787294U discloses a multi-spectral three-dimensional imaging system, which obtains the parallax images of a target through two identical multi-spectral cameras. Each camera includes a near-infrared and a far-infrared camera and a beam splitter to ensure a high overlap degree between the collected images, and sends the parallax images to a visual information processing system to obtain real-time and clear three-dimensional images. However, this system has many electronic devices and is difficult to adapt to the high-temperature and high-pressure environment in the blast furnace. Moreover, under extremely low-light conditions, it is impossible to obtain high-definition and high-brightness three-dimensional images of the burden surface. In addition, infrared cameras are easily affected by the high-concentration dust concentration in the furnace, greatly reducing the detection accuracy.

[0004] Patent Publication No. CN205643889U discloses a stereoscopic imaging system, including a laser scanner, a kinematic differential GPS, an attitude measurement device, a data processor, and a stereoscopic image synthesis device. The attitude, distance, position, and other information of the target are measured by the laser scanner, the attitude measurement device, and the kinematic differential GPS, and the data is transmitted to the data processor. Finally, a stereoscopic image is synthesized by the stereoscopic image synthesis device. However, this system contains many devices and is difficult to install in the limited installation space of the blast furnace. Moreover, the high-temperature dust in the furnace easily adheres to the probes of devices such as the laser scanner, forming crusts and blockages, affecting the precision of the imaging system. Therefore, this system does not have great practical value in the harsh environment of the blast furnace.

[0005] Patent Publication No. CN111273272B discloses a 3D radar scanner for blast furnace burden surface imaging and a blast furnace burden surface detection system. The 3D radar scanner includes a base member, a high-temperature isolation cover member, and a multi-dimensional radar member installed at the top of the blast furnace. This system can measure the burden level height information of multiple points in the blast furnace in a high-temperature environment and transmit it to the host computer through optical fibers for mathematical modeling. The 3D image of the burden surface in the blast furnace is simulated and displayed, and the height and coordinates of each measurement point are included in the image. However, the 3D image simulated through the height information lacks details such as the burden surface contour and burden particles and cannot reflect the overall operating conditions of the blast furnace. At the same time, using the radar scanning method to measure the burden level height is easily affected by the dust in the furnace, resulting in large measurement errors and limiting the accuracy of the simulated 3D image. Summary of the Invention

[0006] The method and system for obtaining images with a dim-light high-temperature industrial endoscope provided by the present invention solve the technical problem of difficultly obtaining three-dimensional burden surface images with high definition and high brightness under extremely low light conditions.

[0007] To solve the above technical problems, the method for obtaining images with a dim-light high-temperature industrial endoscope proposed by the present invention includes:

[0008] Obtain the original video stream collected by the optical imaging system, where the optical imaging system is used to image the burden surface in the industrial furnace, and the number of groups of the optical imaging system is greater than 1;

[0009] Perform high-brightness video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream;

[0010] Extract the key frame images of the brightness-enhanced output video stream during the process of the burden surface descending movement within the charging cycle, and obtain the relative position estimation of the burden surface - endoscope through spatio-temporal matching of the key frame images;

[0011] According to the relative position estimation of the burden surface - endoscope, reverse infer the spatial position of the virtual endoscope;

[0012] Construct a virtual-real multi-camera array according to the spatial position of the virtual endoscope;

[0013] Based on the virtual-real multi-camera array, reconstruct the material surface inside the industrial furnace to obtain a three-dimensional image of the blast furnace burden surface.

[0014] Furthermore, perform high-brightness video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream, including:

[0015] Calculate the fidelity function according to the similarity between the brightness-enhanced video frame estimate and the low-light image in the input original video stream;

[0016] Calculate the consistency function according to the temporal consistency between the brightness-enhanced video frame estimate and the brightness-enhanced video frame estimate of the previous frame;

[0017] Calculate the regularization function according to the brightness-enhanced video frame estimate;

[0018] Adopt the maximum a posteriori probability method to fuse the fidelity function, the consistency function and the regularization function to obtain the brightness-enhanced output video frame corresponding to the brightness-enhanced video frame estimate;

[0019] Obtain the brightness-enhanced output video stream according to the brightness-enhanced output video frame.

[0020] Furthermore, adopt the maximum a posteriori probability method to fuse the fidelity function, the consistency function and the regularization function to obtain the brightness-enhanced output video frame corresponding to the brightness-enhanced video frame estimate, including:

[0021] Adopt the maximum a posteriori probability method to fuse the fidelity function, the consistency function and the regularization function, and calculate the brightness-enhanced video frame corresponding to the brightness-enhanced video frame estimate in each group of optical imaging systems respectively;

[0022] Calculate multiple groups of fidelity functions according to the similarity between the brightness-enhanced output video frame estimate and each group of brightness-enhanced video frames;

[0023] Adopt the maximum a posteriori probability method to perform weighted fusion on multiple groups of fidelity functions to obtain the brightness-enhanced output video frame.

[0024] Furthermore, the calculation formula for adopting the maximum a posteriori probability method to fuse the fidelity function, the consistency function and the regularization function, and calculating the brightness-enhanced video frame corresponding to the brightness-enhanced video frame estimate in each group of optical imaging systems respectively is:

[0025]

[0026] Among them, i represents the group number of the optical imaging system, and Respectively represent the estimated brightness-enhanced video frames of the i-th group of optical imaging systems at time t and t-1, Y it Represents the brightness-enhanced video frame of the i-th group of optical imaging systems at time t, I ij Represents the low-light image of the i-th group of optical imaging systems at time j, And Respectively represent the fidelity function, the consistency function and the regularization function, Represents The posterior probability of, D and B respectively represent the blurring matrix and the downsampling matrix, Represents the variance of the joint distribution, Represents To I ij The motion compensation matrix of, and Where ρ is a scalar, Represents And The motion compensation matrix of, And Respectively represent moving the image By l and h pixels in the horizontal and vertical directions respectively. Wide is the size of the moving window, r is the number of low-light images before time t of the original video stream used for the brightness-enhanced video frame estimation, and b is the number of low-light images after time t of the original video stream used for the brightness-enhanced video frame estimation.

[0027] Furthermore, using the maximum a posteriori probability method, the weighted fusion of multiple groups of fidelity functions is carried out, and the calculation formula for obtaining the brightness-enhanced output video frame is:

[0028]

[0029] Among them, the number of groups of optical imaging systems is three, And Respectively represent the posterior probabilities of the brightness-enhanced output video frame estimations of the first to third groups of optical imaging systems, O 1t 、O 2t And O 3t Respectively represent the calculation results of the brightness-enhanced output video frames of the first to third groups of optical imaging systems. D and B respectively represent the blurring matrix and the downsampling matrix, and σ1, σ2 and σ3 are respectively the standard deviations of the joint distributions of the three groups of optical imaging systems, And Are respectively To And The motion compensation matrix of, And Are respectively To And The motion compensation matrix of,​ and are respectively to and 's motion compensation matrices. μ1, μ2, and μ3 are respectively user-defined composite weight values, and all three composite weight values are greater than 0 and less than 1.

[0030] Furthermore, based on the estimation of the relative position between the burden surface and the endoscope, the spatial position of the virtual endoscope is deduced as follows:

[0031] Based on the estimation of the relative position between the burden surface and the endoscope, combined with spatial triangular transformation, the spatial position of the virtual endoscope is deduced. The calculation formula for the spatial position of the virtual endoscope is:

[0032]

[0033] where, is the coordinate of any pixel point P in the key frame image of the brightness-enhanced output video stream output by the i-th group of optical imaging systems in the virtual endoscope coordinate system, is the coordinate of pixel point P in the world coordinate system, c i and w i are respectively the virtual endoscope coordinate system and the world coordinate system of the i-th group of optical imaging systems, K i and M i are respectively the endoscope internal parameter matrix and the external parameter matrix of the i-th group of optical imaging systems, is the true endoscope focal length of the i-th group of optical imaging systems, x is the width of the object in the key frame image, y is the height of the object in the key frame image, s i is the tilt factor of the i-th group of optical imaging systems, (u i , v i ) is the intersection point of the true endoscope main axis of the i-th group of optical imaging systems and the plane of the key frame image, R i and T i are the rotation transformation matrix and the translation transformation matrix from the world coordinate system to the endoscope coordinate system of the i-th group of optical imaging systems.

[0034] Furthermore, based on the spatial position of the virtual endoscope, a virtual-real multi-camera array is constructed as follows:

[0035] Select virtual endoscopes with parallax information to construct a virtual endoscope array, and fuse the pose information of the real endoscope camera to construct a virtual-real multi-camera array.

[0036] Furthermore, based on the virtual-real multi-camera array, the burden surface in the industrial furnace is reconstructed as follows:

[0037] Using the sliding window matching method, the parallax value of the pixel points of the key frame images in the brightness-enhanced output video streams output by any two optical imaging systems is calculated based on the Mahalanobis distance. The specific calculation formula is as follows:

[0038]

[0039] where P1, P2, and P3 are the parallax values between the key frame images in the brightness-enhanced output video streams output by two different optical imaging systems at time t, P0 is the sum of the average parallax value between different optical imaging systems and the parallax value of the front and rear key frames of the selected i-th group of optical imaging systems, W is the one-dimensional window width, d is the translation distance of the sliding window, m and n are the column number and row number of the pixel points of the key frame image, respectively, V it is the key frame image in the brightness-enhanced output video stream output by the i-th group of optical imaging systems at time t, and the value range of i is i ∈ {1, 2, 3}, s0 represents the sliding window range, and ∑ is the covariance matrix;

[0040] The spatial position coordinates of the reconstructed points are obtained by using the triangulation method:

[0041]

[0042] where (X, Y, Z) are the spatial position coordinates of the reconstructed points, the coordinate origin of the coordinate system where the spatial position coordinates of the reconstructed points are located is the virtual endoscope center corresponding to the reconstructed key frame image, and the reconstructed key frame image is the key frame image in the brightness-enhanced output video stream output by the optical imaging system corresponding to the reconstructed points, d L is the distance between the virtual and real endoscopes of adjacent optical imaging systems, X L and Y L are the horizontal distance and vertical distance from the pixel point corresponding to the reconstructed point in the reconstructed key frame image to the center of the reconstructed key frame image, respectively, and F is the distance from the real endoscope to the CCD imaging chip in the optical imaging system corresponding to the reconstructed points;

[0043] Based on the spatial position coordinates of the reconstructed points, the material surface in the industrial furnace is reconstructed.

[0044] Furthermore, the optical imaging system includes an objective lens group, an optical relay system lens group, and a zoom lens group that are connected in sequence, where:

[0045] The objective lens group adopts a retrofocus objective lens structure and is used for large field of view imaging;

[0046] The optical relay system lens group is used to transmit the image obtained by the objective lens group to the photosensitive chip without loss;

[0047] The zoom lens group is used to achieve autofocus during the shooting process.

[0048] The dim-light high-temperature industrial endoscope imaging system provided by the present invention includes:

[0049] a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the dim-light high-temperature industrial endoscope imaging method provided by the present invention are implemented.

[0050] Compared with the prior art, the advantages of the present invention are as follows:

[0051] The dim-light high-temperature industrial endoscope imaging method and system provided by the present invention obtain the original video stream collected by the optical imaging system, which is used to image the material surface in the industrial furnace. The original video stream collected by the optical imaging system is subjected to high-brightness video synthesis to obtain a brightness-enhanced output video stream. Key frame images of the brightness-enhanced output video stream during the downward movement of the material surface within the cloth cycle are extracted. Through the spatio-temporal matching of the key frame images, the relative position estimation of the material surface - endoscope is obtained. According to the relative position estimation of the material surface - endoscope, the spatial position of the virtual endoscope is deduced backward. According to the spatial position of the virtual endoscope, a virtual-real multi-camera array is constructed, and based on the virtual-real multi-camera array, the material surface in the industrial furnace is reconstructed, thereby obtaining a three-dimensional image of the blast furnace material surface, solving the technical problem of being difficult to obtain a high-definition and high-brightness three-dimensional material surface image under extremely weak light conditions. By performing high-brightness synthesis on the video streams collected by multiple groups of optical imaging systems and constructing a virtual-real multi-endoscope array to reconstruct the blast furnace material surface, the brightness of the output blast furnace material surface image is greatly improved, thereby realizing three-dimensional high-brightness imaging of the blast furnace material surface image in a dim-light environment.

[0052] The objective of the present invention:

[0053] The objective of the present invention is to design an imaging method in a dim-light environment (illuminance lower than 0.0001 Lux) based on a dim-light level high-temperature industrial endoscope, so as to achieve high-definition and high-brightness optical image acquisition under extremely weak light, high temperature and high pressure, and airtight and dusty environments.

[0054] The objective of the present invention is to propose a video high-brightness synthesis algorithm based on spatio-temporal domain self-adaptation, which respectively superimposes and synthesizes the front and back frames of the video streams collected by each group of optical imaging systems as brightness-enhanced frames, and fuses multiple groups of brightness-enhanced output frames through a regular consistency function to generate a coherent multi-dimensional high-brightness high-definition video stream.

[0055] The objective of the present invention is to design a three-dimensional imaging method. By estimating the relative position of the material surface - endoscope through multiple groups of brightness-enhanced output frame images, combining with triangular transformation, deducing backward the spatial position of the virtual endoscope, fusing the real endoscope pose, constructing a virtual-real endoscope array, and reconstructing the material surface morphology according to the parallax information of each point in the image, three-dimensional imaging is realized.

[0056] Advantages of the present invention:

[0057] 1. In the optical imaging system, an anastigmat objective lens group is adopted to obtain the full-surface image information of the blast furnace with a wide field of view and a large depth of field; three identical Hopkins rod lens image transmission lens groups are adopted to transmit the images obtained by the objective lens group over a long distance and in an equal-proportion and lossless manner; a zoom lens group is adopted to automatically focus during the shooting process, improving the shooting efficiency of the optical system and ensuring the real-time clarity of the video.

[0058] 2. In the video high-brightness synthesis algorithm, key information in the time domain such as motion compensation, fuzzy rules, and downsampling algorithms is combined with the spatial domain regularization consistency function. First, the front and rear frames of the three-channel video stream are respectively superimposed to form a brightness-enhanced frame, and then the three-channel brightness-enhanced frames are fused into three groups of high-brightness image frames, greatly improving the brightness of the obtained video stream and realizing three-dimensional high-brightness imaging in a dim-light environment.

[0059] 3. In the stereoscopic imaging method, the virtual endoscope spatial position is deduced by the relative motion between the material surface and the endoscope, the real endoscope pose is fused, a virtual-real multi-mirror endoscope array is constructed, and each point on the material surface is reconstructed according to the parallax value of each pixel point in the image, thereby realizing the stereoscopic imaging of the blast furnace material surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the overall equipment diagram of the dim-light high-temperature industrial stereoscopic endoscope according to the second embodiment of the present invention;

[0061] Figure 2 It is the optical imaging system optical path and structure diagram of the dim-light high-temperature industrial stereoscopic endoscope according to the second embodiment of the present invention;

[0062] Figure 3 It is the schematic diagram of the video high-brightness synthesis algorithm of the dim-light high-temperature industrial stereoscopic endoscope according to the second embodiment of the present invention;

[0063] Figure 4 It is the schematic diagram of the virtual-real multi-mirror endoscope array of the dim-light high-temperature industrial stereoscopic endoscope according to the second embodiment of the present invention;

[0064] Figure 5 It is the overall flowchart of the imaging algorithm of the dim-light high-temperature industrial stereoscopic endoscope according to the third embodiment of the present invention;

[0065] Figure 6 It is the structural block diagram of the image acquisition system of the dim-light high-temperature industrial stereoscopic endoscope according to the embodiment of the present invention.

[0066] Reference Signs:

[0067] M1: Light-shielding cover; M2: Front-end sleeve; M3: Connecting rod; M4: Dim-light imaging system; M5: Fixed collar; M6: Stepper motor; M7: CCD chip; M8: Water inlet pipe; M9: Air inlet pipe; M10: Water outlet pipe; M11: Steering gear; M12: Imaging drive circuit; M13: Power supply; M14: Video signal line; M15: Rear-end sleeve; M16: Power supply signal line; 10. Memory; 20. Processor. Detailed implementation mode

[0068] To facilitate the understanding of the present invention, the following will describe the present invention more comprehensively and meticulously in conjunction with the accompanying drawings of the specification and preferred embodiments, but the protection scope of the present invention is not limited to the following specific embodiments.

[0069] The following will describe in detail the embodiments of the present invention with reference to the accompanying drawings, but the present invention can be implemented in many different ways defined and covered by the claims.

[0070] Embodiment 1

[0071] The dim-light high-temperature industrial stereo endoscope imaging method provided by Embodiment 1 of the present invention includes:

[0072] Step S101, obtaining the original video stream collected by the optical imaging system, where the optical imaging system is used to image the material surface in the industrial furnace, and the number of groups of the optical imaging system is greater than 1;

[0073] Step S102, performing high-brightness video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream;

[0074] Step S103, extracting the key frame images of the brightness-enhanced output video stream during the material surface descending movement within the cloth cycle, and obtaining the relative position estimation of the material surface - endoscope through spatio-temporal matching of the key frame images;

[0075] Step S104, inversely inferring the spatial position of the virtual endoscope according to the relative position estimation of the material surface - endoscope;

[0076] Step S105, constructing a virtual-real multi-camera array according to the spatial position of the virtual endoscope;

[0077] Step S106, reconstructing the material surface in the industrial furnace based on the virtual-real multi-camera array to obtain a three-dimensional image of the blast furnace material surface.

[0078] The method for obtaining images of a dim-light high-temperature industrial stereo endoscope provided by the embodiments of the present invention includes: acquiring an original video stream collected by an optical imaging system, where the optical imaging system is used to image the material surface in an industrial furnace; performing high-brightness video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream; extracting key-frame images of the brightness-enhanced output video stream during the downward movement of the material surface within the cloth cycle; obtaining an estimation of the relative position between the material surface and the endoscope through spatio-temporal matching of the key-frame images; inversely inferring the spatial position of the virtual endoscope based on the estimation of the relative position between the material surface and the endoscope; constructing a virtual-real multi-camera array based on the spatial position of the virtual endoscope; and reconstructing the material surface in the industrial furnace based on the virtual-real multi-camera array to obtain a three-dimensional image of the blast furnace material surface, thereby solving the technical problem of being difficult to obtain a three-dimensional material surface image with high definition and high brightness under extremely low-light conditions. By performing high-brightness synthesis on the video streams collected by multiple optical imaging systems and constructing a virtual-real multi-endoscope array to reconstruct the blast furnace material surface, the brightness of the output blast furnace material surface image is greatly increased, thereby realizing three-dimensional high-brightness imaging of the blast furnace material surface image in a dim-light environment.

[0079] Specifically, the reason for using multiple optical imaging systems in this embodiment to perform high-brightness synthesis on the collected video streams is that in the dim-light environment inside the blast furnace, the brightness improvement of the image synthesized by a single optical imaging system is limited, and many details of the material surface image are still very dim and difficult to distinguish with the human eye. By performing high-brightness synthesis on the video stream images collected by multiple optical imaging systems, the brightness and clarity of the captured images can be increased several times, so as to see most of the details and contours of the material surface inside the furnace, meeting the requirements of material surface imaging in a dim-light environment. A single optical imaging system can only capture partial material surface information. Since multiple optical imaging systems are installed at different positions, the captured field of view is wider, so the captured images contain more material surface information in different directions, and more details such as material surface textures and contours can be seen, making it easier to detect abnormal furnace conditions in a timely manner. In addition, compared with a single optical imaging system, multiple optical imaging systems can obtain multi-dimensional material surface video streams, obtain image parallax information based on this, construct a virtual-real multi-array, and obtain the depth information of the material surface image, and finally obtain the three-dimensional depth image of the material surface, providing more intuitive and rich optical image information for on-site workers. The effects are that the brightness and clarity of the synthesized image are higher than those of a single optical system, and a high-brightness image can be output in the dim-light environment inside the furnace; the image field of view is also larger than that of the image collected by a single optical system, and more comprehensive material surface information can be obtained; the multi-dimensional video stream images can be calculated to obtain image depth information, laying a foundation for realizing three-dimensional imaging.

[0080] Embodiment 2

[0081] Embodiment 2 of the present invention is based on a dim-light high-temperature industrial endoscope and proposes a method for three-dimensional imaging in a dim-light environment to obtain high-definition and high-brightness three-dimensional topography images under extremely weak light conditions. The imaging method includes the design of the optical path and structure of the optical imaging system, a video high-brightness synthesis algorithm based on spatio-temporal domain adaptation, and a three-dimensional imaging method based on a virtual-real multi-eye array.

[0082] Referring to Figure 1 , Figure 1 Figure is the overall equipment diagram of the dim-light high-temperature industrial stereo endoscope used in this embodiment to obtain a high-definition and high-brightness full-surface image in the dim-light environment inside the blast furnace. This equipment specifically includes the following structures: light-shielding cover M1, front-end sleeve M2, connecting rod M3, dim-light imaging system M4, fixed collar M5, stepping motor M6, CCD chip M7, water inlet pipe M8, air inlet pipe M9, water outlet pipe M10, steering gear M11, imaging drive circuit M12, power supply M13, video signal line M14, rear-end sleeve M15, and power supply signal line M16.

[0083] Referring to Figure 2 , Figure 2 Figure is the optical path and structure diagram of the dim-light high-temperature industrial endoscope according to the embodiment of the present invention. Specifically, in this embodiment, three optical imaging systems with the same specifications are used to form a dim-light imaging system, and the main optical path and structure of each optical imaging system are designed. Its structure consists of three parts: the front part is the objective lens group, the middle part is the optical relay system lens group, which is responsible for transmitting the image obtained by the objective lens to the photosensitive chip without loss, and the rear part is the zoom lens group, which adjusts the focal length of the optical system to ensure that the floating of the blast furnace material level does not affect the imaging clarity. In addition, the dim-light high-temperature industrial endoscope according to the embodiment of the present invention also includes:

[0084] (1) Structure and principle of the objective lens group

[0085] The objective lens group is the image-taking element of the optical imaging system. To meet the large field of view angle and short focal length in the design specifications, a retrofocus objective lens structure is adopted, with a focal length of f′. The optical power of the front group of lenses L1 is negative, with the ability to collect a large field of view and diverge the light beam; the optical power of the rear group of lenses L2 is positive, with a relatively large aperture and the ability to achieve strong light transmission. When the light beam is incident and diverged by the front group of lenses, it is imaged on the focal plane after passing through the rear group of lenses. For the off-axis large-angle (α) light beam, after being diverged by the front group, the field of view angle α′ becomes smaller relative to the rear group, thus achieving the purpose of large-field imaging. The retrofocus objective lens group structure designed according to the above principle consists of 6 lenses, and its first and second lenses are the objective negative lens group, and the third to sixth lenses are the objective positive lens group.

[0086] (2) Structure and principle of the relay lens group

[0087] The relay lens group is mainly responsible for transmitting the image obtained by the objective lens group over a long distance without loss to the back-end image processing unit. It consists of three identical image transmission lens groups. The focal length of the front lens group of the image transmission lens group is the same as the focal length f′ of the objective lens group, so that the light rays emitted from the imaging point of the objective lens group become parallel light rays after passing through the front lens group. After the parallel light rays pass through the rear lens group, they are focused on the focal plane again to achieve proportional transmission of the image.

[0088] Based on the above analysis, the structure of each image transmission lens group adopts two symmetrical Hopkins rod lenses. Each rod lens is glued together by a thick biconvex lens and two completely symmetrical negative lenses. To achieve equal-proportion image transmission, the magnification of the image transmission lens group is -1. According to the relationship between the conjugate distance and the focal length, it can be known that:

[0089]

[0090] In the formula, M is the conjugate distance between the object and image planes of the image transmission lens group, f′ is the focal length of the image transmission lens group, and β e is the transverse magnification of the image transmission lens group. It can be obtained that the conjugate distance M = 4f′, that is, the length l e of the image transmission lens group = 4f′. Therefore, three image transmission lens groups are adopted to transmit the material surface image obtained by the objective lens group over a long distance and in equal proportion.

[0091] (3) Zoom lens group

[0092] According to Gauss optical theory, the optical power of an optical system composed of two components is:

[0093]

[0094] Among them are the optical powers of the two components respectively, which cannot be changed. Therefore, the principle of the zoom lens group is to change the system focal length by changing the component interval d. Since the height of the material surface in the blast furnace changes up and down continuously during the ironmaking process such as burden distribution, its distance from the optical system will also change accordingly. Therefore, this optical system adopts a zoom lens group with a focal length range of 10 - 200 mm, which automatically focuses during the shooting process, improves the shooting efficiency of the optical system, and ensures the real-time clarity of the material surface video. The first lens of the zoom lens group is the front fixed group, the second lens is the moving lens group, and the movement of the lens group is realized by a stepper motor. The third lens is the rear fixed group.

[0095] Referring to Figure 3 , based on the video highlight synthesis algorithm based on spatio-temporal domain adaptation in this embodiment, the specific steps for performing highlight video synthesis on the original video stream collected by the optical imaging system to obtain the brightness-enhanced output video stream are as follows:

[0096] (1) Calculate the fidelity function according to the similarity between the estimated brightness-enhanced video frame and the low-light image in the input original video stream. The fidelity function The luminance-enhanced frame image of the current optical imaging system is measured and the input low-light image I ij The similarity between them is defined as:

[0097]

[0098] where D and B represent the blurring matrix and the downsampling matrix respectively, represents the joint distribution variance, represents from ij to I

[0099] (2) The temporal consistency between the luminance-enhanced video frame estimate and the luminance-enhanced video frame estimate of the previous frame is estimated, and the consistency function is calculated. The consistency term is used to control the luminance-enhanced frame estimate and The temporal consistency between them is defined as:

[0100]

[0101] where ρ is a scalar,

[0102]

[0103] (3) According to the luminance-enhanced video frame estimate, the regularization function is calculated. The regularization function is the prior function of the bilateral total variation, suppressing the noise information of the image:

[0104]

[0105] and respectively represent moving the image by l and h pixels in the horizontal and vertical directions respectively. Wide is the size of the moving window, r is the number of low-light image frames of the original video stream used for the luminance-enhanced video frame estimate before time t, and b is the number of low-light image frames of the original video stream used for the luminance-enhanced video frame estimate after time t.

[0106] (4) The maximum a posteriori probability method is adopted to fuse the fidelity function, the consistency function and the regularization function, and the luminance-enhanced video frames corresponding to the luminance-enhanced video frame estimate in each group of optical imaging systems are calculated respectively. Specifically, in this embodiment, the maximum a posteriori probability method is adopted to fuse the fidelity function, the consistency function and the regularization function, and the calculation formula for calculating the luminance-enhanced video frames corresponding to the luminance-enhanced video frame estimate in each group of optical imaging systems is:

[0107]

[0108] wherein, i represents the group number of the optical imaging system, and respectively represent the estimated luminance enhanced video frames of the i-th group of optical imaging systems at times t and t - 1, Y it represents the luminance enhanced video frame of the i-th group of optical imaging systems at time t, I ij represents the low-light image of the i-th group of optical imaging systems at time j; and respectively represent the fidelity function, the consistency function, and the regularization function, represents the posterior probability of, D and B respectively represent the blurring matrix and the downsampling matrix, represents the variance of the joint distribution, represents to I ij the motion compensation matrix, and where ρ is a scalar, represents to the motion compensation matrix, and respectively represent moving the image by l and h pixels in the horizontal and vertical directions respectively, Wide is the size of the moving window, r is the number of low-light image frames of the original video stream used for the estimated luminance enhanced video frame before time t, and b is the number of low-light image frames of the original video stream used for the estimated luminance enhanced video frame after time t.

[0109] (5) Calculate the multi-group fidelity functions according to the similarity between the estimated luminance enhanced output video frame and each group of luminance enhanced video frames;

[0110] (6) Adopt the maximum a posteriori probability method to perform weighted fusion on the multi-group fidelity functions to obtain the luminance enhanced output video frame.

[0111] Specifically, in this embodiment, the maximum a posteriori probability method is adopted to perform weighted fusion on the multi-group fidelity functions, and the calculation formula for obtaining the luminance enhanced output video frame is:

[0112]

[0113] wherein, the number of groups of the optical imaging system is three, and respectively represent the posterior probabilities of the estimated luminance enhanced output video frames of the first to third groups of optical imaging systems, O 1t 、O 2t and O 3trespectively represent the calculation results of the brightness-enhanced output video frames of the first to third groups of optical imaging systems, D and B respectively represent the blurring matrix and the downsampling matrix, and σ1, σ2, and σ3 are respectively the joint distribution standard deviations of the three groups of optical imaging systems. and are respectively to and motion compensation matrices of and are respectively to and motion compensation matrices of and are respectively to and motion compensation matrices of, μ1, μ2, and μ3 are respectively user-defined synthesis weights, and all three synthesis weights are greater than 0 and less than 1.

[0114] The main steps of the stereoscopic imaging method based on the virtual-real multi-eye array in this embodiment are as follows:

[0115] (1) Perform time-division multiplexing on the three optical imaging systems. Using the parallax when the furnace top burden surface and the endoscope move relative to each other as a clue, extract the key frames of the three-dimensional high-definition and high-brightness video stream during the downward movement of the burden surface within the burden cycle. Through spatio-temporal matching of the key frames, obtain the relative position estimation of the burden surface - endoscope, and combine spatial triangular transformation to inversely deduce the spatial position of the virtual endoscope. That is, take any pixel point P in the key frame image of the brightness-enhanced output video stream output by the i-th group of optical imaging systems, then the relative position estimation of the virtual endoscope is:

[0116]

[0117] where, is the coordinate of any pixel point P in the key frame image of the brightness-enhanced output video stream output by the i-th group of optical imaging systems in the virtual endoscope coordinate system, is the coordinate of pixel point P in the world coordinate system, c i and w i are respectively the virtual endoscope coordinate system and the world coordinate system of the i-th group of optical imaging systems, K i and M i are respectively the endoscope internal parameter matrix and the external parameter matrix of the i-th group of optical imaging systems, is the true endoscope focal length of the i-th group of optical imaging systems, x is the width of the object in the key frame image, y is the height of the object in the key frame image, s i is the tilt factor of the i-th group of optical imaging systems, (u i , v i) is the plane intersection point of the true endoscope main axis of the i-th group of optical imaging systems and the key frame image, R i and T i are the rotation transformation matrix and the translation transformation matrix from the world coordinate system to the endoscope coordinate system of the i-th group of optical imaging systems.

[0118] (2) Preferably, a virtual endoscope array with parallax information is constructed, and the pose information of the real endoscope camera is fused to construct a virtual-real multi-view camera array. For details, refer to Figure 4 . The parallax information in this embodiment specifically refers to the direction difference generated when different endoscopes photograph the same target.

[0119] (3) Using the sliding window matching method, the parallax value of the pixel points of the key frame images in the brightness-enhanced output video streams output by any two optical imaging systems is calculated based on the Mahalanobis distance. The specific calculation formula is:

[0120]

[0121] where P1, P2, and P3 are the parallax values between the key frame images in the brightness-enhanced output video streams output by two different optical imaging systems at time t, P0 is the sum of the average parallax between different optical imaging systems and the parallax value of the front and back key frames of the selected i-th group of optical imaging systems, W is the one-dimensional window width, d is the translation distance of the sliding window, m and n are the column number and row number of the pixel points of the key frame image respectively, V it is the key frame image in the brightness-enhanced output video stream output by the i-th group of optical imaging systems at time t, and the value range of i is i ∈ {1, 2, 3}, s0 represents the sliding window range, and ∑ is the covariance matrix.

[0122] (4) Calculate the parallax values of all pixel points, and the spatial position coordinates of the reconstructed point corresponding to this point can be obtained by using the triangulation method:

[0123]

[0124] where (X, Y, Z) are the spatial position coordinates of the reconstructed point, the coordinate origin of the coordinate system where the spatial position coordinates of the reconstructed point are located is the virtual endoscope center corresponding to the reconstructed key frame image, and the reconstructed key frame image is the key frame image in the brightness-enhanced output video stream output by the optical imaging system corresponding to the reconstructed point, d L is the distance between the virtual and real endoscopes of adjacent optical imaging systems, X L and Y L are the horizontal distance and vertical distance from the pixel point corresponding to the reconstructed point in the reconstructed key frame image to the center of the reconstructed key frame image respectively, and F is the distance from the real endoscope to the CCD imaging chip in the optical imaging system corresponding to the reconstructed point.

[0125] (5) Reconstruct the burden surface inside the industrial furnace based on the spatial position coordinates of the reconstructed points.

[0126] According to the above method, finally, a three-dimensional stereoscopic video of the high-furnace burden surface with high definition and high brightness can be obtained under the dim light environment, so as to provide reliable burden surface morphology information for reflecting the smelting state inside the blast furnace, guiding the burden distribution operation, and improving the iron-making efficiency.

[0127] The current mainstream stereoscopic imaging technologies include the TOF method, the structured light method, the stereoscopic vision method, the triangulation method, etc. However, the blast furnace is integrally sealed and filled with toxic gases such as carbon monoxide inside. To ensure the airtightness of the blast furnace, the number of openings on the furnace top needs to be strictly limited, and the installation space on the blast furnace top is also extremely limited. The detection systems, projection systems, and laser instruments required by the TOF method, the structured light method, and the triangulation method are difficult to install on the blast furnace top, and multiple installation holes need to be opened, which is not conducive to the overall airtightness. At the same time, the equipment is easily interfered by dust and water vapor, affecting the image reconstruction accuracy. Therefore, the multi-view stereoscopic vision method can most effectively realize the reconstruction of stereoscopic images based on the existing equipment. At the same time, the stereoscopic vision method has the advantages of fast speed and good real-time performance. Compared with the traditional stereoscopic vision method, on the basis of constructing a virtual multi-view array, the embodiment of the present invention adds a real multi-view endoscope array, with the real multi-view array as the main and the virtual multi-view array as the auxiliary. The combination of the two has a much higher reconstruction accuracy than the traditional virtual multi-view reconstruction technology, can output more accurate image depth information, and is convenient for on-site workers to monitor the full-range burden surface changes of the blast furnace in real time, so as to better guide the burden distribution operation.

[0128] Embodiment III

[0129] Refer to Figure 5 , the stereoscopic imaging method based on the dim light level high-temperature industrial endoscope in this embodiment includes:

[0130] (1) Install the equipment to operate inside the blast furnace, and obtain three-channel burden surface images through three optical imaging systems.

[0131] (2) Through the video highlight synthesis algorithm based on spatio-temporal domain self-adaptation, use the motion compensation in the video stream encoding Fuzzy rule B and downsampling algorithm D and other key information in the time domain, and based on the fidelity function Respectively stack the front and back frame images I i(t-1) 、I it 、I i(t+1) of the original video streams of the three groups of optical imaging systems into a brightness-enhanced frame estimate

[0132]

[0133] (3) In the spatial domain regularization consistency function Under the constraint of frames remain coherent, and are fused according to different weights to form three groups of estimated video frames of enhanced luminance output to generate a three-dimensional high-brightness and high-definition video stream of the stock surface, with specific parameter formula (8).

[0134] (4) Extract the key frames during the process of the stock surface descending within the cloth cycle in the three-dimensional high-brightness and high-definition video stream, perform spatio-temporal matching on the key frames, obtain the relative position estimation of the stock surface - endoscope, and combine spatial triangular transformation to inversely deduce the spatial position of the virtual endoscope, with specific parameter formula (9).

[0135] (5) Optimize the virtual endoscope with sufficient parallax information to generate an array of virtual multi-view endoscopes for the stock surface, and fuse the pose information of the real endoscope to construct an array of virtual and real multi-view endoscopes.

[0136] (6) Use the sliding window matching method to calculate the parallax values of the pixel points in the key frame images of the luminance-enhanced output video streams output by any two optical imaging systems based on the Mahalanobis distance, specifically referring to formula (10).

[0137] (7) Use the triangulation method to obtain the positions (X, Y, Z) of the reconstructed points corresponding to all pixel points, and realize three-dimensional stereoscopic imaging of the blast furnace stock surface, specifically referring to formula (11).

[0138] Referring to Figure 6 , the dim-light high-temperature industrial stereoscopic endoscope imaging system proposed in the embodiment of the present invention includes:

[0139] A memory 10, a processor 20, and a computer program stored on the memory 10 and executable on the processor 20. Among them, when the processor 20 executes the computer program, it implements the steps of the dim-light high-temperature industrial stereoscopic endoscope imaging method proposed in this embodiment.

[0140] The specific working process and working principle of the dim-light high-temperature industrial stereoscopic endoscope imaging system in this embodiment can refer to the working process and working principle of the dim-light high-temperature industrial stereoscopic endoscope imaging method in this embodiment.

[0141] The above is only the preferred embodiment of the present invention, and it is not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for taking images of a dim-light high-temperature industrial stereoscopic endoscope, characterized in that, The method includes: Obtaining an original video stream collected by an optical imaging system for imaging the material surface in an industrial furnace, where the number of groups of the optical imaging system is greater than 1; Performing highlight video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream, where performing highlight video synthesis on the original video stream collected by the optical imaging system to obtain a brightness-enhanced output video stream includes: Calculating a fidelity function according to the similarity between the brightness-enhanced video frame estimate and the low-light image in the input original video stream; Calculating a consistency function according to the temporal consistency between the brightness-enhanced video frame estimate and the brightness-enhanced video frame estimate of the previous frame; Calculating a regularization function according to the brightness-enhanced video frame estimate; Using the maximum a posteriori probability method to fuse the fidelity function, the consistency function, and the regularization function to obtain a brightness-enhanced output video frame corresponding to the brightness-enhanced video frame estimate, where using the maximum a posteriori probability method to fuse the fidelity function, the consistency function, and the regularization function to obtain a brightness-enhanced output video frame corresponding to the brightness-enhanced video frame estimate includes: Using the maximum a posteriori probability method to fuse the fidelity function, the consistency function, and the regularization function to calculate the brightness-enhanced video frame corresponding to the brightness-enhanced video frame estimate in each group of optical imaging systems respectively; Calculating multiple groups of fidelity functions according to the similarity between the brightness-enhanced output video frame estimate and each group of brightness-enhanced video frames; Using the maximum a posteriori probability method to perform weighted fusion on multiple groups of fidelity functions to obtain a brightness-enhanced output video frame; Obtaining a brightness-enhanced output video stream according to the brightness-enhanced output video frame; Extracting key frame images of the brightness-enhanced output video stream during the material surface descending movement within the cloth cycle, and obtaining the relative position estimate of the material surface - endoscope through spatio-temporal matching of the key frame images; Back-calculating the spatial position of the virtual endoscope according to the relative position estimate of the material surface - endoscope; Constructing a virtual-real multi-camera array according to the spatial position of the virtual endoscope; Reconstructing the material surface in the industrial furnace based on the virtual-real multi-camera array to obtain a stereoscopic image of the blast furnace material surface.

2. The method for obtaining an image of a dim-light high-temperature industrial endoscope according to claim 1, characterized in that, The calculation formula for using the maximum a posteriori probability method to fuse the fidelity function, the consistency function, and the regularization function to calculate the brightness-enhanced video frame corresponding to the brightness-enhanced video frame estimate in each group of optical imaging systems respectively is: where i represents the group number of the optical imaging system, and respectively represent the estimated brightness-enhanced video frames of the i-th group of optical imaging systems at times t and t-1, Y it represents the brightness-enhanced video frame of the i-th group of optical imaging systems at time t, I ij represents the low-light image of the i-th group of optical imaging systems at time j, and respectively represent the fidelity function, the consistency function, and the regularization function, represents the posterior probability of, D and B respectively represent the blurring matrix and the downsampling matrix, represents the joint distribution variance, represents to I ij the motion compensation matrix, and where ρ is a scalar, represents and the motion compensation matrix, and respectively represent moving the image by l and h pixels in the horizontal and vertical directions respectively. Wide is the size of the moving window, r is the number of low-light image frames before time t of the original video stream used for the brightness-enhanced video frame estimation, and b is the number of low-light image frames after time t of the original video stream used for the brightness-enhanced video frame estimation.

3. The method for obtaining an image by using the dim-light high-temperature industrial stereo endoscope according to claim 2, wherein, The calculation formula for using the maximum a posteriori probability method to perform weighted fusion on multiple groups of fidelity functions to obtain a brightness-enhanced output video frame is: Among them, the number of groups of the optical imaging system is three, and respectively represent the posterior probabilities of the brightness-enhanced output video frame estimates of the first to third groups of optical imaging systems, O 1t 、O 2t and O 3t respectively represent the calculation results of the brightness-enhanced output video frames of the first to third groups of optical imaging systems, D and B respectively represent the blurring matrix and the downsampling matrix, and σ1, σ2, and σ3 are respectively the joint distribution standard deviations of the three groups of optical imaging systems, and are respectively to and 's motion compensation matrices, and are respectively to and 's motion compensation matrices, and are respectively to and 's motion compensation matrices, μ1, μ2, and μ3 are respectively user-defined synthesis weights, and all three synthesis weights are greater than 0 and less than 1.

4. The method for obtaining images of a faint-light high-temperature industrial endoscope according to any one of claims 1-3, characterized in that, Back-calculating the spatial position of the virtual endoscope according to the relative position estimate of the material surface - endoscope includes: Based on the relative position estimate of the material surface - endoscope, combined with spatial triangle transformation, back-calculating the spatial position of the virtual endoscope, where the calculation formula for the spatial position of the virtual endoscope is: Among them, is the coordinate of any pixel point P in the key frame image of the brightness enhancement output video stream output by the i-th group of optical imaging systems in the virtual endoscope coordinate system, is the coordinate of pixel point P in the world coordinate system, c i and w i are respectively the virtual endoscope coordinate system and the world coordinate system of the i-th group of optical imaging systems, K i and M i are respectively the internal parameter matrix and the external parameter matrix of the endoscope of the i-th group of optical imaging systems, is the true endoscope focal length of the i-th group of optical imaging systems, x is the width of the object in the key frame image, y is the height of the object in the key frame image, s i is the tilt factor of the i-th group of optical imaging systems, (u i , v i ) is the intersection point of the true endoscope main axis of the i-th group of optical imaging systems and the plane of the key frame image, R i and T i are respectively the rotation transformation matrix and the translation transformation matrix from the world coordinate system to the endoscope coordinate system of the i-th group of optical imaging systems.

5. The method for taking images of a dim-light high-temperature industrial stereo endoscope according to claim 4, characterized in that, Constructing a virtual-real multi-camera array according to the spatial position of the virtual endoscope includes: Selecting virtual endoscopes with parallax information to construct a virtual endoscope array, and fusing the pose information of the real endoscope camera to construct a virtual-real multi-camera array.

6. The method for taking images of a dim-light high-temperature industrial stereo endoscope according to claim 5, characterized in that Reconstructing the material surface in the industrial furnace based on the virtual-real multi-camera array includes: Using the sliding window matching method, the parallax value of the pixel points of the key frame images in the brightness-enhanced output video stream output by any two optical imaging systems is calculated based on the Mahalanobis distance. The specific calculation formula is as follows: Among them, P1, P2, and P3 are respectively the parallax values between the key-frame images in the brightness-enhanced output video streams output by two different optical imaging systems at time t. P0 is the sum of the average parallax value between different optical imaging systems and the parallax value of the front and back key frames of the selected i-th group of optical imaging systems. W is the one-dimensional window width, d is the translation distance of the sliding window, m and n are respectively the column number and row number of the pixel points of the key-frame image, and V it is the key-frame image in the brightness-enhanced output video stream output by the i-th group of optical imaging systems at time t, and the value range of i is i ∈ {1, 2, 3}. s0 represents the sliding window range, and ∑ is the covariance matrix; Using the triangulation method to obtain the spatial position coordinates of the reconstructed points: Among them, (X, Y, Z) are the spatial position coordinates of the reconstructed point. The coordinate origin of the coordinate system where the spatial position coordinates of the reconstructed point are located is the virtual endoscope center corresponding to the reconstructed key-frame image, and the reconstructed key-frame image is the key-frame image in the brightness-enhanced output video stream output by the optical imaging system corresponding to the reconstructed point, d L is the distance between the virtual and real endoscopes of adjacent optical imaging systems, X L and Y L are respectively the horizontal distance and the vertical distance from the pixel point corresponding to the reconstructed point in the reconstructed key-frame image to the center of the reconstructed key-frame image, and F is the distance from the real endoscope to the CCD imaging chip in the optical imaging system corresponding to the reconstructed point; Based on the spatial position coordinates of the reconstructed points, the surface of the charge in the industrial furnace is reconstructed.

7. The method for obtaining images of the dim-light high-temperature industrial stereo endoscope according to claim 6, characterized in that, The optical imaging system includes an objective lens group, an optical relay system lens group, and a zoom lens group connected in sequence, where: The objective lens group adopts a retrofocus objective lens structure for large-field imaging; The optical relay system lens group is used to transmit the image obtained by the objective lens group to the photosensitive chip without loss; The zoom lens group is used to achieve autofocus during the shooting process.

8. A dim-light high-temperature industrial stereoscopic endoscope imaging system, the system includes: A memory (10), a processor (20), and a computer program stored on the memory (10) and executable on the processor (20), wherein the processor (20) implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

Images