Vision-based fully mechanized coal mining face target positioning method

Abstract

The invention discloses a fully mechanized coal mining face target positioning method based on vision, and the method comprises the following steps: 1, calling two cameras adjacent to a fully mechanized coal mining face target object to obtain a monitoring image, and determining the center coordinates of the target object in the image; 2, the coordinates of the target object are accurately corrected, and the corrected center coordinates of the target object in the monitoring image are obtained; step 3, calculating a horizontal deviation angle of the target in the monitoring image; step 4, based on a binocular ranging principle, measuring and calculating the distance between the target object and the camera device to obtain positioning data of the target object; and step 5, updating the rack number of the adjacent camera at the next moment. According to the method, the horizontal distance between the target object and the camera adjacent to the target object is accurately calculated on the premise that the camera mounting rack number position is known through the target recognition technology and the specific fully-mechanized coal mining face image correction technology, and efficient target positioning of the fully-mechanized coal mining face is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a vision-based fully mechanized mining face target positioning method. Background Art

[0002] At present, with the continuous improvement of the automation level of coal mine comprehensive mining, more and more attention has been paid to the safety status of underground production equipment and operators in coal mines. Accurate positioning of underground production equipment and operators in coal mines has become the key to ensuring the safe production of comprehensive mining working faces.

[0003] Due to the special environment of the fully-mechanized mining face, it is impossible to receive satellite signals such as GPS, so direct precise positioning faces challenges. Although the existing ultra-wideband (UWB) positioning technology can carry positioning devices on the workers and deploy positioning base stations along the working face, the base stations are often installed in the support equipment, resulting in serious signal absorption, affecting the accuracy of ultra-wideband positioning. In addition, changes in temperature and humidity on the working face will also interfere with positioning accuracy. The existing infrared positioning technology requires the installation of an infrared transmitter on the coal mining machine and a receiver on the hydraulic support. However, the vibration of the coal mining machine during operation often causes unstable infrared signal reception, thus affecting the positioning effect.

[0004] In the context of intelligent construction of coal mines, most intelligent or automated fully-mechanized mining faces have been equipped with video monitoring systems for remote monitoring and shooting and viewing of working videos. This hardware configuration provides a basis for the implementation of the vision-based fully-mechanized mining face target positioning method. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for target positioning in a fully mechanized mining face based on vision. The present invention uses target recognition technology combined with a specific fully mechanized mining face image correction technology to accurately calculate the horizontal distance between the target object and its adjacent camera on the premise that the camera installation frame number position is known, thereby achieving efficient target positioning in the fully mechanized mining face.

[0006] The object of the present invention is achieved through the following technical scheme: a method for positioning a target in a fully mechanized mining face based on vision, comprising the following steps:

[0007] Step 1: Call two cameras near the target object of the fully mechanized mining working face to obtain their monitoring images respectively, and use the target recognition algorithm based on deep neural network to determine its center coordinates in each image;

[0008] Step 2: Obtain the focal length and distortion coefficient of the camera through camera calibration, apply specific fully mechanized mining working face image correction technology, accurately correct the coordinates of the target object, and obtain the corrected center coordinates of the target object in the monitoring image;

[0009] Step 3: Calculate the horizontal offset angle of the target in the monitoring image by combining the focal length information of the camera;

[0010] Step 4: Based on the binocular distance measurement principle and in combination with the known installation position of the neighboring camera, the distance between the target object and the camera device is calculated to obtain the positioning data of the target object;

[0011] Step 5: After obtaining the current positioning data of the target object, the frame number of the neighboring camera at the next moment is updated.

[0012] The beneficial effects of the present invention are as follows: the present invention innovatively proposes a vision-based fully mechanized mining face target positioning algorithm, which can achieve accurate target positioning of fully mechanized mining faces by relying only on the existing fully mechanized mining face video monitoring system without adding additional hardware facilities. The advantages include the following:

[0013] (1) The present invention uses target recognition technology to calculate the horizontal offset angle of the target object relative to the camera. Combined with the installation position information of the two cameras and based on the binocular ranging principle, the position of the target object is accurately calculated, thereby achieving high-precision positioning of the target object.

[0014] (2) When processing camera image distortion, the present invention not only corrects radial distortion and tangential distortion, but also implements environmental distortion correction for specific challenges in the complex environment of a fully mechanized mining face, such as coal dust, water mist, and special lighting conditions. Therefore, the obtained coordinate position of the target object is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the method for positioning a target in a fully mechanized mining face based on vision of the present invention;

[0016] Figure 2 Schematic diagram of camera correction angle;

[0017] Figure 3 The geometric diagram of the target object appearing between the two cameras;

[0018] Figure 4 Schematic diagram of the geometry when the target appears on the same side of the two cameras. DETAILED DESCRIPTION

[0019] The fully mechanized mining face is a long tunnel, mainly composed of 200 hydraulic supports arranged side by side. The targets for horizontal movement in it are mainly coal mining machines and workers. Horizontal positioning is mainly to obtain the support number of the coal mining machine drum or the worker. For example, the position obtained is between supports 47-48, 33 cm away from support 47 (the width of the support itself is generally 1-1.5 meters). The camera is installed every few frames on the fully mechanized mining face. The frame number of the camera is the frame number where the camera is installed. The current position of the camera is also determined based on this frame number.

[0020] The coordinates described in this embodiment refer to the coordinates in the rectangular coordinate system established with the upper left corner of the monitoring image as the origin. The origin of the coordinate system can also be selected from other positions as long as the same coordinate system is used in the parameter calculation process. The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0021] like Figure 1 As shown, a vision-based fully mechanized mining face target positioning method of the present invention comprises the following steps:

[0022] Step 1: Call two cameras near the target object of the fully mechanized mining working face to obtain their monitoring images respectively, and use the target recognition algorithm based on deep neural network to determine its center coordinates in each image;

[0023] When performing target positioning for the first time, due to the unknown camera frame numbers of the target objects, all cameras in the fully mechanized mining face are patrolled first to collect their monitoring images in turn. Using the target recognition algorithm based on deep neural networks, it is possible to identify the cameras whose monitoring images each target object may appear at the current moment, and record the corresponding camera frame numbers;

[0024] Considering the intervals between the cameras, the same target object may appear in the monitoring range of multiple cameras at the same time. For this reason, the two cameras in the middle of the cameras where the target object appears are selected as its neighboring cameras. If the target object appears in the monitoring images of an odd number of cameras, the camera in the middle and its adjacent camera (left or right) are selected as neighboring cameras (given the actual layout of the cameras in the fully mechanized mining face, the same target object will appear in the monitoring images of at least two cameras).

[0025] For the neighboring camera monitoring image of the target object, we take the center coordinate T of the target object bounding box obtained in the target recognition algorithm as the exact position coordinate of the target object in the monitoring image.

[0026] Step 2: Obtain the focal length and distortion coefficient of the camera through camera calibration, apply specific fully mechanized mining face image correction technology, accurately correct the coordinates of the target object, and obtain the corrected center coordinates of the target object in the monitoring image; the specific implementation method is:

[0027] The camera of the comprehensive mining face uses a 180° wide-angle lens, so its imaging has a large distortion. Through the camera calibration process, we obtained the focal length of the camera and the radial and tangential distortion coefficients. Since the cameras used in a single comprehensive mining face are of the same model and from the same production batch, the focal lengths and radial and tangential distortion coefficients of these cameras are consistent;

[0028] The shape of the camera lens does not completely conform to the ideal spherical shape, resulting in inconsistent magnification of the center and edge of the image, and radial distortion x rd and rd :

[0029]

[0030] where k 1 , k 2 , k 3 is the radial distortion coefficient; T(x t ,y t ) is the coordinate position of the ideal undistorted target object in the monitoring image, obtained from step 1; O(x 0 ,y 0 ) is the center coordinate position of the monitoring image (in a rectangular coordinate system with the upper left corner of the monitoring image as the origin), d e is the Euclidean distance between T and O, calculated as:

[0031]

[0032] The non-orthogonality between the lens and the imaging plane results in tangential distortion x td and td :

[0033]

[0034] where p 1 、p 2 is the tangential distortion parameter;

[0035] The underground fully mechanized mining working surface is dusty, and the light conditions are affected by the installation position of the light source, which leads to the camera optical system having environmental distortion caused by the environment in addition to radial distortion and tangential distortion caused by the lens during imaging. ed and ed :

[0036]

[0037] where q 1 ,q 2 is the environmental distortion parameter, d c is the Canberra distance between T and O:

[0038]

[0039] Environmental distortion parameter q 1 ,q 2 Affected by the dust concentration and light conditions of the underground fully-mechanized mining working face, the dust concentration of the underground fully-mechanized mining working face is not the same everywhere, and the position of each camera from the light source is also different. Therefore, the environmental distortion parameters of each camera are slightly different. According to the actual situation on the site of the underground fully-mechanized mining working face, q 1 The value range of q is between 0.59 and 0.81. 2 The value range is between 0.17 and 0.33.

[0040] By correcting the distortion of the image, the corrected coordinate center position x of the target object T in the image is obtained. ct and ct :

[0041]

[0042] Step 3: Combine the focal length information of the camera to calculate the horizontal offset angle of the target in the monitoring image; the imaging of the target object in the camera monitoring screen is as follows: Figure 2 As shown, let O be the optical center of the camera, T be the target object, and take the upper left corner of the image as the coordinate origin, O'(x O' ,y O' ) is the center coordinate position of the monitoring image, T'(x ct' ,y ct' ) is the coordinate center position of the target object in the monitoring screen, and the distance between OO' is the focal length f of the camera;

[0043] The horizontal offset angle of the target object in the surveillance image is obtained as:

[0044]

[0045] When α<0, it means that the target object is located on the left side of the bracket number where the current camera is located; when α>0, it means that the target object is located on the right side of the bracket number where the current camera is located; when α=0, it means that the target object is located on the bracket number where the current camera is located, thus obtaining the position of the target object.

[0046] Step 4: Based on the binocular distance measurement principle and in combination with the known installation position of the neighboring camera, the distance between the target object and the camera device is calculated to obtain the positioning data of the target object;

[0047] The specific implementation method of step 4 is: when the horizontal offset angle α of the target object in the monitoring image of its neighboring camera l and α r When the product is less than 0 (i.e. α l ×α r <0), indicating that the horizontal offset angles of the target object relative to the center of the image in the monitoring image of the called camera are opposite (one positive and one negative). At this time, the target object is located between the called cameras, and its imaging situation is as follows Figure 3 As shown in the figure, O l , O r are the optical center positions of the two cameras, O l ′、O r ′ are the center coordinates of the monitoring images of the two cameras, T is the actual plane position of the target object, T l 、T r are the projection positions of the two cameras on the plane where the target object is located, T 0 is the projection position of the target object T on the plane where the camera is located, α l ∠T l 'O l O l ′,α r ∠T r 'O r O r ′.

[0048] According to their geometric relationship,

[0049] ∠T l 'TT O =∠T l 'O l O l '=α l

[0050] ∠T r 'TT O =∠T r 'O r O r '=α r

[0051] Available,

[0052] TT O tanα l =T O O l

[0053] TT O tanα r =T O O r

[0054] because,

[0055] T O O l +T O O r =O l O r

[0056] get:

[0057] TT O tanα l +TT O tanα r =O l O r

[0058] Further we get:

[0059]

[0060] When the target object is horizontally offset by an angle α in the monitoring image of its neighboring camera l and α r When the product is greater than 0 (i.e. α l ×α r >0), it means that the horizontal offset angle of the target object relative to the center of the image in the monitored image of the called camera is consistent (all positive or all negative). At this time, the target object is on the same side of the called camera, and its imaging situation is as follows Figure 4 As shown, according to its geometric relationship, there is

[0061] ∠T l 'TT O =∠T l 'O l O l '=α l

[0062] ∠T r 'TT O =∠T r 'O r O r '=α r

[0063] Available,

[0064] TT O tanα l =T O Ol

[0065] TT O tanα r =T O O r

[0066] because

[0067] |T O O l -T O O r |=O l O r

[0068] get:

[0069] |TT O tanα l -TT O tanα r |=O l O r

[0070] Further we get:

[0071]

[0072] According to the obtained TT O TT l and TT r The value of is combined with the installation position information of the two cameras to calculate and determine the precise position of the target object, thereby achieving precise positioning of the target object T.

[0073] Step 5: After obtaining the current positioning data of the target object, the frame number of the neighboring camera at the next moment is updated immediately; the specific operation steps are: according to the current positioning data of the target object, the frame number of the neighboring camera is re-determined, and the calculated frame number is designated as the frame number of the camera to be called at the next moment;

[0074]

[0075] Where o is the hydraulic support number of the target object at the current moment, o l and r The hydraulic support number of the camera to be called at the next moment, N(n 1 ,n 2 ……n k ) is the hydraulic support number where all cameras are located in the comprehensive mining working face.

[0076] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for fully mechanized mining face target positioning based on vision, characterized in that: The steps include: Step 1: Call two cameras near the target object of the fully mechanized mining working face to obtain their monitoring images respectively, and use the target recognition algorithm based on deep neural network to determine its center coordinates in each image; Step 2: Obtain the focal length and distortion coefficient of the camera through camera calibration, accurately correct the coordinates of the target object, and obtain the corrected center coordinates of the target object in the monitoring image; Step 3: Calculate the horizontal offset angle of the target in the monitoring image by combining the focal length information of the camera; Step 4: Based on the binocular distance measurement principle and in combination with the known installation position of the neighboring camera, the distance between the target object and the camera device is calculated to obtain the positioning data of the target object; Step 5: After obtaining the current positioning data of the target object, the frame number of the neighboring camera at the next moment is updated.

2. A method for target positioning in a fully mechanized mining face based on vision according to claim 1, characterized in that: When executing step 1 for the first time, the system will traverse all cameras on the comprehensive mining working face and collect their monitoring images in sequence; by applying the target recognition algorithm of the deep neural network, it will identify which cameras’ monitoring images the target object appears in at the current moment; then, the two middle cameras will be selected from the recognition results as neighboring cameras.

3. A method for target positioning in a fully mechanized mining face based on vision according to claim 1, characterized in that: The specific implementation method of step 2 is: since the cameras used in a single fully mechanized mining face are of the same model and come from the same production batch, the focal lengths and radial and tangential distortion coefficients of these cameras are consistent; The shape of the camera lens does not completely conform to the ideal spherical shape, resulting in inconsistent magnification of the center and edge of the image, and radial distortion x rd and rd : Where k1, k2, k3 are radial distortion coefficients; T(x t ,y t ) is the coordinate position of the ideal undistorted target object in the monitoring image, obtained by step 1; O(x0, y0) is the center coordinate position of the monitoring image, d e is the Euclidean distance between T and O, calculated as: The non-orthogonality between the lens and the imaging plane results in tangential distortion x td and td : Where p1 and p2 are tangential distortion parameters; In the process of imaging, the camera optical system has not only radial distortion and tangential distortion caused by the lens, but also environmental distortion caused by the environment. ed and ed : Where q1 and q2 are environmental distortion parameters, d c is the Canberra distance between T and O: By correcting the distortion of the image, the corrected coordinate center position x of the target object T in the image is obtained. ct and ct :

4. A method for target positioning in a fully mechanized mining face based on vision according to claim 1, characterized in that: The specific implementation method of step 3 is: let O be the optical center of the camera, T be the target object, and take the upper left corner of the image as the coordinate origin, O'(x O' ,y O' ) is the center coordinate position of the monitoring image, T'(x ct' ,y ct' ) is the coordinate center position of the target object in the monitoring screen, and the distance between OO' is the focal length f of the camera; The horizontal offset angle of the target object in the surveillance image is obtained as:

5. The method for target positioning in a fully mechanized mining face based on vision according to claim 1 is characterized in that: The specific implementation method of step 4 is: when the horizontal offset angle α of the target object in the monitoring image of its adjacent camera is l and α r When the product is less than 0, the target object is located between the called cameras; l , O r are the optical center positions of the two cameras, O l ′、O r ′ are the center coordinates of the monitoring images of the two cameras, T is the actual plane position of the target object, T l , T r are the projection positions of the two cameras on the plane where the target object is located, T0 is the projection position of the target object T on the plane where the camera is located, α l ∠T l 'O l O l ′,α r ∠T r 'O r O r ′; According to their geometric relationship, ∠T l 'TT O =∠T l 'O l O l '=α l ∠T r 'TT O =∠T r 'O r O r '=α r get TT O tanα l =T O O l TT O tanα r =T O O r because, T O O l +T O O r =O l O r get: TT O tanα l +TT O tanα r =O l O r Further we get: When the target object is horizontally offset by an angle α in the monitoring image of its neighboring camera l and α r When the product is greater than 0, the target object is located on the same side of the called camera; according to their geometric relationship, ∠T l 'TT O =∠T l 'O l O l '=α l ∠T r 'TT O =∠T r 'O r O r '=α r get TT O tanα l =T O O l TT O tanα r =T O O r because |T O O l -T O O r |=O l O r get: |TT O tanα l -TT O tanα r |=O l O r Further we get: According to the obtained TT O TT l and TT r The value of is combined with the installation position information of the two cameras to calculate and determine the precise position of the target object, thereby achieving precise positioning of the target object T.

6. A method for fully mechanized mining face target positioning based on vision according to claim 1, characterized in that: The specific implementation method of the step in step 5 is: based on the positioning data of the target object at the current moment, re-determine the frame number of the camera adjacent to it, and designate the calculated frame number as the frame number of the camera to be called at the next moment; Where o is the hydraulic support number of the target object at the current moment, o l and r The hydraulic support number of the camera to be called at the next moment, N (n1, n2...n k ) is the hydraulic support number where all cameras are located in the comprehensive mining working face.

Images