Blasting fragmentation identification method and device based on binocular vision and electronic equipment

Abstract

The invention discloses a blasting fragmentation identification method and device based on binocular vision and electronic equipment, and relates to the field of computer vision. The method comprises the following steps: acquiring a double-view-angle image of a rock fragment scene shot by a binocular camera; based on a triangulation principle, performing depth value extraction on the left view angle image and the right view angle image to obtain a corresponding depth image; the number of broken stones in each depth image is determined, a target depth image is determined according to the number of the broken stones, and the target depth image is the depth image with the largest number of the broken stones in the depth images; extracting a broken stone contour point set included in the target depth image; and calculating the target size of each rock fragment in the target depth image according to the broken stone contour point set. According to the technical scheme provided by the invention, the target size of each rock fragment is accurately identified by using a binocular vision technology in combination with instance segmentation and geometric calculation, and the blasting fragmentation identification efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, device and electronic equipment for identifying blasting block size based on binocular vision. Background Art

[0002] With the development of the global economy and the increasing demand for natural resources, mining is one of the main ways to obtain resources. The efficiency and effect of ore blasting directly affect the economy and sustainability of resource development. In the mining industry, blasting is a common means of rock crushing, especially in the process of mining ores and building materials such as sandstone and limestone. Correct blasting can not only maximize the extraction rate of resources, but also affect the efficiency and cost of subsequent safe production. At the same time, the method of automatic identification of blasting block size is expected to become an effective way to solve the problems of ore flow and potential geological disasters, and achieve a balance between safe production and environmental protection.

[0003] At present, most measurements of gravel block size rely on manual size judgment. Manual measurement requires a lot of time and manpower for on-site operations. Especially in large mines, this process is very time-consuming and has the problem of low efficiency.

[0004] Therefore, there is an urgent need for a method, device and electronic equipment for identifying blasting fragmentation based on binocular vision. Summary of the invention

[0005] The present application provides a method, device and electronic equipment for blasting fragmentation recognition based on binocular vision, which utilizes binocular vision technology, combined with instance segmentation and geometric calculation, to accurately identify the target size of each rock fragment, thereby improving the efficiency of blasting fragmentation recognition.

[0006] In a first aspect of the present application, a method for identifying blasting fragmentation based on binocular vision is provided. In the method, a dual-perspective image of a rock fragment scene taken by a binocular camera is obtained, the binocular camera includes two left and right sub-cameras with overlapping fields of view, and the dual-perspective image includes a left-perspective image and a right-perspective image; based on the principle of triangulation, depth values ​​of the left-perspective image and the right-perspective image are extracted to obtain corresponding depth images; the number of gravels in each of the depth images is determined, and a target depth image is determined based on the number of gravels, the target depth image being the depth image with the largest number of gravels in the depth image; a gravel contour point set included in the target depth image is extracted; and the target size of each rock fragment in the target depth image is calculated based on the gravel contour point set.

[0007] By adopting the above technical solution, by obtaining the dual-view images of the rock fragment scene taken by the binocular camera, the depth value of the left and right view images is extracted using the triangulation principle to obtain the corresponding depth image. Then the number of gravels in each depth image is determined, the depth image with the largest number of gravels is selected as the target depth image, the gravel contour point set is extracted, and the target size of each rock fragment is calculated based on the gravel contour point set. The depth information of the scene is obtained by binocular vision technology, which overcomes the limitation that monocular vision is difficult to estimate the true size of the object. By selecting the depth image with the largest number of gravels for analysis, more comprehensive and representative blasting fragment size information can be obtained. The steps of extracting the gravel contour point set and calculating the target size realize the quantitative evaluation of the rock fragment size and provide data support for the evaluation of the blasting effect. The entire method process has a high degree of automation, which avoids the tediousness of manual measurement and improves the efficiency and accuracy of blasting fragment size recognition.

[0008] Optionally, based on the triangulation principle, depth values ​​of the left-view image and the right-view image are extracted to obtain a corresponding depth image, which specifically includes: matching the dual-view images to identify corresponding gravel areas in the left-view image and the right-view image; triangulating the gravel area according to the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera to calculate the depth value of each pixel point in the gravel area; and fusing the depth value of each pixel point with the corresponding color image to obtain the depth image.

[0009] By adopting the above technical solution, the left-view image and the right-view image are matched, and the corresponding gravel areas in the left-view image and the right-view image are identified, ensuring the accuracy of the subsequent depth calculation. Then, the intrinsic and extrinsic matrix of the binocular camera is used to triangulate the gravel area and calculate the depth value of each pixel. Finally, the depth value is fused with the color image to obtain a depth image. This technical solution makes full use of the geometric constraints of the binocular vision system. Through the matching and triangulation of the left-view image and the right-view image, the depth information in the rock fragment scene is obtained. The fusion of depth information with the color image can intuitively show the spatial distribution and relative position relationship of rock fragments, which provides a basis for subsequent gravel extraction and size estimation. This method can accurately and efficiently obtain the three-dimensional information of the blasting block size.

[0010] Optionally, matching the dual-view images to identify corresponding gravel areas in the left-view image and the right-view image specifically includes: performing feature extraction on the left-view image and the right-view image to obtain a first feature point set and a second feature point set, the first feature point set corresponding to the left-view image, and the second feature point set corresponding to the right-view image; performing feature point matching on the first feature point set and the second feature point set to obtain matching feature point pairs; determining matching rectangular neighborhood windows of the left-view image and the right-view image with each pair of the matching feature point pairs as the center; determining candidate gravel areas based on the normalization coefficient of the matching rectangular neighborhood windows, and performing a morphological closing operation on the candidate gravel areas to obtain the gravel areas.

[0011] By adopting the above technical solution, firstly, the left view image and the right view image are feature extracted to obtain two sets of feature point sets. Then, the two sets of feature point sets are matched to find the corresponding matching feature point pairs. With the matching feature point pairs as the center, the matching rectangular neighborhood windows of the left view image and the right view image are determined, and the candidate gravel area is determined according to the normalized coefficient of the matching rectangular neighborhood windows. Finally, the accurate gravel area is obtained through morphological closing operation. This technical solution utilizes the image feature matching technology, and establishes the corresponding relationship between the two perspectives by extracting and matching the feature points of the left view image and the right view image. With the matching feature point pairs as the center, the rectangular neighborhood window is determined, and gravel extraction can be performed in the local area, reducing the influence of background interference. The morphological closing operation further optimizes the boundary of the gravel area, making the extraction result more accurate and complete. This technical solution realizes the automatic and accurate identification of the gravel area in the dual-view image, laying the foundation for the subsequent depth calculation and size estimation.

[0012] Optionally, triangulating the gravel area according to the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera to calculate the depth value of each pixel point in the gravel area specifically includes: calculating the basic matrix between the left-view image and the right-view image according to the intrinsic parameter matrix and the extrinsic parameter matrix; matching the gravel area based on the basic matrix to obtain matching pixel point pairs, and calculating disparity values ​​according to the coordinates of the matching pixel point pairs; and calculating the depth value of each pixel point by combining the disparity value, the baseline distance of the binocular camera, and the focal length of the binocular camera using the triangulation principle.

[0013] By adopting the above technical solution, firstly, the basic matrix between the left-view image and the right-view image is calculated according to the intrinsic parameter matrix and the extrinsic parameter matrix. The basic matrix contains the geometric constraint relationship of the binocular vision system and is the key parameter for triangulation. Then, the gravel area is matched based on the basic matrix to obtain the matching pixel pairs and calculate their disparity values. The disparity value reflects the horizontal offset of the matching pixel points in the left and right view images, which is related to the distance of the object from the camera. Finally, the depth value of each pixel is calculated by the triangulation principle based on the disparity value, the baseline distance and the focal length of the binocular camera. Through the constraints of the basic matrix and the calculation of the disparity value, the depth information of each pixel in the gravel area is accurately estimated, providing reliable data support for the subsequent blasting block size analysis.

[0014] Optionally, before determining the number of gravels in each of the depth images, the method further includes: inputting the depth image into a pre-trained instance segmentation model; generating instance masks for each of the rock fragments through the instance segmentation model, where different rock fragments correspond to instance masks of different colors; and fusing each of the instance masks with the depth image to obtain segmented rock fragment areas and background areas.

[0015] By adopting the above technical solution, the depth image is input into the pre-trained instance segmentation model to generate an instance mask for each rock fragment. Different fragments correspond to instance masks of different colors. Finally, the instance mask is fused with the depth image to obtain the segmented rock fragment area and background area. The generated instance mask is fused with the depth image to realize the automatic segmentation and extraction of the rock fragment area, providing an accurate target area for subsequent gravel quantity statistics and size estimation. This technical solution combines depth images and semantic segmentation information to achieve accurate positioning and extraction of rock fragments, and improves the intelligent level of blasting block size recognition.

[0016] Optionally, determining the number of gravel in each of the depth images specifically includes: performing a connected domain analysis on the rock fragment area to obtain a plurality of rock fragments included in the depth image; counting the number of the rock fragments in each of the depth images, and using the number of rock fragments as the number of gravel.

[0017] By adopting the above technical solution, each independent rock fragment in the depth image can be identified by performing connected domain analysis on the segmented rock fragment area. Then the number of rock fragments in each depth image is counted and used as the number of gravels in the depth image, realizing the automatic counting of the number of gravels. By comparing the number of gravels in different depth images, the depth image with the largest number of gravels is selected as the target depth image for subsequent analysis, which improves the representativeness and comprehensiveness of the blasting block size assessment.

[0018] Optionally, calculating the target size of each rock fragment in the target depth image based on the gravel contour point set specifically includes: traversing the target points of the gravel contour point set of each rock fragment, and taking the target point as a candidate center of a circle, the target point being any point in the gravel contour point set; determining a target center of a circle among the candidate centers of a circle, the target center of a circle being a candidate center of a circle that can generate a circle with a maximum radius with the boundary points of the gravel contour point set; calculating the distance from each boundary point to the target center of a circle, and sorting the distances from small to large to obtain the shortest distance; taking the shortest distance as the maximum circumscribed circle radius of the rock fragment; determining a maximum circumscribed circle diameter based on the maximum circumscribed circle radius, and taking the maximum circumscribed circle diameter as the target size.

[0019] By adopting the above technical solution, the gravel contour point set of each rock fragment is first traversed, and each point is used as a candidate circle center. Then, among all the candidate circle centers, the candidate circle center that can generate the maximum radius circle with the boundary point of the gravel contour point set is selected as the target circle center. Then, the distance from the boundary point to the target circle center is calculated, and sorted from small to large, and the shortest distance is taken as the maximum circumscribed circle radius of the rock fragment. The shortest distance ensures that the circle is as large as possible without exceeding the contour boundary of the rock fragment. Finally, based on the maximum circumscribed circle radius, the maximum circumscribed circle diameter is determined and used as the target size of the rock fragment. This technical solution cleverly uses the idea of ​​computational geometry and estimates the true size of the rock fragment by fitting the maximum circumscribed circle. By traversing all possible circle centers and selecting the optimal target circle center, the reliability of the estimation result is guaranteed, providing a quantitative size index for blasting block size analysis.

[0020] In a second aspect of the present application, a blasting block size recognition device based on binocular vision is provided, the device comprising: In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0021] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0022] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By obtaining the dual-view images of the rock fragment scene taken by the binocular camera, the depth values ​​of the left and right view images are extracted using the triangulation principle to obtain the corresponding depth images. Then the number of gravels in each depth image is determined, the depth image with the largest number of gravels is selected as the target depth image, the gravel contour point set is extracted, and the target size of each rock fragment is calculated based on the gravel contour point set. The use of binocular vision technology to obtain the depth information of the scene overcomes the limitation that monocular vision is difficult to estimate the true size of the object. By selecting the depth image with the largest number of gravels for analysis, more comprehensive and representative blasting fragment size information can be obtained. The steps of extracting the gravel contour point set and calculating the target size realize the quantitative evaluation of the rock fragment size and provide data support for the evaluation of the blasting effect. The entire method process has a high degree of automation, which avoids the tediousness of manual measurement and improves the efficiency and accuracy of blasting fragment size recognition.

[0023] 2. Match the left-view image and the right-view image to identify the corresponding gravel area in the left-view image and the right-view image, ensuring the accuracy of the subsequent depth calculation. Then use the intrinsic and extrinsic matrix of the binocular camera to triangulate the gravel area and calculate the depth value of each pixel. Finally, the depth value is fused with the color image to obtain a depth image. This technical solution makes full use of the geometric constraints of the binocular vision system. Through the matching and triangulation of the left-view image and the right-view image, the depth information in the rock fragment scene is obtained. Fusion of depth information with color images can intuitively show the spatial distribution and relative position relationship of rock fragments, providing a basis for subsequent gravel extraction and size estimation. This method can accurately and efficiently obtain three-dimensional information of blasting block size.

[0024] 3. First, extract features from the left-view image and the right-view image to obtain two sets of feature points. Then match the two sets of feature points to find the corresponding matching feature point pairs. With the matching feature point pairs as the center, determine the matching rectangular neighborhood windows of the left-view image and the right-view image, and determine the candidate gravel area according to the normalized coefficient of the matching rectangular neighborhood windows. Finally, the precise gravel area is obtained through morphological closing operations. This technical solution utilizes the image feature matching technology to establish the correspondence between the two perspectives by extracting and matching the feature points of the left-view image and the right-view image. With the matching feature point pairs as the center, determine the rectangular neighborhood window, and gravel extraction can be performed in the local area, reducing the influence of background interference. The morphological closing operation further optimizes the boundary of the gravel area, making the extraction result more accurate and complete. This technical solution realizes the automatic and accurate identification of gravel areas in dual-view images, laying the foundation for subsequent depth calculation and size estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1It is a flowchart of a method for identifying blasting block size based on binocular vision disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a blasting block size recognition device based on binocular vision disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0026] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0027] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0028] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0029] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0030] This application provides a method for identifying blasting fragmentation based on binocular vision. Figure 1 , Figure 1 1 is a flow chart of a method for identifying blasting block size based on binocular vision provided in an embodiment of the present application. The method is applied to a server and includes steps S101 to S105, which are as follows: Step S101: acquiring a dual-perspective image of a rock fragment scene captured by a binocular camera, wherein the binocular camera includes two left and right sub-cameras with overlapping fields of view, and the dual-perspective image includes a left-perspective image and a right-perspective image.

[0031] In step S101, the binocular camera is arranged and debugged at the blasting site. The binocular camera consists of two left and right sub-cameras with overlapping fields of view. The optical axes of the two sub-cameras are parallel, and the camera parameters (such as focal length, principal point coordinates, etc.) have been obtained through calibration. When the blasting operation is completed, the server sends a shooting command to the binocular camera. After the binocular camera receives the shooting command, the left and right sub-cameras synchronously shoot the rock fragment scene to obtain a left-view image and a right-view image. The left-view image and the right-view image constitute a dual-view image, which contains three-dimensional information of the rock fragment scene.

[0032] Next, the binocular camera transmits the captured left-view image and right-view image to the server. The server receives the left-view image and right-view image transmitted by the binocular camera, and completes the acquisition of the binocular image.

[0033] Step S102: Based on the triangulation principle, extract the depth values ​​of the left-view image and the right-view image to obtain a corresponding depth image.

[0034] In step S102, the dual-view images are matched to identify the corresponding gravel areas in the left-view image and the right-view image; the gravel area is triangulated according to the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera to calculate the depth value of each pixel in the gravel area; the depth value of each pixel is fused with the corresponding color image to obtain a depth image.

[0035] Specifically, the server matches the dual-view images and identifies the corresponding gravel areas in the left-view image and the right-view image. This step uses feature point matching algorithms, such as SIFT and SURF. Specifically, the server first extracts feature points from the left and right view images respectively, and then finds the corresponding feature point pairs in the left and right view images by comparing the descriptors of the feature points. Since the gravel area usually has obvious texture and edge features, the feature points are mostly distributed in the gravel area. By clustering and filtering the feature points, the server can identify the corresponding gravel area in the left and right view images.

[0036] Next, the server triangulates the gravel area based on the intrinsic and extrinsic matrix of the binocular camera and calculates the depth value of each pixel in the gravel area. The intrinsic matrix describes the internal parameters of the camera, such as focal length and principal point coordinates; the extrinsic matrix describes the relative position and posture relationship between the left and right cameras, which can be obtained through binocular camera calibration. Using the principle of triangulation, the server can calculate the disparity of each pixel in the gravel area in the left and right perspective images, and then obtain its depth value. Specifically, for each pixel in the gravel area, the server searches for its matching point in the other perspective image, calculates the disparity between the two matching points, and then calculates the depth value of the pixel based on the disparity, baseline length, and focal length parameters.

[0037] After calculating the depth value of each pixel, the server fuses the depth value of each pixel with the corresponding color image to obtain a depth image. The grayscale value of each pixel in the depth image represents the depth value of the pixel. The server first creates a blank image of the same size as the left-view image, then traverses each pixel in the gravel area, maps its depth value to a grayscale range of 0-255, and fills it in the corresponding position. For pixels without depth values ​​(such as background areas), the server can assign a default value (such as 0 or 255). Finally, the server can perform post-processing on the depth image, such as median filtering and bilateral filtering, to smooth the depth values ​​and eliminate outliers caused by noise and mismatching.

[0038] For example, suppose the server obtains a dual-view image after a blasting operation and needs to generate a corresponding depth image. The server first uses the SIFT algorithm to extract feature points in the left and right view images, and then finds the corresponding feature point pairs in the left and right view images by comparing the descriptors of the feature points. By clustering and filtering the feature points, the server identifies the corresponding gravel areas in the left and right view images. Next, the server triangulates each pixel in the gravel area based on the internal and external parameters of the binocular camera and calculates its depth value. Finally, the server creates a blank image of the same size as the left view image, maps the depth value of each pixel in the gravel area to the grayscale range of 0-255, fills it in the corresponding position, and obtains the depth image. After median filtering and smoothing, the gravel area in the depth image shows continuous grayscale changes, indicating the three-dimensional structure and height difference of the gravel pile.

[0039] Through the above steps, the server completes the depth value extraction of the dual-view image and obtains the corresponding depth image.

[0040] In a possible implementation, dual-view images are matched to identify corresponding gravel areas in the left-view image and the right-view image, specifically including: extracting features from the left-view image and the right-view image to obtain a first feature point set and a second feature point set, the first feature point set corresponding to the left-view image, and the second feature point set corresponding to the right-view image; performing feature point matching on the first feature point set and the second feature point set to obtain matching feature point pairs; determining matching rectangular neighborhood windows of the left-view image and the right-view image with each pair of matching feature point pairs as the center; determining candidate gravel areas based on normalized coefficients of the matching rectangular neighborhood windows, and performing morphological closing operations on the candidate gravel areas to obtain gravel areas.

[0041] Specifically, the server preprocesses the left-view image and the right-view image respectively, including grayscale, smoothing filtering and histogram equalization; in the preprocessed left-view image and right-view image, the SIFT (Scale-Invariant Feature Transform) algorithm is used to extract feature points to obtain a first feature point set for the left-view image and a second feature point set for the right-view image, each feature point including position coordinates and scale information; then, the server calculates the SIFT feature descriptor of each feature point in the first feature point set to obtain a first feature descriptor set corresponding to the left-view image; similarly, the server calculates the SIFT feature descriptor of each feature point in the second feature point set to obtain a second feature descriptor set corresponding to the right-view image; wherein the SIFT feature descriptor is a 128-dimensional vector obtained based on the gradient statistics of the pixels in the neighborhood of the feature point; Based on the above-mentioned first feature descriptor set and second feature descriptor set, the server uses the nearest neighbor distance ratio matching method to match feature points to obtain matching feature point pairs of the left-view image and the right-view image; specifically, for each feature point of the left-view image, the server finds the two feature points with the closest Euclidean distance among the feature points of the right-view image. If the ratio of the closest distance to the second closest distance is less than the preset threshold, the feature point pair is determined as a matching feature point pair; based on the matching feature point pairs between the left-view image and the right-view image, the server uses the RANSAC (Random Sample Consensus) algorithm to estimate the homography matrix between the left-view image and the right-view image, eliminates the wrong matching point pairs, and obtains accurate matching feature point pairs; the preset threshold can be set according to the specific application scenario and image characteristics, and a suitable threshold range can usually be determined by empirical values ​​or multiple tests. For example, it can be selected between 0.6 and 0.8, which can not only ensure a sufficient number of matching point pairs, but also effectively eliminate most of the wrong matches. This application does not limit the specific value of the preset threshold, and it can be flexibly set according to actual needs.

[0042] The server identifies the corresponding gravel areas in the left-view image and the right-view image based on the precise matching feature point pairs. Specifically, the server determines the rectangular neighborhood windows in the left-view image and the right-view image respectively with each pair of matching feature points as the center, and calculates the normalized correlation coefficient of each pair of matching rectangular neighborhood windows. If the normalized correlation coefficient is greater than the preset coefficient threshold, the pair of rectangular neighborhood windows is determined as a candidate gravel area; then, the server performs a morphological closing operation on the candidate gravel area, connects adjacent gravel areas, and obtains the gravel areas in the left-view image and the right-view image. The selection of the preset coefficient threshold is used to balance the integrity and accuracy of the gravel area, and can be set according to the texture characteristics and imaging quality of the rock fragments. For example, for gravel with clear texture and obvious edges, a higher preset coefficient threshold can be selected, such as 0.8-0.9; for gravel with fuzzy texture and unclear edges, a lower preset coefficient threshold can be selected, such as 0.6-0.7. This application does not limit the specific value of the preset coefficient threshold, and it can be flexibly set according to actual needs.

[0043] In a possible implementation, the gravel area is triangulated according to the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera, and the depth value of each pixel in the gravel area is calculated, specifically including: calculating the basic matrix between the left-view image and the right-view image according to the intrinsic parameter matrix and the extrinsic parameter matrix; matching the gravel area based on the basic matrix to obtain matching pixel pairs, and calculating the disparity value according to the coordinates of the matching pixel pairs; and calculating the depth value of each pixel by using the triangulation principle based on the comprehensive disparity value, the baseline distance of the binocular camera, and the focal length of the binocular camera.

[0044] Specifically, the server calculates the basic matrix between the left-view image and the right-view image based on the intrinsic parameter matrix and the extrinsic parameter matrix. The intrinsic parameter matrix includes the internal parameters of the camera, such as focal length and principal point coordinates; the extrinsic parameter matrix includes the relative position and posture relationship between the left and right cameras. The basic matrix is ​​a 3×3 matrix that includes the geometric information of the binocular camera system and is used to describe the epipolar constraint relationship between the left and right view images. Preferably, in an embodiment of the present application, the server uses Zhang's calibration method to estimate the intrinsic parameter matrix and the extrinsic parameter matrix based on multiple calibration images taken by the binocular camera, and then calculates the basic matrix according to the formula.

[0045] Next, the server matches the gravel area based on the basic matrix to obtain matching pixel pairs, and calculates the disparity value according to the coordinates of the matching pixel pairs. Specifically, the server first selects a pixel in the gravel area of ​​the left-view image, calculates the corresponding epipolar line of the pixel in the right-view image according to the basic matrix, and searches for the pixel closest to the epipolar line in the gravel area of ​​the right-view image as the matching pixel. The server repeats the above matching process for all pixels in the gravel area to obtain a set of matching pixel pairs. For each matching pixel pair, the server calculates the coordinate difference between the left-view image and the right-view image, that is, the disparity value. Finally, the server calculates the depth value of each pixel by using the triangulation principle based on the disparity value, the baseline distance of the binocular camera and the focal length of the binocular camera. Triangulation is a method of measuring the distance of a target point using the geometric relationship of a triangle. In the binocular vision system provided in the embodiment of the present application, the optical center of the left and right cameras and the target point form a triangle. Knowing the baseline distance (the distance between the optical centers of the left and right cameras), the focal length and the disparity value, the depth value of the target point can be calculated according to the principle of similar triangles. The server calculates the corresponding depth value for each matching pixel pair using its disparity value, baseline distance and focal length. In this way, each pixel in the gravel area gets a depth value, forming a depth map.

[0046] For example, the server obtains a binocular image after a blasting operation. The intrinsic and extrinsic matrix of the binocular camera is known, and the depth value of each pixel in the gravel area needs to be calculated. The server first calculates the basic matrix between the left and right view images based on the intrinsic and extrinsic matrix. Then, the server selects a pixel point A in the gravel area of ​​the left view image, and calculates the corresponding epipolar line L of the pixel point A in the right view image based on the basic matrix. The server searches for the pixel point B closest to the epipolar line L in the gravel area of ​​the right view image, and obtains the matching pixel point pair (A, B). The server calculates the coordinate difference between the pixel points A and B in the left and right view images to obtain the disparity value d. It is known that the baseline distance of the binocular camera is b and the focal length is f. According to the principle of similar triangles, the depth value of pixel point A is Z=bf / d. The server repeats the above process for all pixels in the gravel area to obtain a depth map.

[0047] Step S103: determining the number of gravels in each depth image, and determining a target depth image according to the number of gravels, wherein the target depth image is a depth image having the largest number of gravels among the depth images.

[0048] In one possible implementation, before step S103, the method further includes: inputting the depth image into a pre-trained instance segmentation model; generating instance masks for each rock fragment through the instance segmentation model, where different rock fragments correspond to instance masks of different colors; and fusing each instance mask with the depth image to obtain segmented rock fragment areas and background areas.

[0049] Specifically, the server inputs the depth image into a pre-trained instance segmentation model. Instance segmentation is a semantic segmentation task that not only classifies each pixel in the image, but also distinguishes different individual instances in the same category. Preferably, in an embodiment of the present application, the pre-trained instance segmentation model adopts a deep learning architecture, such as Mask R-CNN. The server predetermines a rock fragment dataset and pre-trains the rock fragment dataset to obtain an instance segmentation model.

[0050] The server generates instance masks for each rock fragment through the instance segmentation model. The instance segmentation model performs feature extraction, candidate region generation, and mask prediction on the depth image to output the instance mask for each rock fragment. The instance mask is a binary image of the same size as the depth image, where the pixel value of the same rock fragment is 1 and the other pixel values ​​are 0. In order to distinguish different rock fragment instances, the server assigns a unique color to each instance mask, such as red, green, blue, etc. In this way, the instance masks corresponding to different rock fragments have different colors.

[0051] Finally, the server fuses each instance mask with the depth image to obtain the segmented rock fragment area and background area. The server superimposes the instance mask on the depth image and distinguishes different rock fragment areas according to the color of the instance mask. For the background area, the server sets its mask value to 0, indicating that it does not belong to any rock fragment. The fusion result is a colorful segmentation map, where different colored areas correspond to different rock fragments and black areas correspond to the background.

[0052] In step S103, the number of gravels in each depth image is determined, specifically including: performing connected domain analysis on the rock fragment area to obtain multiple rock fragments included in the depth image; counting the number of rock fragments in each depth image, and using the number of rock fragments as the number of gravels.

[0053] Specifically, the server performs a connected domain analysis on the rock fragment areas in each depth image. Connected domain analysis is an image segmentation algorithm that divides pixels connected together in an image into independent areas, called connected domains. In an embodiment of the present application, the server uses the segmented depth image as input and performs a connected domain analysis on the rock fragment areas therein. Specifically, the server uses a region growing algorithm, and may also use a connected domain analysis method based on graph theory, such as an algorithm based on graph cuts. The result of the connected domain analysis is that each depth image is divided into multiple independent rock fragment areas, and each rock fragment area corresponds to a rock fragment.

[0054] After segmenting to obtain multiple rock fragments, the server counts the number of rock fragments in each depth image and uses it as the number of gravel in the depth image. The server uses the method in the above steps to traverse each depth image, count the rock fragments it contains, and obtain the number of gravel in the depth image. At the same time, the server compares the number of gravel in all depth images and determines the depth image with the largest number of gravel as the target depth image. The server stores the number of gravel in each depth image in an array or list, and uses a sorting algorithm to sort the number of gravel in descending order. After sorting, the depth image with the largest number of gravel is located in the first position of the array or list, and the server uses it as the target depth image. If there are multiple depth images with the same number of gravel and they are all the largest, the server randomly selects one of them as the target depth image.

[0055] Step S104: extracting a gravel contour point set included in the target depth image.

[0056] In step S104, after determining the target depth image, the server preprocesses the target depth image to enhance the contrast and clarity of the image. The preprocessing steps include histogram equalization, Gaussian filtering, etc. The server performs edge detection on the preprocessed target depth image to extract the outline of the gravel. Specifically, the server can use the Canny function in the image processing library such as OpenCV to perform edge detection on the target depth image to obtain a binary edge image. Next, the server performs contour extraction on the edge image to obtain a pixel coordinate point set of the gravel outline, that is, a gravel contour point set.

[0057] Step S105: Calculate the target size of each rock fragment in the target depth image according to the rock fragment contour point set.

[0058] In step S105, the target points of the gravel contour point set of each rock fragment are traversed, and the target point is used as a candidate circle center, where the target point is any point in the gravel contour point set; the target circle center is determined among the candidate circle centers, where the target circle center is the candidate circle center that can generate a circle with a maximum radius with the boundary points of the gravel contour point set; the distance from each boundary point to the target circle center is calculated, and the distances are sorted from small to large to obtain the shortest distance; the shortest distance is used as the maximum circumscribed circle radius of the rock fragment; the maximum circumscribed circle diameter is determined based on the maximum circumscribed circle radius, and the maximum circumscribed circle diameter is used as the target size.

[0059] Specifically, the server traverses the gravel contour point set of each rock fragment, and uses each point in the gravel contour point set as a candidate circle center. The server uses a loop or iteration method to visit each point in the contour point set in turn, and uses its coordinates as the coordinates of the candidate circle center. Then, the server determines the target circle center among all candidate circle centers. The target circle center is the candidate circle center that can generate a maximum radius circle with the boundary points of the gravel contour point set. Specifically, for each candidate circle center, the server calculates the distance from the center to all points in the contour point set, and finds the largest distance as the maximum radius corresponding to the candidate circle center. The server compares the maximum radius of all candidate circle centers, and selects the candidate circle center with the largest maximum radius as the target circle center.

[0060] After determining the target circle center, the server calculates the distance from each boundary point to the target circle center and sorts these distances. The server uses the Euclidean distance formula to calculate the distance from each point in the gravel contour point set to the target circle center to obtain a set of distance values. The server uses a sorting algorithm, such as quick sort, to sort this set of distance values ​​from small to large. After sorting, the server takes the first distance value, that is, the smallest distance value, as the shortest distance of the rock fragment.

[0061] Next, the server uses the shortest distance as the maximum circumscribed circle radius of the rock fragment. Since the shortest distance corresponds to the point in the gravel contour point set that is closest to the target circle center, the circle with the shortest distance as the radius can be exactly tangent to the gravel contour and become the maximum circumscribed circle of the gravel. The server saves the shortest distance as the maximum circumscribed circle radius of the rock fragment to characterize the size of the gravel. Finally, the server determines the maximum circumscribed circle diameter based on the maximum circumscribed circle radius and uses it as the target size of the rock fragment.

[0062] In summary, through the above steps, the embodiment of the present application uses binocular vision technology to obtain the depth information of the rock fragment scene, and combines instance segmentation and geometric calculation to accurately identify the target size of each rock fragment. The identified target size can be used for subsequent blasting effect evaluation to improve production efficiency and safety.

[0063] Reference Figure 2The present application also provides a blasting fragmentation recognition device based on binocular vision, which is a server. The server includes an acquisition module 201, which is used to obtain a dual-perspective image of a rock fragment scene taken by a binocular camera. The binocular camera includes two left and right sub-cameras with overlapping fields of view, and the dual-perspective image includes a left-perspective image and a right-perspective image; the processing module 202 is also used to extract depth values ​​of the left-perspective image and the right-perspective image based on the triangulation principle to obtain a corresponding depth image; the processing module 202 is also used to determine the number of gravels in each depth image, and determine a target depth image based on the number of gravels, and the target depth image is a depth image with the largest number of gravels in the depth image; the processing module 202 is also used to extract a gravel contour point set included in the target depth image; the processing module 202 is also used to calculate the target size of each rock fragment in the target depth image based on the gravel contour point set.

[0064] In a possible implementation, the processing module 202 extracts depth values ​​from the left-view image and the right-view image based on the principle of triangulation to obtain a corresponding depth image, which specifically includes: the processing module 202 matches the dual-view images to identify the corresponding gravel areas in the left-view image and the right-view image; the processing module 202 triangulates the gravel area according to the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera to calculate the depth value of each pixel in the gravel area; and the depth value of each pixel is fused with the corresponding color image to obtain a depth image.

[0065] In a possible implementation, the processing module 202 matches the dual-view images to identify corresponding gravel areas in the left-view image and the right-view image, specifically including: the processing module 202 performs feature extraction on the left-view image and the right-view image to obtain a first feature point set and a second feature point set, the first feature point set corresponds to the left-view image, and the second feature point set corresponds to the right-view image; the processing module 202 performs feature point matching on the first feature point set and the second feature point set to obtain matching feature point pairs; the processing module 202 determines matching rectangular neighborhood windows of the left-view image and the right-view image with each pair of matching feature point pairs as the center; the processing module 202 determines candidate gravel areas based on the normalization coefficients of the matching rectangular neighborhood windows, and performs morphological closing operations on the candidate gravel areas to obtain gravel areas.

[0066] In a possible implementation, the processing module 202 triangulates the gravel area according to the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera, and calculates the depth value of each pixel in the gravel area, specifically including: the processing module 202 calculates the basic matrix between the left-view image and the right-view image according to the intrinsic parameter matrix and the extrinsic parameter matrix; the processing module 202 matches the gravel area based on the basic matrix to obtain matching pixel pairs, and calculates the disparity value according to the coordinates of the matching pixel pairs; the processing module 202 combines the disparity value, the baseline distance of the binocular camera, and the focal length of the binocular camera, and uses the triangulation principle to calculate the depth value of each pixel.

[0067] In a possible implementation, before the processing module 202 determines the amount of gravel in each depth image, the method further includes: the processing module 202 inputs the depth image into a pre-trained instance segmentation model; the processing module 202 generates instance masks for each rock fragment through the instance segmentation model, and different rock fragments correspond to instance masks of different colors; the processing module 202 fuses each instance mask with the depth image to obtain segmented rock fragment areas and background areas.

[0068] In one possible implementation, the processing module 202 determines the number of gravels in each depth image, specifically including: the processing module 202 performs a connected domain analysis on the rock fragment area to obtain multiple rock fragments included in the depth image; the processing module 202 counts the number of rock fragments in each depth image, and uses the number of rock fragments as the number of gravels.

[0069] In a possible implementation, the processing module 202 calculates the target size of each rock fragment in the target depth image based on the gravel contour point set, specifically including: the processing module 202 traverses the target points of the gravel contour point set of each rock fragment, and uses the target point as a candidate center of a circle, where the target point is any point in the gravel contour point set; the processing module 202 determines the target center of a circle among the candidate centers of a circle, where the target center of a circle is a candidate center of a circle that can generate a circle with a maximum radius with the boundary points of the gravel contour point set among the candidate centers of a circle; the processing module 202 calculates the distance from each boundary point to the target center of a circle, and sorts the distances from small to large to obtain the shortest distance; the processing module 202 uses the shortest distance as the maximum circumscribed circle radius of the rock fragment; determines the maximum circumscribed circle diameter based on the maximum circumscribed circle radius, and uses the maximum circumscribed circle diameter as the target size.

[0070] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0071] The present application also provides an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0072] The communication bus 302 is used to realize the connection and communication between these components.

[0073] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0074] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0075] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0076] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 The memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a blasting block size recognition method based on binocular vision.

[0077] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program stored in the memory 305 for a method for identifying the size of blasting fragments based on binocular vision. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0078] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0084] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0085] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for identifying blasting fragmentation based on binocular vision, characterized in that: The method comprises: Acquire a dual-perspective image of a rock fragment scene captured by a binocular camera, wherein the binocular camera includes two left and right sub-cameras with overlapping fields of view, and the dual-perspective image includes a left-perspective image and a right-perspective image; Based on the triangulation principle, extracting depth values ​​of the left-view image and the right-view image to obtain corresponding depth images; Determine the number of gravels in each of the depth images, and determine a target depth image according to the number of gravels, wherein the target depth image is a depth image having the largest number of gravels among the depth images; Extracting a gravel contour point set included in the target depth image; The target size of each rock fragment in the target depth image is calculated according to the rock fragment contour point set.

2. The method according to claim 1, characterized in that: The extracting depth values ​​of the left-view image and the right-view image based on the triangulation principle to obtain corresponding depth images specifically includes: Matching the dual-view images to identify corresponding gravel areas in the left-view image and the right-view image; According to the internal parameter matrix and the external parameter matrix of the binocular camera, triangulation is performed on the gravel area to calculate the depth value of each pixel point in the gravel area; The depth value of each pixel is fused with the corresponding color image to obtain the depth image.

3. The method according to claim 2, characterized in that The matching of the dual-view images to identify the corresponding gravel areas in the left-view image and the right-view image specifically includes: Performing feature extraction on the left-view image and the right-view image to obtain a first feature point set and a second feature point set, wherein the first feature point set corresponds to the left-view image, and the second feature point set corresponds to the right-view image; Performing feature point matching on the first feature point set and the second feature point set to obtain matching feature point pairs; Determine a matching rectangular neighborhood window of the left-view image and the right-view image with each pair of the matching feature point pairs as the center; A candidate gravel region is determined according to the normalized coefficient of the matching rectangular neighborhood window, and a morphological closing operation is performed on the candidate gravel region to obtain the gravel region.

4. The method according to claim 2, characterized in that: The triangulation measurement of the gravel area is performed according to the internal parameter matrix and the external parameter matrix of the binocular camera to calculate the depth value of each pixel point in the gravel area, specifically including: Calculating a basic matrix between the left-view image and the right-view image according to the intrinsic parameter matrix and the extrinsic parameter matrix; Based on the basic matrix, the gravel area is matched to obtain matching pixel pairs, and the disparity value is calculated according to the coordinates of the matching pixel pairs; The parallax value, the baseline distance of the binocular camera and the focal length of the binocular camera are comprehensively considered, and the depth value of each pixel point is calculated using the triangulation principle.

5. The method according to claim 1, characterized in that Before determining the number of gravels in each of the depth images, the method further includes: Inputting the depth image into a pre-trained instance segmentation model; Generate instance masks of the rock fragments by using the instance segmentation model, where different rock fragments correspond to instance masks of different colors; Each of the instance masks is fused with the depth image to obtain segmented rock fragment regions and background regions.

6. The method according to claim 5, characterized in that The determining the number of gravel in each of the depth images specifically includes: Performing a connected domain analysis on the rock fragment region to obtain a plurality of rock fragments included in the depth image; The number of the rock fragments in each of the depth images is counted, and the number of the rock fragments is used as the number of gravel.

7. The method according to claim 1, characterized in that The step of calculating the target size of each rock fragment in the target depth image according to the gravel contour point set specifically includes: Traversing the target point of the gravel contour point set of each of the rock fragments, and taking the target point as a candidate circle center, the target point being any point in the gravel contour point set; Determine a target circle center among the candidate circle centers, the target circle center being the candidate circle center among the candidate circle centers that can generate a circle with a maximum radius with the boundary points of the gravel contour point set; Calculate the distance from each boundary point to the center of the target circle, and sort the distances from small to large to obtain the shortest distance; Taking the shortest distance as the maximum circumscribed circle radius of the rock fragment; The maximum circumscribed circle diameter is determined according to the maximum circumscribed circle radius, and the maximum circumscribed circle diameter is used as the target size.

8. A blasting block size recognition device based on binocular vision, characterized in that: The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to acquire a dual-perspective image of a rock fragment scene captured by a binocular camera, wherein the binocular camera comprises two left and right sub-cameras with overlapping fields of view, and the dual-perspective image comprises a left-perspective image and a right-perspective image; The processing module (202) is further used to extract depth values ​​from the left-view image and the right-view image based on the triangulation principle to obtain corresponding depth images; The processing module (202) is further used to determine the number of gravels in each of the depth images, and determine a target depth image according to the number of gravels, wherein the target depth image is the depth image with the largest number of gravels in the depth images; The processing module (202) is further used to extract a gravel contour point set included in the target depth image; The processing module (202) is further configured to calculate the target size of each rock fragment in the target depth image based on the rock fragment contour point set.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images