Binocular vision rock fragment fragmentation identification method and system based on portal frame
Through the binocular vision technology based on the portal frame, the three-dimensional structure identification and volume calculation of rock fragments are realized, which solves the problem of low manual detection efficiency in the existing technology and improves the identification efficiency and accuracy.
Patent Information
- Application Number
- CN202411484983.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-01-21
AI Technical Summary
The existing rock fragment block recognition methods rely on manual detection, are inefficient and time-consuming, making it difficult to meet the precise measurement needs of mining automation.
Using a door -based two -eyed visual technology, the multi -frame image of the target vehicle across the door frame is obtained, and the target detection and pixel depth calculation are performed. Based on the two -dimensional border frame, the three -dimensional outline of the rock fragments is determined, and the pieces of the fragments are calculated.
It achieves efficient conversion from two-dimensional images to three-dimensional structures, accurately obtains three-dimensional structural information and volumes of rock fragments, improves the efficiency of identifying rock fragments, and reduces manual inspection time.
Smart Images
Figure CN119338894A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a binocular vision rock fragment size recognition method, system, electronic device and storage medium based on a portal frame. Background Art
[0002] With the increasing automation in the mining industry, accurate measurement of rock fragment size, i.e., the volume of rock fragments on transport vehicles, has become the key to improving efficiency and reducing resource waste. This measurement is not only related to cost control, but also directly affects the decision-making of subsequent processing and resource allocation.
[0003] At present, existing rock fragment size identification methods mostly rely on workers to manually detect and calculate the rock fragments transported in vehicles to determine the rock fragment size. However, in practical applications, since the workload of rock fragment identification is often large, manual detection by workers requires repeated tedious measurements and consumes a lot of detection time, resulting in low efficiency in rock fragment size identification. Summary of the invention
[0004] The present application provides a binocular vision rock fragment size recognition method, system, electronic device and storage medium based on a portal frame, which has the effect of improving the efficiency of rock fragment size recognition.
[0005] In a first aspect, the present application provides a binocular vision rock fragment size recognition method based on a portal frame, comprising: Acquire multiple frame images of the target vehicle driving through the portal frame; Performing target detection on each of the frame images to determine a two-dimensional boundary box of rock fragments in each of the frame images, a plurality of boundary pixel points, and pixel coordinates corresponding to each of the boundary pixel points; For each of the frame images, based on each of the pixel coordinates, determine a pixel coordinate difference between any two of the boundary pixel points, and calculate a pixel depth of each of the boundary pixel points according to each of the pixel coordinate differences and an average distance between the cameras on the portal frame; The three-dimensional contour of the rock fragment is determined in combination with each of the pixel depths and the two-dimensional bounding box, and the fragment volume of the rock fragment in the frame image is calculated based on the contour parameters of the three-dimensional contour.
[0006] In a second aspect of the present application, a binocular vision rock fragment size recognition system based on a portal frame is provided, the system comprising: An image acquisition module, used to acquire multiple frame images of the target vehicle driving through the portal frame; A target detection module is used to perform target detection on each of the frame images, and determine a two-dimensional boundary box of rock fragments in each of the frame images, a plurality of boundary pixel points, and pixel coordinates corresponding to each of the boundary pixel points; An image processing module, configured to determine, for each frame image, a pixel coordinate difference between any two of the boundary pixel points based on each of the pixel coordinates, and calculate a pixel depth of each of the boundary pixel points according to each of the pixel coordinate differences and an average distance between each of the cameras on the portal frame; The rock fragment identification module is used to determine the three-dimensional contour of the rock fragment in combination with each of the pixel depths and the two-dimensional bounding box, and calculate the fragment volume of the rock fragment in the frame image according to the contour parameters of the three-dimensional contour.
[0007] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can implement a binocular vision rock fragment size recognition method based on a portal frame when loaded and executed by the processor.
[0008] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements a binocular vision rock fragment size identification method based on a portal frame.
[0009] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By adopting the above technical solution, multiple frames of images of the target vehicle are obtained and rock fragments are detected and pixel depth is calculated, and then the three-dimensional contour is determined in combination with the two-dimensional bounding box, so that the three-dimensional structural information and fragment volume of the rock fragments can be effectively obtained. The parallax principle of multi-camera imaging is utilized, and the depth information of each pixel is accurately calculated according to the coordinate change of the pixel points of the bounding box under different camera viewing angles, and then converted into a three-dimensional contour corresponding to the two-dimensional bounding box, thereby realizing an efficient conversion from a two-dimensional image to a three-dimensional structure. The obtained three-dimensional contour of the rock fragment fully retains its spatial information, and the precise volume of the fragment can be calculated based on it. The camera set by the portal frame takes a picture of the target vehicle and automatically identifies the rock fragments, saving the time consumed by repeated manual detection, thereby improving the efficiency of rock fragment block recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 It is a flow chart of a binocular vision rock fragment size recognition method based on a portal frame provided in an embodiment of the present application; Figure 2 It is a schematic diagram of a door frame provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a binocular vision rock fragment size recognition system based on a portal frame provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.
[0011] Description of reference numerals: 400, electronic device; 401, processor; 402, communication bus; 403, user interface; 404, network interface; 405, memory. DETAILED DESCRIPTION
[0012] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.
[0013] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.
[0014] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.
[0015] The present application embodiment provides a method for monitoring abnormal crane operation. In one embodiment, please refer to Figure 1 , Figure 1 The flowchart of the crane operation abnormality monitoring method provided by the embodiment of the present application can be implemented by a computer program, which can be integrated into an application or run as an independent tool application. The method can also be implemented by a single-chip microcomputer or run on a crane operation abnormality monitoring system based on the von Neumann system. Specifically, the method can include the following steps: Step 101: Acquire multiple frame images of a target vehicle driving through a portal frame.
[0016] The frame image refers to a continuous static image in a video sequence, and in the embodiment of the present application, it can be understood as a continuous vehicle image captured and extracted by a camera in a portal frame. The frame image is used to record the state of the vehicle and the rock fragments on the vehicle at different times, and a video contains multiple frame images in time sequence.
[0017] refer to Figure 2 The portal frame refers to a portal-shaped structure made of materials such as steel frames. In the embodiments of the present application, it can be understood as a steel frame support body set at the entrance and exit of a mining site, and multiple cameras are fixedly installed on the portal frame. The portal frame is used for when a target vehicle passes under the portal frame, and its camera can take multi-angle photos of the vehicle to obtain image information of rock fragments in the vehicle.
[0018] Specifically, in order to realize the automated three-dimensional modeling and volume calculation of rock fragments on transport vehicles, it is necessary to first obtain the image information of rock fragments in the vehicle. Here, the method of setting a portal frame is adopted to obtain images, because the portal frame can fix multiple cameras. When the target vehicle passes through the portal frame, each camera can capture images of the vehicle at different angles, thereby obtaining multi-view information of rock fragments in the vehicle. A portal steel frame is set at the entrance and exit of the mining site in advance, and multiple cameras are fixed on the steel frame to aim at the passage through which the vehicle passes. The position and angle of the camera ensure that all rock fragments in the vehicle enter; when the vehicle passes under the portal steel frame, the control system automatically triggers each camera to continuously shoot the vehicle, and obtains images at different times during the vehicle's driving process.
[0019] Step 102: Perform target detection on each frame image to determine a two-dimensional boundary box of rock fragments in each frame image, a plurality of boundary pixel points, and pixel coordinates corresponding to each boundary pixel point.
[0020] The two-dimensional bounding box refers to the smallest rectangular box that represents the outline of the target in the image plane, and in the embodiment of the present application, it can be understood as the bounding rectangle of the rock fragment in the frame image. The two-dimensional bounding box is used to represent the two-dimensional outline of the rock fragment in the image plane.
[0021] Boundary pixels refer to pixel samples that constitute the boundary of the target, and in the embodiment of the present application, can be understood as pixels that constitute the two-dimensional boundary box of the rock fragment. Boundary pixels are used to represent detailed information of the edge of the rock fragment.
[0022] Pixel coordinates refer to the digital index representing each pixel in the image in the row and column direction, and in the embodiments of the present application, can be understood as representing the coordinate position of the boundary pixel point on the frame image plane. Pixel coordinates are used to determine and record the row and column position of each boundary pixel point in the image matrix.
[0023] Specifically, in order to accurately extract the visual information of rock fragments from the frame image, it is necessary to detect rock fragments in each frame image and determine the position of rock fragments in the image. Here, a pre-trained target detection model is used to detect rock fragments in each frame image. The target detection model can identify the position of rock fragments in the image in real time and give its bounding box. First, a target detection model for detecting rock fragments is obtained by using the collected image sample data set containing rock fragments through the model training algorithm; then the acquired vehicle frame images are input into the model in turn, and the target detection model will output the two-dimensional bounding box corresponding to the rock fragments in each frame image, that is, the rock fragment frame represented in pixels in the image plane; then all the boundary pixels constituting the frame are determined based on the bounding box, and the row and column coordinates of each boundary pixel in the image matrix, that is, the pixel coordinates, are recorded. In this way, the rock fragment area can be automatically located through target detection, and its boundary information, including the bounding box, boundary pixels and corresponding pixel coordinates, can be obtained, providing key data for subsequent three-dimensional modeling, avoiding the error of manually selecting the area or extracting the boundary, and improving the accuracy.
[0024] Based on the above embodiment, as an optional embodiment, in step 102: performing target detection on each frame image, determining a two-dimensional boundary box of rock fragments in each frame image, a plurality of boundary pixel points, and pixel coordinates corresponding to each boundary pixel point, this step may also include the following steps: Step 201: Input each frame image into a pre-trained target detection model to obtain edge features of rock fragments in each frame image, and extract multiple boundary pixel points in each edge feature.
[0025] Among them, the pre-trained target detection model refers to a neural network model obtained by training with sample data and capable of realizing specific target recognition and positioning. In the embodiments of the present application, it can be understood as a target detection model obtained by training with images containing rock fragments and capable of detecting rock fragments in images.
[0026] The edge feature refers to the visual information of the boundary of the object in the image, and in the embodiment of the present application, it can be understood as the pixel intensity feature of the edge area of the rock fragment. The edge feature is used to represent the details of the boundary of the rock fragment.
[0027] Boundary pixels refer to pixel samples that constitute the boundary of the target, and in the embodiments of the present application, can be understood as pixels that constitute the edge features of rock fragments. Boundary pixels are used to represent detailed information on the edge of rock fragments.
[0028] Specifically, in order to accurately detect the contour information of rock fragments from the frame image, it is necessary to use a pre-trained target detection model. First, collect image samples with annotated edge features of rock fragments, and use these annotated data to train and optimize the target detection model; then input the collected vehicle frame images into this pre-trained model in turn, and the target detection model will perform forward calculations on each input frame image and output the edge features corresponding to the rock fragments in each image, that is, the area with large pixel intensity changes at the edge; then post-process the rock fragment edge feature image output by the model to extract the key pixel points that constitute the boundary in the feature area. In this way, with the help of the pre-trained target detection model, the detection and positioning of the edges of rock fragments can be automatically and efficiently completed, and the pixel points that express the edge details can be extracted without manual labeling or extraction, avoiding subjective errors and improving the accuracy of the results.
[0029] Based on the above embodiment, as an optional embodiment, in step 201: inputting each frame image into a pre-trained target detection model, before this step, the following steps may also be included: Step 211: Obtain image samples corresponding to various rock fragments; perform data cleaning on each image sample to obtain an image training set corresponding to each image sample, and attach edge feature labels corresponding to the rock fragments in each image training set to each image training set.
[0030] The image samples refer to data source images used for model training, which can be understood as sample images containing rock fragments in the embodiments of the present application. The image samples are used to provide the model with sufficient learning materials and improve the model's adaptability to different targets.
[0031] The image training set refers to a sample data set that has been preprocessed and annotated. In the embodiment of the present application, it can be understood as a set of images annotated with edge features of rock fragments. The image training set is used to provide the model with sufficient and accurate training materials.
[0032] The edge feature label refers to the identification information of the target edge area marked on the image sample, and in the embodiment of the present application, it can be understood as a label artificially drawn on the rock fragment sample image to represent its edge feature.
[0033] Specifically, in order to train a target detection model with strong edge feature detection capabilities for various rock fragments, sufficient and accurate sample data needs to be prepared. First, images of rock fragments of different shapes and sizes are collected to form a diverse image sample library; then the collected sample images are preprocessed with necessary data cleaning, including removing image noise, cropping to retain the main body of the rock fragments, adjusting the resolution, etc.; then, manual annotation is performed on the preprocessed sample images, and the edge feature areas of the rock fragments are manually drawn and corresponding labels are added to obtain an image training set with target edge feature annotations. In this way, by obtaining sufficient and accurate sample data, normalizing and feature-annotating the sample images, it can provide a rich and accurate training source for subsequent target detection models, improve the model's edge detection and extraction capabilities for rock fragments in different situations, and enable it to output more accurate edge feature positioning results in practical applications.
[0034] Step 221: Perform model training on the pre-built initial training model in combination with each image training set to obtain a target detection model after model training.
[0035] The target detection model refers to a neural network that can identify a specific target in an image and locate its position. In the embodiments of the present application, it can be understood as a model obtained through training that can detect the edge features of rock fragments in an image. The target detection model is used to perform forward calculations on the input image and output the position information of the target in the image.
[0036] Specifically, in order to obtain a target detection model with strong ability to detect edge features of various rock fragments, it is necessary to use the labeled image training set to effectively train the initial model. First, an initial target detection model based on a convolutional network is constructed; then the prepared image training set with labeled edge features of rock fragments is loaded, and the initial model is input in turn for forward calculation to obtain edge detection results; combined with the edge feature label information of the sample, the error between the detection result and the actual annotation is calculated as the loss, and the model parameters are updated through the back propagation algorithm to reduce the loss; after multiple cycles of training, the model parameters are gradually optimized, and finally a target detection model with more accurate edge feature detection of rock fragments is obtained. In this way, through effective training with a large number of labeled samples, the model's ability to detect and adapt to the edges of different rock fragments can be significantly enhanced, so that it can output the key edge features of rock fragments more accurately and efficiently in actual use.
[0037] Step 202: For each edge feature, based on the position of each boundary pixel point in the frame image, determine the pixel coordinates corresponding to each boundary pixel point.
[0038] Specifically, in order to establish the correspondence between image pixel information and actual three-dimensional coordinates, it is necessary to clarify the coordinate position of each boundary pixel in the frame image. After inputting the pre-trained model, a series of discrete boundary pixel points can be extracted; then, according to the row and column indexes of these boundary pixel points in the input frame image matrix, the coordinate position of each boundary pixel point on the image plane, that is, the pixel coordinates, is determined; the pixel coordinates of each boundary pixel point and the corresponding edge feature area are recorded. In this way, by marking each boundary pixel point with a pixel coordinate, an index correspondence from two-dimensional image pixels to actual three-dimensional space points is established, which provides a basis for accurately inferring three-dimensional coordinates based on image information, and also makes the subsequent three-dimensional structure registration and texture mapping more accurate, avoiding the reconstruction deviation problem caused by the lack of boundary point index information.
[0039] Step 203: Based on the pixel coordinates of each boundary pixel point, determine the minimum rectangular area surrounding the rock fragment, and use the minimum rectangular area as the two-dimensional boundary box of the rock fragment.
[0040] The minimum rectangular area refers to the smallest rectangular frame that can contain all boundary points of the target, and in the embodiment of the present application, it can be understood as the smallest circumscribed rectangle fitted based on the boundary pixel points of the rock fragment. The minimum rectangular area is used to represent the overall outline of the rock fragment in the image plane.
[0041] Specifically, in order to quickly represent the overall contour area of the rock fragment in the image plane, it is necessary to further construct a two-dimensional bounding box. According to the extracted and labeled coordinates of each boundary pixel point, the minimum circumscribed rectangle of these boundary pixel points is calculated as the two-dimensional bounding box of the rock fragment, that is, the minimum rectangular box that can contain all boundary pixels. In this way, a two-dimensional bounding box is fitted through a set of boundary pixels, which not only retains the information of edge detail pixels, but also provides a simplified rectangular box to express the overall contour of the rock fragment, making image processing and three-dimensional registration simpler and faster. At the same time, the rectangular box structure is also convenient for parameterized representation, laying the foundation for the subsequent extraction of pixel coordinates in the frame and restoration of the three-dimensional structure.
[0042] Step 103: For each frame image, based on each pixel coordinate, determine the pixel coordinate difference between any two boundary pixel points, and calculate the pixel depth of each boundary pixel point according to each pixel coordinate difference and the average distance between each camera on the portal frame.
[0043] Among them, pixel depth refers to the number of color value bits contained in each pixel in the image. In the embodiments of the present application, it can be understood as the number of gray levels corresponding to each pixel. Pixel depth is used to indicate the level of color detail that can be represented by a pixel in the image.
[0044] Specifically, in order to establish an accurate correspondence between image pixels and three-dimensional space, it is necessary to calculate the depth information of each boundary pixel. First, based on the pixel coordinates of the boundary pixel points in the image coordinate system, we obtain the pixel coordinates of the boundary pixel points; then select any two points in the image and calculate the pixel coordinate difference between the two points in the image coordinate system; then, based on the known average distance between the cameras in the gantry frame and the pixel coordinate difference between the two selected points, we calculate the depth of each boundary pixel point relative to the gantry frame through the similar triangle relationship, that is, the pixel depth. In this way, we can obtain the specific depth information of each boundary pixel point. In the subsequent three-dimensional reconstruction, the coordinate position of the pixel point is used to determine its precise position in the three-dimensional space, so that the reconstructed three-dimensional model can restore the real structure of the target with high precision.
[0045] Based on the above embodiment, as an optional embodiment, in step 103: for each frame image, based on each pixel coordinate, determining the pixel coordinate difference between any two boundary pixel points, this step may also include the following steps: Step 301: In each frame image, select any two boundary pixels.
[0046] Specifically, in order to calculate the depth information of each boundary pixel point, it is necessary to select two boundary pixel points in the image as reference points for calculation. First, randomly select any two boundary pixel points in the image from the extracted boundary pixel point set, as long as there is a certain pixel distance between the two points in the image coordinate system. The two points selected in this way can form a reference pixel distance. Based on this distance and the actual distance of the camera in the portal frame, the depth of each boundary pixel point to the imaging plane, that is, the pixel depth, can be calculated through similar triangular relationships. Selecting appropriate reference points is very important to improve the accuracy of pixel depth calculation, and also lays the foundation for the subsequent accurate mapping of image pixels and three-dimensional point cloud coordinates.
[0047] Step 302: Based on the pixel coordinates of any two boundary pixel points, a first difference in the horizontal direction and a second difference in the vertical direction of the two boundary pixel points are calculated.
[0048] Among them, the first difference refers to the difference in the horizontal coordinate values of the two selected boundary pixel points in the image plane. In the embodiment of the present application, it can be understood as the horizontal pixel distance difference between the two boundary pixel points in the horizontal axis direction of the image. The first difference is used to represent the horizontal pixel distance between the two boundary pixel points in the image coordinate system.
[0049] The second difference refers to the difference in the vertical coordinate values of the two selected boundary pixel points in the image plane. In the embodiment of the present application, it can be understood as the vertical pixel distance difference between the two boundary pixel points in the vertical axis direction of the image. The second difference is used to represent the vertical pixel distance between the two boundary pixel points in the image coordinate system.
[0050] Specifically, in order to subsequently use the pixel distance between the two selected boundary pixels to calculate the target depth, it is necessary to first find the precise pixel distance between the two points in the image coordinate system. First obtain the image plane coordinates of the two points, such as (x1, y1) and (x2, y2), and then calculate the horizontal first difference of the two points in the x-axis direction of the image as x1-x2, and calculate the vertical second difference of the two points in the y-axis direction of the image as y1-y2. In this way, the horizontal and vertical pixel distances between the two selected points in the image coordinate system can be accurately obtained. These two difference parameters abstractly summarize the distance relationship between the two points on the image plane, and are the data basis for accurately calculating the depth information of each boundary pixel.
[0051] Step 303: taking the first difference and the second difference as the pixel coordinate difference between any two boundary pixel points.
[0052] Specifically, in order to establish the distance relationship between two boundary pixels on the image plane, it is necessary to use the calculated differences between the two points in the horizontal and vertical directions as pixel coordinate differences to form an abstract representation. The first difference between the two points in the x-axis direction and the second difference in the y-axis direction are combined to represent the pixel distance between the two points in the image coordinate system as the pixel coordinate difference between the two boundary pixels. In this way, the spatial distance relationship between the two points on the image plane is expressed by the combination of the first difference and the second difference. This pixel coordinate difference is the basic parameter for accurately calculating the target depth using the camera intrinsic parameter information, and plays a key role in establishing an accurate mapping between pixel points and actual three-dimensional coordinates.
[0053] Based on the above embodiment, as an optional embodiment, in step 102: calculating the pixel depth of each boundary pixel point according to the pixel coordinate difference and the average distance between the cameras on the portal frame, this step may also include the following steps: Step 304: Obtain the focal length of each camera on the portal frame; multiply the focal length by the average distance between the cameras on the portal frame to obtain a depth coefficient.
[0054] Among them, the focal length refers to an internal parameter of the camera, which represents the distance from the optical center to the imaging plane. In the embodiments of the present application, it can be understood as the focal length parameter of each camera in the portal frame.
[0055] The depth coefficient refers to the proportional coefficient between the mapped image pixel distance and the actual physical distance. In the embodiments of the present application, it can be understood as the scaling ratio of the corresponding relationship between the pixel depth and the real distance obtained using the camera intrinsic parameters. The depth coefficient is used to estimate the actual depth of each pixel point based on the pixel coordinate difference.
[0056] Specifically, in order to establish an accurate correspondence between the pixel distance on the image plane and the actual three-dimensional space distance, it is necessary to calculate a depth mapping scale coefficient based on the camera's intrinsic parameters. First, obtain the focal length parameters of each camera in the portal frame multi-camera system; then use the focal length and the average distance between cameras, and use their product as a mapping scale coefficient, namely the depth coefficient. In this way, the depth coefficient abstractly summarizes the linear conversion relationship between the pixel distance and the actual distance. In subsequent calculations, this depth coefficient can be used to accurately convert the pixel coordinate difference to the actual physical distance, that is, the corresponding depth value, in proportion.
[0057] Step 305: For each boundary pixel point, divide the depth coefficient by the pixel coordinate difference corresponding to the boundary pixel point to obtain the pixel depth of the boundary pixel point.
[0058] Specifically, in order to obtain the depth information of each boundary pixel point, it is necessary to use the depth coefficient and the corresponding pixel coordinate difference to calculate the pixel depth. First, for each boundary pixel point, obtain the pixel coordinate difference between it and the previously selected reference pixel point; then substitute this coordinate difference into the denominator of the depth coefficient, with the depth coefficient as the numerator, and perform division calculation to obtain the depth value of the boundary pixel point relative to the imaging plane, that is, the pixel depth. In this way, by converting the relationship between the depth coefficient and the pixel coordinate difference, the calculation from the image plane pixel coordinate mapping to the actual physical depth of the three-dimensional space is realized, and the obtained pixel depth provides data support for subsequent three-dimensional modeling.
[0059] Step 104: Determine the three-dimensional contour of the rock fragment by combining the depth of each pixel and the two-dimensional bounding box, and calculate the fragment volume of the rock fragment in the frame image according to the contour parameters of the three-dimensional contour.
[0060] The 3D contour refers to the boundary contour of the target in the 3D space, which can be understood as the 3D coordinate point set of the surface boundary of the rock block obtained by pixel depth calculation in the embodiment of the present application. The 3D contour is used to represent the external structure of the target in the 3D space.
[0061] Contour parameters refer to a set of parameters that describe 3D contour information, and in the embodiments of the present application, can be understood as data composed of 3D coordinates of each point that constitutes the 3D contour. Contour parameters are used to represent the spatial information of the target 3D contour.
[0062] The fragment volume refers to the three-dimensional volume of the rock fragment, and in the embodiment of the present application, it can be understood as the volume of the three-dimensional model of the rock fragment constructed by the three-dimensional contour. The fragment volume is used to indicate the three-dimensional space occupation of the rock fragment.
[0063] Specifically, in order to obtain the three-dimensional spatial information of rock fragments, it is necessary to determine the three-dimensional contour based on the pixel depth combined with the two-dimensional bounding box, and calculate the volume of the fragments. First, according to the pixel coordinate position of each boundary pixel point and the calculated pixel depth, the three-dimensional coordinates of the corresponding point in the three-dimensional space can be mapped and determined, thereby forming the three-dimensional contour of the rock fragment. Then, according to the structural information described by the extracted three-dimensional contour parameters, a three-dimensional model of the rock fragment is constructed through a three-dimensional modeling algorithm. Finally, the volume of the three-dimensional model is calculated to obtain the fragment volume of the fragment. In this way, not only the three-dimensional contour information of the fragment is obtained, but also its specific volume size in the three-dimensional space is obtained, which provides key data support for the subsequent generation of the overall fragment accumulation body.
[0064] Based on the above embodiment, as an optional embodiment, in step 104: calculating the fragment volume of the rock fragments in the frame image according to the contour parameters of the three-dimensional contour, this step may also include the following steps: Step 401: Match the boundary coordinates in the two-dimensional boundary box with the depth of each pixel to determine the three-dimensional coordinate point set of the rock fragment boundary.
[0065] Among them, the three-dimensional coordinate point set refers to a data set of a series of three-dimensional coordinate points that constitute a three-dimensional contour. In the embodiment of the present application, it can be understood as a coordinate set of three-dimensional points on the boundary of rock fragments obtained by matching and transforming the two-dimensional bounding box pixel coordinates with the corresponding pixel depth.
[0066] Specifically, in order to obtain the three-dimensional spatial information of the boundary of the rock fragment, it is necessary to match the two-dimensional bounding box with the calculated pixel depth to determine the three-dimensional coordinate point set. First, extract the two-dimensional bounding box of the rock fragment image and obtain the pixel coordinates in the bounding box. Then, take out each bounding box pixel coordinate in turn, and match and search for its corresponding depth value in the pixel depth set of each boundary pixel point calculated previously. By combining each bounding box pixel coordinate with the matched pixel depth, the precise three-dimensional coordinate of the pixel point in the three-dimensional space can be determined. Finally, the three-dimensional coordinates of all bounding box pixel points are summarized to obtain a three-dimensional coordinate point set describing the three-dimensional spatial boundary information of the rock fragment. In this way, the conversion from two-dimensional boundary to three-dimensional structure is realized, providing data support for subsequent three-dimensional modeling.
[0067] Step 402: Fit each set of three-dimensional coordinate points to obtain a three-dimensional contour of the rock fragment.
[0068] Specifically, in order to obtain accurate three-dimensional contour information of rock fragments, it is necessary to further fit the calculated three-dimensional coordinate point set. First, a three-dimensional contour fitting algorithm is used to perform contour fitting with the three-dimensional coordinate point set as the original data. The algorithm continuously approximates the shape of the three-dimensional coordinate point set through iteration to obtain a fitting curve that can accurately describe the three-dimensional contour shape of the fragment. Then, the coordinates of the points on the fitting curve are output as the final three-dimensional contour information of the rock fragment. In this way, not only the noise in the three-dimensional coordinate point set is filtered out, but also the three-dimensional contour shape of the fragment is made more accurate and smooth. The obtained fitted three-dimensional contour provides reliable three-dimensional structural data for subsequent modeling and analysis.
[0069] Based on the above embodiment, as an optional embodiment, in step 104: calculating the fragment volume of the rock fragments in the frame image according to the contour parameters of the three-dimensional contour, this step may also include the following steps: Step 403: construct a reference cube corresponding to the three-dimensional contour according to the maximum contour length, the maximum contour width and the maximum contour height, and calculate the volume of the reference cube.
[0070] Among them, the reference cube refers to a cube model constructed according to the target three-dimensional contour parameters for estimating the target volume. In the embodiment of the present application, it can be understood as a cube constructed according to the three-dimensional contour size parameters of the rock block fragments for quickly estimating the volume of the fragments. The reference cube is used to quickly estimate the approximate volume of the fragments through the cube model.
[0071] Specifically, in order to quickly obtain the volume estimation of rock fragments, it is necessary to calculate by constructing a reference cube. First, the three parameters of the maximum contour length, maximum contour width and maximum contour height obtained from the extracted fragment three-dimensional contour are used as the three side lengths of the fragment reference cube model. Then, a reference cube that matches the shape of the fragment is constructed in the three-dimensional coordinate system based on these three side lengths. After that, the volume calculation function of the cube can be directly called to obtain the model volume of the reference cube. Through this volume result, the volume size of the actual fragment can be quickly estimated. In this way, the time of accurate modeling and then calculating the volume is saved, and an approximate value of the fragment volume close to the actual value is obtained, thereby improving processing efficiency.
[0072] Step 404: determine the volume ratio coefficient between the three-dimensional contour and the reference cube; multiply the volume of the minimum cube by the volume ratio coefficient to obtain the fragment volume of the rock fragment.
[0073] Among them, the volume scale coefficient refers to the proportional relationship between the actual volume of the three-dimensional contour and the corresponding reference cube volume. In the embodiment of the present application, it can be understood as the ratio of the precise three-dimensional contour volume of the rock fragment to the matching reference cube volume. The volume scale coefficient is used to represent the conversion ratio between the three-dimensional contour and the reference cube volume.
[0074] Specifically, first, the three-dimensional contour parameters of the fragments are used to quickly construct a reference cube, and the resulting reference volume value is calculated. Then, the actual three-dimensional contour structure volume of the fragment is calculated through precise modeling. Then, the two volume calculation results are compared to obtain the numerical conversion ratio relationship between them, that is, the volume ratio coefficient. Finally, the reference cube volume is proportionally converted using the ratio coefficient to obtain a final fragment volume result that not only ensures the calculation speed but also improves the accuracy through proportional conversion. This scheme that integrates the rapid calculation of the reference cube with proportional conversion not only takes advantage of the reference cube estimation, but also improves the accuracy of the results through volume ratio correction, achieving the purpose of quickly and accurately calculating the volume of the fragments. The volume result provides data support for subsequent research and judgment of the distribution of gravel accumulation.
[0075] Reference Figure 3 , a binocular vision rock fragment size recognition system based on a portal frame is provided in an embodiment of the present application. The system comprises: an image acquisition module, a target detection module, an image processing module, and a rock fragment recognition module, wherein: An image acquisition module, used to acquire multiple frame images of the target vehicle driving through the portal frame; The target detection module is used to perform target detection on each frame image, determine the two-dimensional boundary box of the rock fragments in each frame image, multiple boundary pixel points and the pixel coordinates corresponding to each boundary pixel point; An image processing module, for determining, for each frame of image, a pixel coordinate difference between any two boundary pixel points based on each pixel coordinate, and calculating a pixel depth of each boundary pixel point according to each pixel coordinate difference and an average distance between each camera on the portal frame; The rock fragment recognition module is used to determine the three-dimensional contour of the rock fragment by combining the depth of each pixel and the two-dimensional bounding box, and calculate the fragment volume of the rock fragment in the frame image according to the contour parameters of the three-dimensional contour.
[0076] On the basis of the above embodiment, the target detection module is also used to input each frame image into a pre-trained target detection model, obtain the edge features of the rock fragments in each frame image, and extract multiple boundary pixel points in each edge feature; for each edge feature, based on the position of each boundary pixel point in the frame image, determine the pixel coordinates corresponding to each boundary pixel point; based on the pixel coordinates of each boundary pixel point, determine the minimum rectangular area surrounding the rock fragment, and use the minimum rectangular area as the two-dimensional bounding box of the rock fragment.
[0077] On the basis of the above embodiments, the target detection module is also used to obtain image samples corresponding to various rock fragments; perform data cleaning on each image sample to obtain an image training set corresponding to each image sample, and attach edge feature labels corresponding to the rock fragments in each image training set to each image training set; perform model training on the pre-constructed initial training model in combination with each image training set to obtain a target detection model after model training.
[0078] Based on the above embodiment, the image processing module is also used to select any two boundary pixel points in each frame image; based on the pixel coordinates of any two boundary pixel points, calculate the first difference in the horizontal direction and the second difference in the vertical direction of the two boundary pixel points; and use the first difference and the second difference as the pixel coordinate difference between any two boundary pixel points.
[0079] Based on the above embodiment, the image processing module is also used to obtain the focal length of each camera on the portal frame; multiply the focal length by the average distance between the cameras on the portal frame to obtain the depth coefficient; for each boundary pixel point, divide the depth coefficient by the pixel coordinate difference corresponding to the boundary pixel point to obtain the pixel depth of the boundary pixel point.
[0080] Based on the above embodiment, the rock fragment identification module is also used to match the boundary coordinates in the two-dimensional boundary box with the depth of each pixel to determine the three-dimensional coordinate point set of the rock fragment boundary; and fit each three-dimensional coordinate point set to obtain the three-dimensional contour of the rock fragment.
[0081] Based on the above embodiment, the rock fragment identification module is also used to construct a reference cube corresponding to the three-dimensional contour according to the maximum contour length, the maximum contour width and the maximum contour height, and calculate the volume of the reference cube; determine the volume ratio coefficient between the three-dimensional contour and the reference cube; multiply the volume of the minimum cube by the volume ratio coefficient to obtain the fragment volume of the rock fragment.
[0082] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0083] The present application also discloses an electronic device. Figure 4 , Figure 4The electronic device 400 may include: at least one processor 401 , at least one network interface 404 , a user interface 403 , a memory 405 , and at least one communication bus 402 .
[0084] The communication bus 402 is used to realize the connection and communication between these components.
[0085] The user interface 403 may include a display interface and a camera interface. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.
[0086] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0087] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 401 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface diagrams and applications, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, and it can be implemented separately through a chip.
[0088] Among them, the memory 405 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 405 includes a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may optionally also be at least one storage device located away from the aforementioned processor 401. Refer to Figure 4 The memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a binocular vision rock fragment size recognition method based on a portal frame.
[0089] exist Figure 4 In the electronic device 400 shown, the user interface 403 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 401 can be used to call the application program stored in the memory 405 for a binocular vision rock fragment block size recognition method based on a portal frame, and when executed by one or more processors 401, the electronic device 400 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.
[0090] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0091] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.
[0095] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.
[0096] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not recorded in the present disclosure. The description and examples are to be regarded as exemplary only.
Claims
1. A binocular vision rock fragment size recognition method based on a portal frame, characterized in that: include: Acquire multiple frame images of the target vehicle driving through the portal frame; Performing target detection on each of the frame images to determine a two-dimensional boundary box of rock fragments in each of the frame images, a plurality of boundary pixel points, and pixel coordinates corresponding to each of the boundary pixel points; For each of the frame images, based on each of the pixel coordinates, determine a pixel coordinate difference between any two of the boundary pixel points, and calculate a pixel depth of each of the boundary pixel points according to each of the pixel coordinate differences and an average distance between the cameras on the portal frame; The three-dimensional contour of the rock fragment is determined in combination with each of the pixel depths and the two-dimensional bounding box, and the fragment volume of the rock fragment in the frame image is calculated based on the contour parameters of the three-dimensional contour.
2. The binocular vision rock fragment size identification method based on portal frame according to claim 1 is characterized in that: The performing target detection on each of the frame images to determine a two-dimensional boundary box of rock fragments in each of the frame images, a plurality of boundary pixel points, and pixel coordinates corresponding to each of the boundary pixel points includes: Inputting each of the frame images into a pre-trained target detection model to obtain edge features of rock fragments in each of the frame images, and extracting a plurality of boundary pixel points in each of the edge features; For each of the edge features, based on the position of each of the boundary pixel points in the frame image, determining the pixel coordinates corresponding to each of the boundary pixel points; Based on the pixel coordinates of each of the boundary pixel points, a minimum rectangular area surrounding the rock fragment is determined, and the minimum rectangular area is used as a two-dimensional boundary box of the rock fragment.
3. The binocular vision rock fragment size identification method based on portal frame according to claim 2 is characterized in that: Before inputting each frame image into the pre-trained target detection model, the method further includes: Obtain image samples corresponding to various rock fragments; Performing data cleaning on each of the image samples to obtain an image training set corresponding to each of the image samples, and attaching edge feature labels corresponding to the rock fragments in each of the image training sets to each of the image training sets; The pre-constructed initial training model is trained in combination with each of the image training sets to obtain the target detection model after model training.
4. The binocular vision rock fragment size identification method based on portal frame according to claim 1 is characterized in that: The step of determining, for each of the frame images, a pixel coordinate difference between any two of the boundary pixel points based on the pixel coordinates, comprises: In each of the frame images, select any two boundary pixel points; Based on the pixel coordinates of the arbitrary two boundary pixel points, calculating a first difference in the horizontal direction and a second difference in the vertical direction of the two boundary pixel points; The first difference and the second difference are used as the pixel coordinate difference between any two of the boundary pixel points.
5. The binocular vision rock fragment size identification method based on portal frame according to claim 1 is characterized in that: The step of calculating the pixel depth of each of the boundary pixels according to the pixel coordinate differences and the average distance between the cameras on the portal frame comprises: Obtaining the focal length of each camera on the portal frame; Multiplying the focal length by the average distance between cameras on the gantry to obtain a depth coefficient; For each of the boundary pixel points, the depth coefficient is divided by the pixel coordinate difference corresponding to the boundary pixel point to obtain the pixel depth of the boundary pixel point.
6. The binocular vision rock fragment size identification method based on portal frame according to claim 1 is characterized in that: The combining the pixel depths and the two-dimensional bounding box to determine the three-dimensional contour of the rock fragment comprises: Matching the boundary coordinates in the two-dimensional boundary frame with each of the pixel depths to determine a three-dimensional coordinate point set of the rock fragment boundary; Each of the three-dimensional coordinate point sets is fitted to obtain the three-dimensional contour of the rock fragment.
7. The binocular vision rock fragment size identification method based on portal frame according to claim 1 is characterized in that: The contour parameters include a maximum contour length, a maximum contour width, and a maximum contour height. The calculating the fragment volume of the rock fragment in the frame image according to the contour parameters of the three-dimensional contour includes: Constructing a reference cube corresponding to the three-dimensional contour according to the maximum contour length, the maximum contour width, and the maximum contour height, and calculating the volume of the reference cube; determining a volume ratio coefficient between the three-dimensional profile and the reference cube; The volume of the minimum cube is multiplied by the volume ratio coefficient to obtain the fragment volume of the rock fragment.
8. A binocular vision rock fragment size recognition system based on a portal frame, characterized in that: The system comprises: An image acquisition module, used to acquire multiple frame images of the target vehicle driving through the portal frame; A target detection module is used to perform target detection on each frame image, determine a two-dimensional boundary box of rock fragments in each frame image, a plurality of boundary pixel points, and pixel coordinates corresponding to each boundary pixel point; An image processing module, configured to determine, for each of the frame images, a pixel coordinate difference between any two of the boundary pixel points based on each of the pixel coordinates, and calculate a pixel depth of each of the boundary pixel points according to each of the pixel coordinate differences and an average distance between each of the cameras on the portal frame; The rock fragment identification module is used to determine the three-dimensional contour of the rock fragment in combination with each of the pixel depths and the two-dimensional bounding box, and calculate the fragment volume of the rock fragment in the frame image according to the contour parameters of the three-dimensional contour.
9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the binocular vision rock fragment size recognition method based on a portal frame as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the binocular vision rock fragment size identification method based on the portal frame as described in any one of claims 1 to 7 is executed.