Material height measuring method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision application, in particular to a material height measuring method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an environment image in a cabin through a preset camera, the environment image comprising stacked materials; performing depth estimation on the environment image by using a preset monocular depth model to generate a relative depth image, and converting the relative depth image into an absolute depth image based on the internal reference of the camera by taking the actual size of the hatch and the pixel size of the hatch in the environment image as a reference; and calculating the height of the material based on the absolute depth image. According to the measurement method, only a monocular camera is needed, the installation process is simple, the system is easy to maintain, and compared with a multi-sensor fusion scheme, the method is lower in installation and maintenance cost.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of computer vision applications, and particularly to a method, device, electronic device, and storage medium for measuring the height of materials. Background Art

[0002] For purposes such as ensuring navigation safety and guaranteeing the quality of goods, it is necessary to monitor the coal loading process during shipping.

[0003] Although traditional stereo vision or multi-sensor solutions can provide good measurement effects, their high equipment costs and complex installation processes limit their practical applications in coal loading monitoring. In addition, multi-sensor solutions require regular maintenance to ensure the accuracy of calibration, increasing the operation difficulty. Summary of the Invention

[0004] The purpose of the present invention is to provide at least a method, device, electronic device, and storage medium for measuring the height of materials, which can at least solve the technical problem of the relatively high installation and maintenance costs of stereo vision or multi-sensor solutions in the prior art, and can at least achieve the effect of providing good measurement while reducing operation and maintenance costs.

[0005] To solve the above technical problems, at least one embodiment of the present application provides a method for measuring the height of materials, including:

[0006] Obtaining an environmental image inside the cabin through a preset camera, where the environmental image includes stacked materials;

[0007] Using a preset monocular depth model to perform depth estimation on the environmental image to generate a relative depth image, where the relative depth image is used to represent the relative depth relationship between different pixel points in the environmental image;

[0008] Based on the internal parameters of the camera, taking the actual size of the cabin opening and the pixel size of the cabin opening in the environmental image as a reference, converting the relative depth image into an absolute depth image;

[0009] Calculating the height of the material based on the absolute depth image.

[0010] At least one embodiment of the present application further provides a device for measuring the height of materials, including:

[0011] An environmental image acquisition module, configured to obtain an environmental image inside the cabin through a preset camera, where the environmental image includes stacked materials;

[0012] A relative depth image generation module, configured to perform depth estimation on the environmental image by using a preset monocular depth model to generate a relative depth image, where the relative depth image is used to represent the relative depth relationship between different pixel points in the environmental image;

[0013] An absolute depth image generation module, configured to convert the relative depth image into an absolute depth image based on the internal parameters of the camera, with the actual size of the hatch and the pixel size of the hatch in the environmental image as a reference;

[0014] A material height calculation module, configured to calculate the height of the material based on the absolute depth image.

[0015] At least one embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned method for measuring the height of the material.

[0016] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned method for measuring the height of the material is implemented.

[0017] Compared with traditional stereo vision or multi-sensor solutions which usually require expensive hardware devices, such as multiple high-precision cameras, laser rangefinders, etc., the method for measuring the height of the material provided by the embodiments of the present application can achieve depth estimation and height measurement only through a preset single camera, significantly reducing the equipment cost. For large-scale application scenarios such as coal ship loading monitoring, the cost-effectiveness is particularly obvious. In addition, multi-sensor solutions usually require complex installation and calibration processes to ensure the coordination and data accuracy between sensors. While the method of the present application only needs to install one camera and pre-configure a monocular depth model, greatly simplifying the installation process and reducing the installation time and labor cost. In subsequent operations, based on computer vision and deep learning technologies, once the model is trained and deployed to actual applications, it usually does not require frequent maintenance and calibration, greatly reducing the operation difficulty and maintenance cost. Therefore, the method for measuring the height of the material of the present application provides good measurement results while reducing the operation and maintenance costs.

[0018] In some optional embodiments, before performing depth estimation on the environmental image by using the preset monocular depth model, it further includes:

[0019] The initial monocular depth model is fine-tuned using a preset internal cabin dataset until the preset training standard is reached, and the monocular depth model is obtained, where the internal cabin dataset includes at least one of a training set with different illuminations inside the cabin and a training set of the pile states of different types of materials.

[0020] In this embodiment, through fine-tuning training, the monocular depth model can better adapt to the specific environment and conditions inside the cabin. There may be complex lighting conditions, different types of materials, and different pile states inside the cabin, and these factors may all affect the accuracy of depth estimation. Using the internal cabin dataset for fine-tuning can enable the model to more accurately capture these features, thereby improving the accuracy of depth estimation.

[0021] In some alternative embodiments, based on the internal parameters of the camera, taking the actual size of the cabin opening and the pixel size of the cabin opening in the environmental image as a reference, converting the relative depth image into an absolute depth image includes:

[0022] Extract the position and shape information of the cabin opening in the environmental image, and calculate the pixel size of the cabin opening based on the position and shape information of the cabin opening;

[0023] Use the actual size and the pixel size to calculate the absolute distance from the cabin opening to the camera;

[0024] According to the ratio of the absolute distance to the depth value from the cabin opening to the camera in the relative depth image, obtain a depth factor;

[0025] Adjust the relative depth image based on the depth factor to obtain the absolute depth image.

[0026] In this embodiment, as a prominent and fixed structural feature inside the cabin, the position and shape information of the cabin opening are usually easier to distinguish in the environmental image. Using this information as a reference benchmark can ensure the accuracy and stability during the depth conversion process. By calculating the proportional relationship between the actual size of the cabin opening and its pixel size in the environmental image, the absolute distance from the camera to the cabin opening can be accurately obtained. This step provides a reliable basis for subsequent depth conversion.

[0027] In some alternative embodiments, the extraction of the position and shape information of the cabin opening in the environmental image includes:

[0028] Use the YOLOv8 model to extract the position and shape information of the cabin opening in the environmental image.

[0029] In this embodiment, YOLOv8 adopts a deeper convolutional neural network structure and introduces technologies such as deformable convolution and multi-scale feature fusion. These technologies enable the model to capture the detailed features of images more accurately, thereby improving the detection accuracy. At the same time, YOLOv8 automatically adjusts the scale and ratio of the anchor boxes through an adaptive anchor box mechanism to adapt to the shape and size of the targets in different scenarios, which helps to locate the hatch more accurately.

[0030] In some alternative embodiments, calculating the absolute distance from the hatch to the camera using the actual size and the pixel size includes:

[0031] Calculating the absolute distance d from the hatch to the camera according to the following calculation formula marker :

[0032]

[0033] where L real is the actual height of the hatch, L pixel is the pixel height of the hatch in the image, and f is the camera focal length of the camera.

[0034] In this embodiment, the actual height of the hatch is usually known or can be obtained through simple measurement. The pixel height can be automatically extracted by image processing technology without manual intervention. The camera focal length of the camera is also known and is usually calibrated when the camera leaves the factory. Therefore, this method is not limited by the type of hatch, the type of material, or the lighting conditions. As long as the camera can clearly capture the hatch and the actual height of the hatch is known, the calculation can be performed. It avoids complex model fitting or iterative optimization processes and improves the accuracy of the calculation results.

[0035] In some alternative embodiments, adjusting the relative depth image based on the depth factor to obtain the absolute depth image includes:

[0036] Multiplying the depth value corresponding to each pixel in the relative depth image by the depth factor to obtain the absolute depth image.

[0037] In this embodiment, the conversion from relative depth to absolute depth is achieved through simple mathematical operations (i.e., multiplying by the depth factor), avoiding complex image processing and calculation processes. This directness makes the conversion process efficient and fast, suitable for scenarios that require real-time processing of depth information.

[0038] In some alternative embodiments, calculating the height of the material based on the absolute depth image includes:

[0039] Extract the position information of the top of the material and the bottom of the coal chute in the absolute depth image;

[0040] Based on the position information of the top of the material and the bottom of the coal chute, calculate the vertical distance difference between the top of the material and the bottom of the coal chute to obtain the height of the material.

[0041] In this embodiment, by using the absolute depth image, the depth information of the top of the material and the bottom of the coal chute can be accurately obtained, so as to calculate an accurate height value, reducing the errors caused by factors such as perspective changes, lighting conditions, or image noise.

[0042] In summary, the method, device, electronic device, and storage medium for measuring the height of the material provided by this application have at least the following beneficial effects:

[0043] 1. High-precision depth estimation:

[0044] By combining the monocular depth model and the self-made internal cabin dataset, this method is customized for the specific environment inside the cabin for training, so as to achieve high-precision depth estimation in the scenario of measuring the height of coal materials. Using the depth factor to convert the relative depth into absolute depth effectively improves the accuracy of depth measurement. Compared with traditional methods, the average error of depth measurement is reduced by 20%. Under specific lighting conditions, the depth estimation accuracy of the system is improved to more than 95%.

[0045] 2. Strong real-time performance:

[0046] Using YOLOv8 for cabin hatch detection and segmentation, it can quickly identify the cabin hatch area and segment its edge, and then combine the deep learning model to estimate the depth information. The overall system can meet the requirements of real-time monitoring and achieve dynamic monitoring of the height of coal materials. This system can monitor at a speed of 25 frames per second, meeting the requirements of rapid changes during the coal loading process.

[0047] 3. Strong adaptability:

[0048] The self-made internal cabin dataset covers different lighting conditions and the states of coal material piles. Therefore, the trained depth estimation model can adapt to the changing environmental conditions inside the cabin, including the interference of factors such as light changes and coal dust.

[0049] 4. Easy installation and maintenance:

[0050] Since only a monocular camera is required, the installation process is simple and the system is also easy to maintain. Compared with the multi-sensor fusion scheme, the installation and maintenance costs of this method are lower, and at the same time, the invasive impact on cabin equipment is reduced. Brief Description of the Drawings

[0051] One or more embodiments are exemplarily illustrated by pictures in the corresponding accompanying drawings, and these exemplary illustrations do not constitute a limitation on the embodiments.

[0052] Figure 1 is the flowchart of the method for measuring the height of materials provided by an embodiment of the present application Figure 1 ;

[0053] Figure 2 is the system architecture diagram of a system for measuring the height of materials provided by an embodiment of the present application;

[0054] Figure 3 is the flowchart of the method for measuring the height of materials provided by another embodiment of the present application Figure 2 ;

[0055] Figure 4 is the schematic diagram of a device for measuring the height of materials provided by another embodiment of the present application. Detailed implementation manners

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are provided for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions required to be protected by the present application can still be implemented. The following division of each embodiment is for convenience of description and should not constitute any limitation on the specific implementation manner of the present application. Each embodiment can be combined and cross-referenced with each other on the premise of no contradiction.

[0057] To solve the above technical problem of the relatively high installation and maintenance costs of using stereo vision or multi-sensor solutions, the present invention proposes a method for measuring the height of materials. The implementation details of the method for measuring the height of materials in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing the solution.

[0058] Embodiment 1:

[0059] The method for measuring the height of materials in this embodiment can be applied to an electronic device with communication, computing, and data storage capabilities. Its specific process can be as Figure 1 shown and includes:

[0060] Step 110, obtaining an environmental image inside the cabin through a preset camera, where the environmental image includes stacked materials;

[0061] In this embodiment, the camera is installed at a suitable position outside or inside the cabin so as to clearly capture the situation of the material stacking. The camera should have an appropriate resolution and viewing angle to ensure that it can cover the entire interior of the cabin and capture the details of the material stacking. After the camera is installed, adjust its angle and focal length to ensure that it can clearly capture the situation of the material stacking, and set appropriate shooting parameters (such as exposure time, white balance, etc.) to obtain high-quality environmental images.

[0062] Step 120: Use a preset monocular depth model to perform depth estimation on the environmental image to generate a relative depth image, where the relative depth image is used to represent the relative depth relationship between different pixel points in the environmental image.

[0063] In this embodiment, the monocular depth model is a machine learning model that can estimate depth information from a single image. In this embodiment, a suitable monocular depth model is selected, such as a depth estimation model based on a convolutional neural network (CNN). The model is trained using a large number of training images containing depth information so that it can learn the mapping relationship from the environmental image to the depth image. By training this model, it can learn the relative depth relationship between different pixel points from the environmental image.

[0064] Input the environmental image obtained in step 110 into the trained monocular depth model to generate a relative depth image. The relative depth image is a special image representation method that does not directly provide the actual physical distance of the objects in the scene, but represents the relative distance relationship between different pixel points or regions in the image. For example, use grayscale values to quantify the depth relationship, where the larger the grayscale value, the farther or closer the distance. Or output a depth map through the monocular depth model, where the value of each pixel represents the distance from the pixel point to the camera, or output a depth probability distribution or other forms of depth representation. The specific output form of the relative depth image is selected according to actual needs and is not limited here.

[0065] Step 130: Based on the internal parameters of the camera, and taking the actual size of the cabin opening and the pixel size of the cabin opening in the environmental image as a reference, convert the relative depth image into an absolute depth image.

[0066] It can be understood that since the relative depth image only represents the relative depth relationship between different pixel points and does not have specific depth units (such as meters, centimeters, etc.), it is necessary to convert it into an absolute depth image with specific depth units through a certain conversion method.

[0067] In this embodiment, the internal parameters of the camera (such as focal length, principal point, etc.) and the actual size of the cabin opening and its pixel size in the environmental image are used as a reference for conversion. For example:

[0068] Obtain the internal parameter information of the camera, including focal length, principal point, etc.;

[0069] Measure the actual size of the cabin hatch and record its pixel size in the environmental image;

[0070] According to the internal parameter information of the camera and the proportional relationship between the actual size and the pixel size of the cabin hatch, calculate the actual depth value corresponding to each pixel;

[0071] Multiply the depth value of each pixel point in the relative depth image by the corresponding scale factor to obtain the absolute depth image.

[0072] Step 140, calculate the height of the material based on the absolute depth image.

[0073] In this embodiment, the height of the material is calculated based on the absolute depth image. Since the absolute depth image contains the actual depth information of each pixel point, the height of the material can be calculated by extracting the depth information of the top of the material and the reference plane. For example: manually or automatically mark the positions of the top of the material and the reference plane in the absolute depth image, and extract the depth values corresponding to these two positions; calculate the difference between these two depth values, which is the height of the material.

[0074] In summary, the method for measuring the height of the material provided in this embodiment, compared with the traditional stereo vision or multi-sensor solutions that usually require expensive hardware devices, such as multiple high-precision cameras, laser rangefinders, etc., can achieve depth estimation and height measurement only through a preset single camera, significantly reducing the equipment cost. For large-scale application scenarios such as coal ship loading monitoring, the cost-effectiveness is particularly obvious. In addition, multi-sensor solutions usually require complex installation and calibration processes to ensure the coordination and data accuracy between sensors. However, the method of this application only needs to install one camera and pre-configure a monocular depth model, greatly simplifying the installation process and reducing the installation time and labor cost. In subsequent operations, based on computer vision and deep learning technologies, once the model is trained and deployed to actual applications, it usually does not require frequent maintenance and calibration, greatly reducing the operation difficulty and maintenance cost. Therefore, the method for measuring the height of the material of this application provides good measurement results while reducing the operation and maintenance costs.

[0075] In some alternative embodiments, before performing depth estimation on the environmental image using the preset monocular depth model, it further includes: fine-tuning and training the initial monocular depth model using a preset dataset of the interior of the cabin until reaching the preset training standard to obtain the monocular depth model, where the dataset of the interior of the cabin includes at least one of a training set with different illuminations inside the cabin and a training set of the stockpile states of different types of materials.

[0076] Specifically, the change in environmental illumination will affect the brightness and contrast of the image, thereby affecting the performance of the monocular depth model. Overly bright or dark illumination conditions may both lead to loss of image information or an increase in noise. Therefore, in the depth estimation task, maintaining the consistency of illumination can significantly improve the accuracy and stability of the model.

[0077] In addition, the diversity of materials and the cabin structure will also affect the performance of the monocular depth model. For example, different types of materials have different surface textures, colors, and shapes, and these differences will affect the performance of the depth estimation model. The structure, size, and shape of the cabin will also affect the extraction of depth information and the calculation of the material height. Therefore, for different types of materials and cabin structures, different depth estimation models may need to be adopted or the model parameters adjusted to improve the accuracy. By training the model and adjusting the parameters for specific materials and cabin structures, the performance of depth estimation and material height calculation can be optimized.

[0078] The following is the preparation process of the dataset of the interior of the cabin and the specific process of model training:

[0079] 1. Dataset preparation

[0080] Illumination conditions: Collect interior images of the cabin taken at different time periods (such as morning, noon, evening, etc.) and different weather conditions (such as sunny, cloudy, rainy, etc.) to ensure that the model can adapt to various illumination changes.

[0081] Material types: Collect images of the stockpile states of different types of materials (such as coal, ore, grain, etc.). These materials may have different colors, textures, and shapes, so the dataset should be as rich as possible.

[0082] 2. Label the dataset

[0083] Label the collected image dataset. In the depth estimation task, pixel-level labeling is usually not required, but depth information labeling is needed. This can be achieved in the following ways:

[0084] Use a laser scanner or a depth camera: While collecting images, use a laser scanner or a depth camera to obtain the depth information of the scene. Then align this depth information with the images to form labeled data.

[0085] Manual annotation: If it is not allowed to use a laser scanner or a depth camera, the depth information of some key points can be manually annotated. These key points can be the top, bottom, or edge of the material pile, etc. Then, the depth of the entire scene can be inferred using this key point information.

[0086] 3. Initialize the monocular depth model

[0087] Select a pre-trained monocular depth model as the starting point. For example, the model can be trained on large datasets (such as KITTI, Make3D, etc.) and can handle depth estimation tasks for general scenes.

[0088] 4. Fine-tuning training

[0089] Use the in-cabin dataset to fine-tune the initial monocular depth model. This process usually includes the following steps:

[0090] 4.1. Data preprocessing: Preprocess the collected images, such as cropping, scaling, normalization, etc., to ensure that they match the requirements of the model input.

[0091] 4.2. Model configuration: Configure the training parameters of the model, such as the learning rate, batch size, number of iterations, etc., according to the size of the dataset and the computing resources.

[0092] 4.3. Training process: Input the preprocessed images and the corresponding depth annotations into the model for training. During the training process, the model will continuously adjust its parameters to minimize the difference between the predicted depth and the true depth.

[0093] 4.4. Validation and evaluation: During the training process, regularly use the validation set to evaluate the performance of the model. If the performance of the model reaches the preset training criteria (such as mean absolute error, mean squared error, etc.), stop the training and save the model to obtain a monocular depth model suitable for the in-cabin monitoring environment.

[0094] In this embodiment, through fine-tuning training, the monocular depth model can better adapt to the specific environment and conditions inside the cabin. There may be complex lighting conditions, different types of materials, and different material pile states inside the cabin, and these factors may all affect the accuracy of depth estimation. Using the in-cabin dataset for fine-tuning can enable the model to more accurately capture these features, thereby improving the accuracy of depth estimation.

[0095] In some alternative embodiments, converting the relative depth image into an absolute depth image based on the internal parameters of the camera, with the actual size of the hatch and the pixel size of the hatch in the environmental image as a reference, includes: extracting the position and shape information of the hatch in the environmental image, and calculating the pixel size of the hatch based on the position and shape information of the hatch; using the actual size and the pixel size to calculate the absolute distance from the hatch to the camera; obtaining a depth factor according to the ratio of the absolute distance to the depth value of the hatch to the camera in the relative depth image; and adjusting the relative depth image based on the depth factor to obtain the absolute depth image.

[0096] In this embodiment, as a prominent and fixed structural feature inside the cabin, the position and shape information of the hatch are usually easier to distinguish in the environmental image. Using this information as a reference benchmark can ensure the accuracy and stability during the depth conversion process. By calculating the proportional relationship between the actual size of the hatch and its pixel size in the environmental image, the absolute distance from the camera to the hatch can be accurately obtained. This step provides a reliable basis for subsequent depth conversion.

[0097] In some alternative embodiments, extracting the position and shape information of the hatch in the environmental image includes: using the YOLOv8 model to extract the position and shape information of the hatch in the environmental image.

[0098] Specifically, the YOLOv8 model is used for joint task processing to achieve the combination of object detection and image segmentation. The YOLOv8 model first uses its object detection ability to identify the hatch area in the image. This step can accurately locate the boundary of the hatch and provide a clear reference point. The detected hatch area is marked by a bounding box. Then, on this basis, the segmentation function of YOLOv8 is further used to accurately segment the edge of the hatch. This processing method ensures that even in a complex environment, the accurate position and shape information of the hatch can be accurately extracted.

[0099] In this embodiment, YOLOv8 adopts a deeper convolutional neural network structure and introduces techniques such as deformable convolution and multi-scale feature fusion. These techniques enable the model to capture the detailed features of the image more accurately, thereby improving the detection accuracy. At the same time, YOLOv8 automatically adjusts the scale and ratio of the anchor box through an adaptive anchor box mechanism to adapt to the shape and size of the target in different scenarios, which helps to more accurately locate the hatch.

[0100] Convert the pixel information in the image into absolute distance through the camera intrinsic parameters. First, calculate the absolute distance from the hatch of the cabin to the camera using the known size and pixel length of the hatch, and then convert the relative depth map into an absolute depth map by calculating the depth factor, so as to obtain the absolute distance information of the cabin and the materials inside it.

[0101] In some alternative embodiments, calculating the absolute distance from the hatch of the cabin to the camera using the actual size and the pixel size includes: calculating the absolute distance d from the hatch of the cabin to the camera according to the following calculation formula marker :

[0102]

[0103] where L real is the actual height of the hatch of the cabin, L pixel is the pixel height of the hatch of the cabin in the image, and f is the focal length of the camera.

[0104] In this embodiment, the actual height of the hatch of the cabin is usually known or can be obtained through simple measurement. The pixel height can be automatically extracted by image processing technology without manual intervention. The focal length of the camera is also known and is usually calibrated when the camera leaves the factory. Therefore, this method is not limited by the type of cabin, the type of material, or the lighting conditions. As long as the camera can clearly capture the hatch of the cabin and the actual height of the hatch of the cabin is known, the calculation can be performed. It avoids the complex model fitting or iterative optimization process and improves the accuracy of the calculation result.

[0105] In some alternative embodiments, adjusting the relative depth image based on the depth factor to obtain the absolute depth image includes: multiplying the depth value corresponding to each pixel in the relative depth image by the depth factor to obtain the absolute depth image.

[0106] Specifically, by calculating the depth factor s, convert the relative depth image generated by the model into an absolute depth image. The calculation formula of the depth factor s is:

[0107]

[0108] where d realmarker is the depth value from the cabin to the camera in the relative depth image.

[0109] Next, apply the depth factor s, multiply each depth value in the relative depth map by the depth factor s to obtain the adjusted absolute depth:

[0110] d abs (x,y) = s × d rel (x,y)

[0111] Among them, d abs (x, y) is the adjusted absolute depth image, and d rel (x, y) is the relative depth of a certain pixel in the relative depth image.

[0112] In this embodiment, the conversion from relative depth to absolute depth is achieved through simple mathematical operations (i.e., multiplying by the depth factor), avoiding complex image processing and calculation processes. This directness makes the conversion process efficient and fast, suitable for scenarios that require real-time processing of depth information.

[0113] In some alternative embodiments, calculating the height of the material based on the absolute depth image includes: extracting the position information of the top of the material and the bottom of the coal chute in the absolute depth image; calculating the vertical distance difference between the top of the material and the bottom of the coal chute based on the position information of the top of the material and the bottom of the coal chute to obtain the height of the material.

[0114] Specifically, the height of the material is calculated using the following calculation formula:

[0115] H material =(Z bottom -Z top )

[0116] Among them, Z bottom is the distance from the coal material to the camera, and Z top is the distance from the bottom of the coal chute to the camera. The bottom of the coal chute is the reference plane described in step 140.

[0117] In this embodiment, by using the absolute depth image, the depth information of the top of the material and the bottom of the coal chute can be accurately obtained, thereby calculating an accurate height value, reducing errors caused by factors such as viewing angle changes, lighting conditions, or image noise.

[0118] Embodiment 2:

[0119] Based on the content of the above embodiment, an application example of the method for measuring the height of the material is provided in this embodiment. The method for measuring the height of the material in this embodiment is applied to a material height measurement system as shown in Figure 2 , and the specific measurement flowchart can be as shown in Figure 3 :

[0120] 1. Configure the experimental environment:

[0121] Use a Hikvision intelligent zoom bullet network camera (model: DS-2CD3646FWDA3 / F-LZS, focal length: 2.7-12mm), with a resolution of 2560*1440. The camera is installed above the cabin, tilted slightly downward to cover the entire cabin area. The camera is connected to an edge computer (model: shown as T808) running the ubuntu20.04 operating system through a network interface. The CPU of the edge computer is: 8-core Arm Cortex-A78AE v8.2 64-bit. The GPU is: 1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores (Max 912MHz). The running memory is: 16GB 128-bit LPDDR5 102.4GB / s.

[0122] 2. Data acquisition:

[0123] The camera acquires image data inside the cabin at a rate of 25 frames per second. To meet the requirements of the model input and reduce the computational load at the same time, each acquired image is normalized to the range [0,1] and resized to fit the input format of the model (such as the input size is 640x480).

[0124] 3. Model training:

[0125] The pre-trained weights of the Depth Anything model (or monocular depth model) are loaded into the PyTorch framework. Fine-tuning is performed using a self-made dataset of the interior of the cabin, which includes 5000 images collected under different lighting conditions and coal loading states. The training parameters are set as follows: learning rate 0.001, batch size 32, number of training epochs 50. The mean squared error (MSE) is used as the loss function, and the model is trained using the Adam optimizer.

[0126] 4. Depth estimation:

[0127] The processed image is input into the Depth AnythingS model, and the generated relative depth map is denoised and smoothed through post-processing steps to improve the accuracy of depth estimation.

[0128] 5. Object detection and segmentation:

[0129] The processed image is input into the YOLOv8 model, and the model identifies the cabin hatch area and further segments the edge of the cabin hatch within the bounding box. The output results include the pixel positions of the cabin hatch and the pixel lengths of the edge contours.

[0130] 6. Depth Conversion: By extracting the focal length of the camera (e.g., f = 6 mm), use the formula to calculate the absolute distance from the hatch to the camera:

[0131]

[0132] Calculate the depth factor and adjust the relative depth map to convert the relative depth value of each pixel to absolute depth.

[0133] 7. Height Measurement: Identify the positions of the top of the coal material and the bottom of the coal chute in the absolute depth map, calculate the vertical distance between the two, and output the height information of the coal material.

[0134] The method for measuring the height of the material in this embodiment has at least the following advantages:

[0135] 1. High-precision depth estimation:

[0136] By combining the Depth Anything model and the self-made internal cabin dataset, this method is customized and trained for the specific environment inside the cabin, thus achieving high-precision depth estimation in the scenario of measuring the height of coal material. Using the depth factor to convert relative depth to absolute depth effectively improves the accuracy of depth measurement. Compared with traditional methods, the average error of depth measurement is reduced by 20%. Under specific lighting conditions, the depth estimation accuracy of the system is improved to over 95%.

[0137] 2. Strong real-time performance:

[0138] Using YOLOv8 for hatch detection and segmentation in the cabin, it can quickly identify the hatch area and segment its edge, and then combine with the deep learning model to estimate the depth information. The overall system can meet the requirements of real-time monitoring and achieve dynamic monitoring of the height of coal material. This system can monitor at a speed of 25 frames per second, meeting the requirements of rapid changes during the coal loading process.

[0139] 3. Strong adaptability:

[0140] The self-made internal cabin dataset covers different lighting conditions and the states of coal material piles. Therefore, the trained depth estimation model can adapt to the changing environmental conditions inside the cabin, including factors such as light changes and coal dust interference.

[0141] 4. Easy installation and maintenance:

[0142] Since only a monocular camera is required, the installation process is simple and the system is also easy to maintain. Compared with the multi-sensor fusion scheme, this method has lower installation and maintenance costs and reduces the invasive impact on cabin equipment.

[0143] Example Three:

[0144] Another embodiment of the present application relates to a device for measuring the height of materials. The implementation details of the device for measuring the height of materials in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution. The schematic diagram of the device for measuring the height of materials in this embodiment can be as Figure 4 shown, including an environmental image acquisition module 410, a relative depth image generation module 420, an absolute depth image generation module 430, and a material height calculation module 440.

[0145] The environmental image acquisition module 410 is used to acquire the environmental image inside the cabin through a preset camera. Among them, the environmental image includes stacked materials;

[0146] The relative depth image generation module 420 is used to perform depth estimation on the environmental image using a preset monocular depth model to generate a relative depth image, where the relative depth image is used to represent the relative depth relationship between different pixel points in the environmental image;

[0147] The absolute depth image generation module 430 is used to convert the relative depth image into an absolute depth image based on the internal parameters of the camera, with the actual size of the cabin opening and the pixel size of the cabin opening in the environmental image as the reference;

[0148] The material height calculation module 440 is used to calculate the height of the material based on the absolute depth image.

[0149] It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovative part of the present application, units that are not closely related to solving the technical problems proposed by the present application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0150] Embodiment 4:

[0151] Another embodiment of the present application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for measuring the height of materials in the above embodiments.

[0152] Among them, the memory and the processor are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be a single component or multiple components, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0153] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store the data used by the processor when executing operations.

[0154] Embodiment Five:

[0155] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the method embodiment for measuring the height of the above-mentioned material.

[0156] That is, those skilled in the art can understand that all or part of the steps in implementing the above-mentioned embodiment methods can be completed by instructing relevant hardware through a program. This program is stored in a storage medium, including several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs that can store program codes.

[0157] Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in practical applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A method for measuring material height, characterized in that: include: Acquire an environmental image of the interior of the cabin through a preset camera, wherein the environmental image includes stacked materials; Using a preset monocular depth model to perform depth estimation on the environment image to generate a relative depth image, wherein the relative depth image is used to characterize the relative depth relationship between different pixel points in the environment image; Based on the internal parameters of the camera, the relative depth image is converted into an absolute depth image with the actual size of the hatch and the pixel size of the hatch in the environment image as a reference; Based on the absolute depth image, the height of the material is calculated.

2. The material height measurement method according to claim 1, characterized in that: Before the depth estimation of the environment image is performed using a preset monocular depth model, the method further includes: The initial monocular depth model is fine-tuned and trained using a preset cabin interior data set until a preset training standard is reached to obtain the monocular depth model, wherein the cabin interior data set includes at least one of a training set for different lighting conditions in the cabin and a training set for stockpile states of different types of materials.

3. The material height measurement method according to claim 1, characterized in that: The converting the relative depth image into an absolute depth image based on the internal parameters of the camera and taking the actual size of the hatch and the pixel size of the hatch in the environment image as a reference includes: Extracting the position and shape information of the hatch opening in the environment image, and calculating the pixel size of the hatch opening based on the position and shape information of the hatch opening; Using the actual size and the pixel size, calculate the absolute distance from the hatch to the camera; Obtaining a depth factor according to a ratio of the absolute distance to a depth value from the hatch to the camera in the relative depth image; The relative depth image is adjusted based on the depth factor to obtain the absolute depth image.

4. The material height measurement method according to claim 3, characterized in that: The extracting the position and shape information of the hatch opening in the environment image includes: The YOLOv8 model is used to extract the position and shape information of the hatch in the environment image.

5. The material height measurement method according to claim 3, characterized in that: The step of calculating the absolute distance from the hatch to the camera by using the actual size and the pixel size includes: According to the following calculation formula, the absolute distance d from the hatch to the camera is calculated: marker : Among them, L real is the actual height of the hatch, L pixel is the pixel height of the hatch in the image, and f is the camera focal length of the camera.

6. The material height measurement method according to claim 3, characterized in that: The adjusting the relative depth image based on the depth factor to obtain the absolute depth image includes: The depth value corresponding to each pixel in the relative depth image is multiplied by the depth factor to obtain the absolute depth image.

7. The material height measurement method according to any one of claims 1 to 6, characterized in that: The calculating the height of the material based on the absolute depth image includes: Extracting position information of the top of the material and the bottom of the coal chute in the absolute depth image; Based on the position information of the top of the material and the bottom of the coal chute, the vertical distance difference between the top of the material and the bottom of the coal chute is calculated to obtain the height of the material.

8. A material height measuring device, characterized in that: include: An environmental image acquisition module, used to acquire an environmental image of the interior of the cabin through a preset camera, wherein the environmental image includes stacked materials; A relative depth image generation module, used to perform depth estimation on the environment image using a preset monocular depth model to generate a relative depth image, wherein the relative depth image is used to characterize the relative depth relationship between different pixel points in the environment image; an absolute depth image generating module, configured to convert the relative depth image into an absolute depth image based on the internal parameters of the camera and taking the actual size of the hatch and the pixel size of the hatch in the environment image as a reference; The material height calculation module is used to calculate the height of the material based on the absolute depth image.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the material height measurement method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the material height measurement method according to any one of claims 1 to 7 is implemented.

Images