A material identification method based on image processing

Abstract

The present application relates to the field of image processing, and particularly relates to a material recognition method based on image processing, obtaining material image information under different visual angles to be recognized; determining material image information under one visual angle as standard image, performing image registration on material image information collected under other different visual angles and the standard image information, obtaining image registration result, and determining the perspective transformation matrix of material image information under different visual angles; based on the perspective transformation matrix, transforming material image information under other different visual angles to be recognized to the visual angle of the standard image, and performing image splicing to obtain spliced image information. That is, the scheme of the present application can perform image processing on material image, avoiding quality problems such as distortion and incompleteness of the image.

Description

Technical Field

[0001] This invention relates to the field of building material identification, and more specifically to a material identification method based on image processing. Background Technology

[0002] During the construction process, it is necessary to transport the materials required for the project to the construction site by transport vehicles. After the materials arrive at the construction site, they need to be counted. Common materials include steel bars, square timber, etc.

[0003] The identification of the aforementioned materials involves directly capturing images of the stacked materials to achieve the purpose of material detection. However, due to factors such as excessive building materials or uneven stacking, the identification effect of some materials may be poor during the camera's image acquisition process. For example, if building materials with edges are tilted outwards, or if some building materials are not stacked neatly, or if the camera has issues such as a single shooting angle or distance, the captured images may have quality problems such as distortion, unclearness, or incompleteness. This can lead to some building materials being missed, resulting in substandard detection results and inaccurate material identification. Summary of the Invention

[0004] The purpose of this invention is to provide a material identification method based on image processing, which solves the problem of material omission caused by image distortion, unclearness, or incompleteness.

[0005] This invention provides a material identification method based on image processing, comprising the following steps:

[0006] Acquire image information of the material to be identified from different perspectives;

[0007] Image registration results are obtained by matching the building material images collected from different perspectives with the front view of the building material collected at the current fixed position of the camera.

[0008] Based on the image registration results, the perspective transformation matrix required to transform the front view of the building material captured at the current fixed position of the camera to the corresponding viewpoints of the front view of the building material captured at different positions is obtained.

[0009] Based on the perspective transformation matrix, the image information of the material to be identified from different perspectives is transformed to the same perspective, and the images are stitched together to obtain the stitched image information.

[0010] Obtain the depth information corresponding to the stitched image information;

[0011] The material is identified based on the acquired stitched image information and depth information.

[0012] Preferably, acquiring the material image information to be identified from different perspectives specifically involves setting up at least two cameras to capture the material image information from different perspectives.

[0013] Preferably, the SIFT feature point image registration method is used to obtain the registration result of the material image under any viewpoint other than the standard viewpoint image and the current standard viewpoint image. From the current registration result, four pairs of feature points after image registration are randomly selected to obtain the perspective transformation matrix between the two images. The image used for image registration with the standard viewpoint image is transformed to belong to the same spatial viewpoint as the current standard viewpoint image through the obtained perspective transformation matrix.

[0014] Preferably, the specific process for identifying materials based on the acquired stitched image information and depth information is as follows:

[0015] A Mask R-CNN network model is constructed, which includes a ResNet network. The ResNet network is optimized to obtain an optimized Mask R-CNN network model. The optimized Mask R-CNN network model is then trained to obtain a trained optimized Mask R-CNN network model.

[0016] The stitched image information and the corresponding depth information are input into the trained and optimized MaskR-CNN network model to identify the material and obtain the recognition result.

[0017] The specific process of optimizing the ResNet network is as follows:

[0018] Obtain the training dataset; input the training dataset into the ResNet network for training, and optimize it using the loss function to obtain the trained ResNet network;

[0019] The process of obtaining the loss function is as follows:

[0020] Obtain a grayscale image of the stitched material image data, extract the image region of any single material category, extract the centroid coordinates of the image region, and obtain the distance between the center coordinates and the centroid coordinates of the grayscale image;

[0021] Obtain the imaging quality assessment value for any single material category;

[0022] An adjustment index is obtained based on the distance value and the imaging quality assessment value; a loss function is constructed based on the adjustment index, the true label value of the image of a single material, and the true label value of the depth image.

[0023] Preferably, the loss function is:

[0024]

[0025] Among them, F j Image A of the current material j The imaging quality assessment value corresponding to each material category, p(K) j ) is the Ath j The true label value of each material category on the material image during the current ResNet network training process, q(K′) j ) is the Ath j The predicted label values ​​of each material category on the material image during the current ResNet network training process, p(L) j ) is the Ath j The true label value of each material category on the depth image during the current ResNet network training process, q(L′) j ) is the Ath j The predicted label values ​​of each material category on the depth image during the current ResNet network training process, where H is the number of all categories, m() is the normalization function, and j is the material number.

[0026] Preferably, the adjustment index is:

[0027] F j =exp(-D j )*E j *G

[0028] Among them, D j The coordinates of the center point of the current grayscale image and the Ath j The distance between the centroid coordinates of each material category, E j Indicates the Ath element in the current grayscale image. j The imaging quality assessment value for each material category, where exp() is an exponential function with the natural number e as the base, and G is a hyperparameter.

[0029] Preferably, the specific process for obtaining the imaging quality assessment value for any single material category is as follows:

[0030] Obtain the information entropy of a grayscale image;

[0031] Calculate the mean grayscale value of the image region corresponding to each material category, and obtain the mean grayscale value of all material categories; calculate the Ath material category... j The absolute value of the difference between the grayscale mean of each material category and the mean is used to obtain the ratio of the absolute value of the difference to the mean. The product of this ratio and the information entropy value is taken as the Ath value. j Imaging quality assessment values ​​for each material category.

[0032] Preferably, the Mask R-CNN network model further includes an FPN network, an RPN neural network, and a RoI Align model.

[0033] The beneficial effects of this invention are as follows:

[0034] The solution of this invention acquires material images from different perspectives and performs image registration and image stitching operations on the acquired material images. This avoids problems with the quality of the acquired images due to the disorderly placement of materials, thus providing important data support for material identification in the process.

[0035] Meanwhile, the solution of this invention optimizes the ResNet network by constructing a Mask R-CNN network model. That is, by combining image data and depth image data to construct a loss function, the importance of the required dependent data can be adjusted in real time according to the imaging quality of the image data, thereby optimizing the network and enabling accurate material identification. Attached Figure Description

[0036] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the steps of an image processing-based material identification method according to the present invention.

[0038] Figure 2 This refers to image information of materials used in construction projects. Detailed Implementation

[0039] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description, in conjunction with the accompanying drawings and preferred embodiments, describes the specific implementation, structure, features, and effects of the deep learning-based material identification method proposed according to the present invention.

[0040] This invention targets the application scenario of material identification in construction engineering. Since workers need to keep track of the status of materials in a timely manner to arrange the subsequent project progress, it is necessary to accurately count and identify the materials. Therefore, a specific solution for a material identification method based on image processing is proposed.

[0041] Specifically, this will be explained using steel reinforcement materials in construction projects as an example. Please refer to [link / reference]. Figure 1The diagram illustrates a flowchart of a material identification method based on image processing according to the present invention, which includes the following steps:

[0042] Acquire image information of the material to be identified from different perspectives;

[0043] The material image information from one perspective is determined as the standard image. The material image information collected from other different perspectives is registered with the standard image information to obtain the image registration result. The perspective transformation matrix of the material image information from different perspectives is then determined.

[0044] Based on the perspective transformation matrix, the material image information to be identified from other different perspectives is transformed to the perspective of the standard image, and the images are stitched together to obtain the stitched image information.

[0045] The material is identified based on the acquired stitched image information and depth information.

[0046] In this embodiment, multiple RGB-D cameras are used to acquire multi-view images and corresponding depth information of the side of the current building material. A better image of the side of the building material is obtained through image stitching. The multi-view images and corresponding depth information from the RGB-D cameras have been calibrated, as shown below. Figure 2 The image shown is of construction material images captured using an RGB-D camera.

[0047] The fixed positions and number of RGB-D cameras can be adjusted by the implementer according to the specific implementation scenario. This embodiment selects three RGB-D cameras to acquire side images and depth information of the building material. The three RGB-D cameras are positioned at three equal points on the side of the building material, maintaining a consistent height. In this embodiment, the height of the three RGB-D cameras is set to 50cm, but the implementer can adjust this according to the specific implementation scenario.

[0048] In this embodiment, after obtaining RGB images corresponding to multiple RGB-D cameras, the image captured by the RGB-D camera at the center position is selected as the standard viewpoint image. The implementer can select images captured by other cameras as the standard viewpoint image according to the specific implementation scenario.

[0049] Then, based on the existing SIFT feature point image registration method, the registration result of any image other than the standard view image and the current standard view image is obtained. From the current registration result, four pairs of feature points after image registration are randomly selected to obtain the perspective transformation matrix between the two images. The image used for image registration with the standard view image is transformed to the same spatial view as the current standard view image through the obtained perspective transformation matrix.

[0050] The image used for image registration with the standard viewpoint image after perspective transformation is then superimposed on the current standard viewpoint image to obtain the stitched result of the image and the current standard viewpoint image, which serves as the new standard viewpoint image. Then, any image other than the standard viewpoint image is selected and stitched with the current standard viewpoint image until no images remain, leaving only one standard viewpoint image, which serves as the final RGB image. The depth image stitching follows the same principle, which will not be elaborated upon in this embodiment.

[0051] The specific process of identifying materials based on the acquired stitched image information and depth information in this embodiment is as follows:

[0052] Obtain the stitched image information and the corresponding depth information;

[0053] Construct a Mask R-CNN network model, which includes a ResNet network; optimize the ResNet network to obtain an optimized Mask R-CNN network model; train the optimized Mask R-CNN network model to obtain a trained optimized Mask R-CNN network model.

[0054] The stitched image information and its corresponding depth information are input into a trained and optimized Mask R-CNN network model to identify materials and obtain the recognition results.

[0055] The network model used in this embodiment is the Mask R-CNN network model. This network model includes a ResNet residual network, an FPN network, an RPN neural network, and a RoI Align model. The specific process is as follows: image features are extracted through the ResNet network. The acquired image data is input into the ResNet network to extract image features from the ResNet network. The image features are then input into the FPN network to obtain feature information at different sampling scales, and then input into the RPN neural network to obtain regions. Classification and regression are performed on the obtained regions, and a Mask branch is added to complete the final image recognition and localization. In the process of classification, regression, and conditional masking, RoI Align is used to align pixels to ensure segmentation accuracy. The FPN network and RPN neural network in this embodiment are all networks in the prior art, and will not be described in detail here.

[0056] The ResNet network in the above embodiments uses the ResNet101 network, but the ResNet50 network can also be used.

[0057] In this embodiment, the specific process of optimizing the ResNet101 neural network in the Mask R-CNN network model is as follows:

[0058] Obtain the training dataset; input the training dataset into the ResNet101 network for training, and optimize it using the loss function to obtain the trained ResNet101 network.

[0059] The training dataset mentioned above is constructed based on the collected image data and depth image data of the materials. That is, the collected image data and depth image data of the materials are labeled to obtain the labeled dataset. In this embodiment, 80% of the data volume of the labeled dataset is used for training the ResNet101 neural network.

[0060] It should be noted that the image data is labeled and the labeling results are encoded. Specifically, the materials in the RGB image are manually labeled. Since Mask R-CNN is an instance segmentation network, when using the ResNet101 network in the ResNet101+FPN structure to extract feature maps, the feature maps used for instance segmentation should be selected.

[0061] This embodiment also includes expanding the acquired data using dataset augmentation techniques, such as rotation or adding noise. In this embodiment, the expanded dataset contains at least 1500 data entries, which can be adjusted by the implementer according to the specific implementation scenario. In this embodiment, one data entry refers to one RGB image and one corresponding depth image included in a single data acquisition process.

[0062] It should be noted that the image data of the material includes multiple steel bars, with each steel bar corresponding to a material category, meaning that multiple material categories are included.

[0063] Therefore, during the training of the ResNet101 network, when annotating RGB image data, it is necessary to annotate according to the data annotation process used for instance segmentation, that is, to perform pixel-level annotation. In the R, G, and B channels of the RGB image, pixels belonging to different materials are labeled with the numbers corresponding to different material categories A1, A2, and A3, such as 1, 2, and 3, which correspond to different materials 1 (steel bar 1), material 2 (steel bar 2), and material 3 (steel bar 3), etc. Pixels that do not belong to materials are labeled as 0, that is, 0 represents the background.

[0064] After labeling the RGB image data, the depth image data also needs to be labeled. The coordinates of the pixels corresponding to different materials in the RGB image are obtained, and then the coordinates of different materials in the depth data are labeled with new numbers. Since the RGB and depth images have the same width and height, the same material will be located in the same position in both images, but with different numbers. That is, material 1 can be labeled as class 1 in the depth data, and material 1 will also be labeled at the same position in the RGB image data and the corresponding depth data, but with different numbers. All numbers must be unique. After labeling the categories corresponding to different materials, one-hot encoding is used to encode them; thus, the training dataset is obtained.

[0065] It should be noted that in this embodiment, if depth image data and RGB image data are directly used as input to the ResNet network for training, the accuracy of some material detection results will decrease. If the image data and depth image data are fused first and then input into the ResNet network, there is a possibility that even when the image quality of the material category is good, the intervention of the depth image data may affect the identification of the material due to the unevenness of the steel bar surface. Therefore, in this embodiment, the RGB image data and depth image data are labeled separately.

[0066] The process of obtaining the loss function in the above embodiments is as follows:

[0067] (1) Obtain a grayscale image of the material image data, extract the image region of any single material category, extract the centroid coordinates of the image region, and obtain the distance between the center coordinates and the centroid coordinates of the grayscale image.

[0068] The process of obtaining the distance value in this embodiment is as follows:

[0069] Obtain the RGB image data input during the current ResNet network training process, i.e., the RGB image data from the i-th training data in the current training set, and use a grayscale conversion algorithm to obtain the grayscale image C corresponding to the RGB image data. i .

[0070] Where the current grayscale image C is obtained i The coordinates of the center point of the image, and to obtain the Ath... j Each material category is in grayscale image C. i The pixels occupied in the middle will be the Ath pixel. j Each material category is in grayscale image C. i If the pixel occupied by the A-th pixel is set to 1 and the rest are set to 0, then the A-th pixel can be obtained. j The mask image for material category A jThe mask image of each material category is used to obtain the Ath material category using a connected component extraction algorithm. j The centroid coordinates of each material category are used to calculate the grayscale image C using the coordinate point distance formula. i The coordinates of the center point and the Ath j The distance D between the centroid coordinates of each material category j , where j is the material number.

[0071] It should be noted that, according to the principles of camera imaging, the farther a material category is from the image center in the imaging result, the more difficult it is to observe its entirety, leading to poorer feature extraction and decreased accuracy in instance segmentation and detection. However, since depth information is not affected by viewing angle differences, the depth data of the rebar end face changes relatively little. Therefore, for material categories farther from the image center, feature extraction can be more biased towards depth data.

[0072] Therefore, the difference in field of view caused by the difference in camera angle during shooting leads to a deterioration in the performance of feature extraction in ResNet101 network training based on RGB images. Therefore, the training results of ResNet101 network should rely more on depth image data.

[0073] (2) Obtain the imaging quality assessment value of any single material.

[0074] In this embodiment, the specific process for obtaining the imaging quality assessment value of any single material is as follows:

[0075] Obtain the information entropy of a grayscale image;

[0076] Calculate the mean grayscale value of the image region corresponding to each material category, and obtain the mean grayscale value of all material categories; calculate the Ath material category... j The absolute value of the difference between the grayscale mean of each material category and the mean is used to obtain the ratio of the absolute value of the difference to the mean. The product of this ratio and the information entropy value is taken as the Ath value. j Imaging quality assessment values ​​for each material category.

[0077] It should be noted that in this embodiment, the influence of illumination also needs to be considered. When the grayscale image C j Middle A j If the lighting conditions for a material category are poor, it will cause the Ath material category to be affected. j The grayscale value of the pixel at each material category is too large or too small, thus causing the Ath pixel to... j Each material category suffers from texture complexity loss. Therefore, it is necessary to calculate the current grayscale image C. j Middle A jThe texture complexity of each material category is represented by the number of pixels. Existing technologies use information entropy to represent texture complexity, but due to excessively weak or strong lighting, information about all materials in a given area will inevitably be lost. Furthermore, lighting itself is a process of gradual grayscale change, which usually makes the representation of texture complexity using information entropy inaccurate.

[0078] Therefore, this embodiment chooses to score the illumination imaging quality corresponding to different materials to obtain the Ath result. j The imaging quality assessment value E corresponding to each material category j It is used to adjust the data dependency tendency during the training of neural networks.

[0079] In the above embodiments, the information entropy of the image is obtained through the grayscale image C. j The histogram distribution can be used to obtain the current grayscale image C. j The overall illumination score is calculated, where a bimodal or unimodal histogram distribution indicates poor overall image quality. The current grayscale image C is then calculated. j Information entropy, as the current grayscale image C j The imaging effect is affected by the entropy value. A higher entropy value indicates a greater amount of information in the lighting model, while a lower entropy value indicates less information. Furthermore, since the site remains relatively stable, large changes in entropy primarily affect the overall image quality. Entropy is a well-known calculation technique, so it will not be elaborated upon in this solution.

[0080] After obtaining the current grayscale image C j After scoring, a higher entropy value indicates better overall lighting, which is used to evaluate the overall score. However, under good lighting conditions, it can be assumed that at least most of the end faces of the current rebar are normal and a small portion are abnormal. Under poor lighting conditions, the imaging quality assessment value of all material categories should decrease.

[0081] This embodiment starts from the entire grayscale image C j Starting with the image, we obtain the current grayscale image's imaging effect. Then, based on the imaging effect of the entire image, we calculate a specific material category A. j Image quality assessment value.

[0082] The imaging quality evaluation value in the above embodiments, for the Ath j For each material category, calculate the current A-th... jThe mean gray level of each material category is obtained, and then the mean of the mean gray level of all material categories is obtained. The mean of the mean gray level of all material categories is affected by the current image as a whole. That is, the greater the information entropy, the better the lighting effect corresponding to the mean gray level of all material categories. Conversely, the smaller the information entropy, the worse the lighting effect corresponding to the mean gray level of all material categories.

[0083] Then calculate the Ath j The absolute value of the difference between the grayscale mean value corresponding to each material category and the mean value. Divide the absolute value of this difference by the mean value to obtain the ratio, where the ratio is related to the current C. j After multiplying the entropy values, we get the current A-th value. j The adjusted value of each material category in the overall information entropy is used as the current A-th value. j Imaging quality assessment value E for illumination corresponding to each material category j The larger the value, the higher the score.

[0084] It should be noted that the information entropy value of a single material category is not selected as the imaging quality assessment value E. j This is because the cross-section of the steel bars is relatively smooth. Although the lighting is different between different steel bars, the difference in information entropy is not significant.

[0085] (3) Based on the distance value and the imaging quality evaluation value, an adjustment index is obtained; based on the adjustment index, the true label value of the image of a single material and the true label value of the depth image, a loss function is constructed.

[0086] The adjustment index in this embodiment is:

[0087] F j =exp(-D j )*E j *G

[0088] Among them, D j The coordinates of the center point of the current grayscale image and the Ath j The distance between the centroid coordinates of each material category, E j Indicates the Ath element in the current grayscale image. j The imaging quality assessment value for each material category, where exp() is an exponential function with the natural number e as the base, and G is a hyperparameter.

[0089] Among them, the Ath j Materials of each material category in grayscale image C jThe better the lighting and viewing angle, the higher the image quality of the material. Therefore, when predicting the results using the ResNet101 network, more reliance should be placed on RGB 3D image data. Conversely, the lower the image quality of the material, the more reliance should be placed on depth image data when predicting the results using the ResNet101 network.

[0090] The larger the distance value in the formula, the larger the grayscale image C. j The A in j The further the material category is from the image center, the more it affects the Ath material category. j The poor field of view for some material categories makes it difficult to observe the entire picture, leading to poor feature extraction results and decreased accuracy in instance segmentation and detection. However, since depth information is not affected by the viewing angle difference, the depth data of the rebar end face changes relatively little. Therefore, for material categories farther from the image center, feature extraction can be more biased towards depth data. Conversely, it should be more biased towards RGB image data. Therefore, the smaller the distance value, the more the metric F should be adjusted. j The larger the value, the higher the grayscale image C. j Middle A j The better the imaging quality of each material category, the more likely the ResNet101 network will favor RGB 3D image data in its predictions. Therefore, a negative correlation mapping is performed on it to obtain exp(-D j Meanwhile, the imaging quality assessment value E j The larger the value, the more it indicates that the current A-th... j Materials of each material category in grayscale image C j In images with good lighting conditions, the prediction results of a ResNet101 network should rely more on RGB 3D image data. Conversely, in images with poor lighting conditions, the prediction results of a ResNet101 network should rely more on depth image data.

[0091] In this embodiment, G is a hyperparameter that can be adjusted by the implementer according to the specific implementation scenario. In this solution, G = 1.

[0092] Thus, based on the RGB images from the current ResNet101 network training process, the material category imaging quality assessment model for the current ResNet101 neural network training process is obtained.

[0093] In this embodiment, the current material imaging quality is evaluated to obtain a material imaging quality assessment value, which is used to adjust the data bias when the ResNet101 neural network model predicts the results, so that the trained ResNet101 neural network model can extract better feature maps.

[0094] The loss function in this embodiment is:

[0095]

[0096] Among them, F j Image A of the current material j The imaging quality assessment value corresponding to each material category; when the imaging quality value of a material category is large, F... j A larger value makes the evaluation of neural network learning performance more reliant on RGB image data, and vice versa; p(K) j ) is the Ath j The true label value of each material category on the material image during the current ResNet101 network training process, q(K′) j ) is the Ath j The predicted label values ​​of each material category on the material image during the current ResNet101 network training process, p(L) j ) is the Ath j The true label value of each material category on the depth image during the current ResNet101 network training process, q(L′) j ) is the Ath j The predicted label values ​​of each material category on the depth image during the current ResNet101 network training process, where H is the number of all categories.

[0097] In the loss function formula above, m() is a normalization function used to adjust the index F. j The adjustment, if the adjustment indicator F is not adjusted j Normalization will adjust the indicator F. j Applying the loss function directly to both cross-entropy loss functions will cause the loss value to fluctuate excessively; therefore, the metric F needs to be adjusted. j The value of varies across different images and material categories. In this embodiment, the adjustment index F is... j Maximum and minimum value normalization is used, where the maximum and minimum values ​​apply to all material categories in the preceding image data. That is, if the training results of the ResNet101 network are more biased towards RGB images, the bias towards depth images should be reduced, and to ensure the weight sum is 1, the sum of the cross-entropy loss function should be relatively stable. The cross-entropy loss function measures the difference between two different probability distributions of the same random variable; in neural network training, this refers to the difference between the true probability distribution and the predicted probability distribution. Therefore, the smaller the cross-entropy value, the better the model's prediction performance.

[0098] It should be noted that the adjustment indicator F j The larger the value, the higher the value of the current material category in the grayscale image C. jIn scenarios with good lighting and a favorable viewing angle, the ResNet101 network model should rely more heavily on RGB 3D image data for prediction. Conversely, it should rely more heavily on depth image data. This means that a larger cross-entropy loss function value prioritizes RGB image data, while a smaller value prioritizes depth image data. This allows the ResNet101 neural network model to extract more effective feature maps. By evaluating the imaging quality of different material categories during ResNet101 network model training, the model's data dependency bias can be adjusted. When the imaging quality of a material category is good, RGB image data is relied upon more, reducing the interference of depth data; conversely, when the imaging quality is poor, depth image data is relied upon more, ensuring accurate material identification and localization even with poor RGB image data.

[0099] In this embodiment, during the training of the ResNet101 network, the weight values ​​of the data on which the prediction results depend are adjusted using the imaging quality evaluation values ​​F of each material category corresponding to the currently input RGB image.

[0100] It should be noted that the adjustment index F obtained in the above steps j For grayscale image C j Material category A j The corresponding material category imaging quality assessment value. And for grayscale image C... j The corresponding image data contains multiple A j That is, an A j This represents a single steel bar in the image, which contains multiple steel bars.

[0101] Thus, the adjustment index for each material category in the image data is obtained, representing the adjustment index F for all steel rebar categories in the image data during the training of the ResNet101 network. A larger value for the adjustment index F indicates that the ResNet101 neural network should favor RGB image data during training and prediction, while a smaller value indicates a bias towards depth data.

[0102] If the current material category has a high image quality assessment value, then during the training of the ResNet101 network, the prediction results should rely more heavily on RGB image data. If the current material category has a low image quality assessment value, then during the training of the ResNet101 network, the prediction results should rely more heavily on depth image data.

[0103] After obtaining the loss corresponding to the image data i After that, loss i The smaller the value, the better the model's prediction performance, thus completing the optimization of the ResNet101 network model.

[0104] In the above embodiments, a higher adjustment index F for the current material category allows the ResNet101 network to use data that is more heavily weighted towards RGB image data during training. Conversely, a lower index F results in a more weighted prediction of depth image data.

[0105] This achieves the goal that, under good lighting and field of view, depth data does not affect the accuracy of RGB image data in material classification. By making the ResNet101 network rely more on depth data to complete material classification under poor lighting and field of view, the accuracy of material recognition is improved.

[0106] Thus, based on the material category imaging quality assessment model in the current ResNet101 network training process, the optimized ResNet101 network is obtained.

[0107] In this embodiment, Mask R-CNN neural network training is performed based on the optimized ResNet101 neural network model to obtain the optimized Mask R-CNN neural network, thereby completing the Mask R-CNN neural network training.

[0108] Specifically, after obtaining the optimized ResNet101 network, the ResNet101 network is trained using the labeled dataset. The neural network training process is a well-known technique and will not be elaborated further. In this scheme, the empirical values ​​for the hyperparameter settings during the training of the ResNet101 network are: learning rate of 0.25, momentum of 0.8, and epoch of 100. Stochastic gradient descent is used to train the ResNet101 network, thus completing the training of the ResNet101 neural network.

[0109] In this embodiment, the RGB image data and depth image data are decoupled from each other. A material category imaging quality assessment model is extracted from the RGB image data. This model is used to adjust the weights of the image data or depth data relied upon by the ResNet network when predicting results for different material categories. Specifically, if the current material category imaging is poor, the ResNet network should rely more on depth data for result prediction; conversely, if the current material category imaging is good, the ResNet network should rely more on image data for result prediction. This constructs the loss function of the ResNet network, providing support for the accuracy of subsequent material identification.

[0110] In this embodiment, the training process for the optimized Mask R-CNN neural network structure is as follows:

[0111] Obtain the training set; use the RGB and depth image data from the training set as input to the Mask R-CNN neural network structure; extract feature maps C1 to C5 using the optimized ResNet101 network in the Mask R-CNN neural network, and input the C1 to C5 feature maps into the FPN neural network to obtain p2 to p6. Then, input the extracted p2 to p6 into the RPN network, and use the newly added MASK branch network of the Mask R-CNN neural network to perform regression learning, bounding box learning, and mask learning to train the Mask R-CNN neural network. The method for extracting feature maps C1 to C5 is a well-known technique and will not be elaborated here; the acquisition of p2 to p6 is also a well-known technique and will not be elaborated here either.

[0112] It should be noted that the training set in this embodiment can be the same as the training dataset described above, or it can be a subset of that dataset.

[0113] It should be noted that the solution of this invention only optimizes the C1 to C5 feature map extraction part of the ResNet101 neural network in the Mask R-CNN neural network, while the other networks remain unchanged. Although image depth data is input to the Mask R-CNN neural network, it is not used subsequently, but only for feature map extraction. The other networks in the Mask R-CNN neural network are trained using only RGB image data.

[0114] At this point, the training of the Mask R-CNN neural network is complete, based on the optimized ResNet101 network and the currently labeled dataset.

[0115] The key point of this invention is to optimize the training results of the ResNet101 network. Then, when training the Mask R-CNN neural network, the optimized ResNet101 network is used to extract the feature maps C1 to C5 of the current input image, thereby completing the training of the current Mask R-CNN neural network and improving the material recognition accuracy of the Mask R-CNN neural network under poor lighting conditions.

[0116] In this embodiment, in the material detection area, the RGB image data and depth image data of the material to be identified obtained by image processing are input into the trained and optimized Mask R-CNN neural network, and the material identification and localization are completed by the Mask R-CNN neural network; the final material counting is completed by obtaining the number of material categories in the current image material identification results from the Mask R-CNN neural network.

[0117] This invention addresses the issue that if only depth image data is used as input to a neural network, the lack of texture information in the depth image data will result in poor material identification. Conversely, if only image data is used as input, the presence of texture information in the image will lead to better material identification results when the material category is well-defined. However, image data acquisition is susceptible to ambient lighting and imaging angle, causing unstable localization results. Therefore, this invention combines image and depth image data to construct a loss function. This allows for real-time adjustment of the importance of the required data based on the image quality, optimizing the network and enabling accurate material identification.

[0118] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0119] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A material identification method based on image processing, characterized in that, Includes the following steps: Acquire image information of the material to be identified from different perspectives; Image registration results are obtained by matching the building material images collected from different perspectives with the front view of the building material collected at the current fixed position of the camera. Based on the image registration results, the perspective transformation matrix required to transform the front view of the building material captured at the current fixed position of the camera to the corresponding viewpoint of the front view of the building material captured at different positions is obtained; Based on the perspective transformation matrix, the image information of the material to be identified from different perspectives is transformed to the same perspective, and the images are stitched together to obtain the stitched image information. Obtain depth information corresponding to the stitched image information; identify materials based on the obtained stitched image information and depth information; The specific process of identifying materials based on the acquired stitched image information and depth information is as follows: Construct a Mask R-CNN network model, which includes a ResNet network; optimize the ResNet network to obtain an optimized Mask R-CNN network model; train the optimized Mask R-CNN network model to obtain a trained optimized Mask R-CNN network model. The stitched image information and the corresponding depth information are input into the trained and optimized Mask R-CNN network model to identify the material and obtain the recognition result. The specific process of optimizing the ResNet network is as follows: Obtain the training dataset; input the training dataset into the ResNet network for training, and optimize it using the loss function to obtain the trained ResNet network; The process of obtaining the loss function is as follows: Obtain a grayscale image of the stitched material image data, extract the image region of any single material category, extract the centroid coordinates of the image region, and obtain the distance between the center coordinates and the centroid coordinates of the grayscale image; Obtain the imaging quality assessment value for any single material category; An adjustment index is obtained based on the distance value and the imaging quality assessment value; a loss function is constructed based on the adjustment index, the true label value of the image of a single material, and the true label value of the depth image; The loss function is: Where Fj is the adjustment index corresponding to the Aj-th material category in the current material image, p(kj) is the true label value of the Aj-th material category on the material image during the current ResNet network training process, q(k'j) is the predicted label value of the Aj-th material category on the material image during the current ResNet network training process, p(Lj) is the true label value of the Aj-th material category on the depth image during the current ResNet network training process, q(Lj') is the predicted label value of the Aj-th material category on the depth image during the current ResNet network training process, H is the number of all categories, m() is the normalization function, and j is the material number; The adjustment indicators are: Where Dj is the distance between the center point of the current grayscale image and the centroid of the Aj-th material category, Ej represents the imaging quality assessment value of the Aj-th material category in the current grayscale image, exp() is an exponential function with the natural number e as the base, and G is a hyperparameter.

2. The material identification method based on image processing according to claim 1, characterized in that, To acquire image information of the material to be identified from different perspectives, at least two cameras are set up to capture image information of the material from different perspectives.

3. The material identification method based on image processing according to claim 1, characterized in that, The SIFT feature point image registration method is used to obtain the registration result of the material image under any viewpoint other than the standard viewpoint image and the current standard viewpoint image. From the current registration result, four pairs of feature points after image registration are randomly selected to obtain the perspective transformation matrix between the two images. The image used for image registration with the standard viewpoint image is transformed to belong to the same spatial viewpoint as the current standard viewpoint image through the obtained perspective transformation matrix.

4. The material identification method based on image processing according to claim 1, characterized in that, The specific process for obtaining the imaging quality assessment value for any single material category is as follows: Obtain the information entropy of a grayscale image; Calculate the mean grayscale value of the image region corresponding to each material category, and obtain the mean grayscale value of all material categories; Calculate the absolute value of the difference between the grayscale mean of the Aj-th material category and the mean value, obtain the ratio of the absolute value of the difference to the mean value, and use the product of the ratio and the information entropy value as the imaging quality evaluation value of the Aj-th material category.

5. The material identification method based on image processing according to claim 1, characterized in that, The Mask R-CNN network model also includes the FPN network, the RPN neural network, and the RoI Align model.

Images