Bucket tooth monitoring method, device, and bucket

Abstract

The present application provides a bucket tooth monitoring method, device, and bucket for use in the field of engineering machinery technology. The bucket tooth monitoring method includes: acquiring multiple bucket images of the bucket in a working state; labeling the overall bucket tooth structure and each bucket tooth included in the overall bucket tooth structure in each bucket image, and generating a label file; training and verifying a neural network model based on multiple bucket images and corresponding label files to obtain a trained bucket recognition model; inputting multiple real-time bucket images into the bucket recognition model and outputting recognition results of multiple real-time bucket images; judging whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images. This solution uses the trained bucket recognition model to recognize bucket images in a working state in real time and monitor the recognition results, which can accurately identify whether the bucket teeth of the bucket are abnormal, eliminate safety hazards at the construction site, and reduce the accident rate.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to a bucket tooth monitoring method, device, and bucket. Background Art

[0002] Excavators are commonly used in construction projects, excavating and loading materials onto trucks. The bucket teeth are the primary working component of an excavator. Exposure to harsh environments for extended periods can cause damage to the teeth, which can even cause them to fall off. These teeth can then mix with material and enter equipment like crushers. Because the teeth are made of hard material, they can easily damage the equipment. Therefore, monitoring the status of the bucket teeth is crucial to the safe operation of construction equipment.

[0003] Existing bucket tooth monitoring methods capture bucket tooth data through a camera, use an object detection algorithm to generate a predicted frame for each tooth, and then use a semantic segmentation model to segment the teeth within the predicted frame to generate a tooth mask image. The difference or ratio between the tooth heights in the mask image is used to determine whether a tooth is missing in the current frame. If the cumulative number of missing teeth reaches a set threshold within a specified timeframe, the tooth is considered missing. The height of each tooth mask image generated by the semantic segmentation algorithm is random and does not perfectly match the tooth. Therefore, accurately determining whether a tooth is missing based on tooth height differences is difficult. Summary of the Invention

[0004] In light of this, the present application is dedicated to providing a bucket tooth monitoring method, device, and bucket. This solution trains and verifies a neural network model to obtain a trained bucket recognition model. This trained bucket recognition model is used to identify images of the bucket in operation in real time and monitor the recognition results. This allows accurate identification of abnormal bucket teeth, effectively resolving the issue in existing technologies where the accuracy of determining missing teeth based on tooth height differences is low.

[0005] In a first aspect, an embodiment of the present application provides a bucket tooth monitoring method of a bucket, comprising:

[0006] Acquire multiple bucket images of the bucket in a working state;

[0007] Label the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image, and generate a label file;

[0008] Based on the multiple bucket images and corresponding label files, a neural network model is trained and verified to obtain a trained bucket recognition model;

[0009] Inputting the acquired multiple real-time bucket images into a bucket recognition model for recognition, and outputting recognition results of the multiple real-time bucket images, the recognition results including a recognition result of the overall structure of the bucket teeth in the real-time bucket images and a recognition result of each bucket tooth in the real-time bucket images;

[0010] Whether the bucket teeth of the bucket are abnormal is determined according to the recognition results of the multiple real-time bucket images.

[0011] Optionally, the step of labeling the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image includes:

[0012] Select the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure using a marking box according to the category name, and generate a category serial number, wherein the category serial number corresponds to the category name;

[0013] The recognition result of the real-time bucket image includes: a prediction frame of the overall structure of the bucket tooth in the real-time bucket image, and a prediction frame of each bucket tooth in the real-time bucket image.

[0014] Optionally, the bucket tooth monitoring method provided in the first aspect of the present application further includes:

[0015] The bucket image is preprocessed, and the preprocessing includes data cleaning and data enhancement. The data enhancement includes simulating rainy, foggy, dusty, and exposed scenes.

[0016] Optionally, the bucket tooth abnormality of the bucket includes missing;

[0017] Determining whether bucket teeth of the bucket are missing according to recognition results of the multiple real-time bucket images includes:

[0018] Matching the predicted frame of the bucket tooth overall structure in each real-time bucket image with the predicted frame of each bucket tooth to determine a matching result;

[0019] When the matching result is that the number of successful matches exceeds a first preset threshold, it is determined that the bucket teeth of the bucket are missing.

[0020] Optionally, matching the predicted frame of the bucket tooth overall structure in each real-time bucket image with the predicted frame of each bucket tooth to determine the matching result includes:

[0021] Determine a first predicted missing bucket tooth position according to a predicted frame of the bucket tooth overall structure in the real-time bucket image;

[0022] Determining a position of a second predicted missing bucket tooth based on a predicted box of each bucket tooth in the real-time bucket image; and

[0023] When the first predicted position of the missing bucket tooth is consistent with the second predicted position of the missing bucket tooth, the matching result is determined to be a successful match.

[0024] Optionally, the bucket tooth abnormality of the bucket includes tilting;

[0025] Determining whether the bucket teeth of the bucket are tilted according to the recognition results of the multiple real-time bucket images includes:

[0026] Calculate the real-time width of a bucket tooth based on the predicted frame of a bucket tooth in each real-time bucket image;

[0027] Calculating a width ratio of the real-time width of the bucket tooth to the factory width according to the real-time width of the bucket tooth and the factory width of the bucket tooth;

[0028] Recording the number of real-time bucket images whose width ratio exceeds a second preset threshold;

[0029] When the number of real-time bucket images whose width ratio exceeds the second preset threshold value exceeds the third preset threshold value, it is determined that a certain bucket tooth of the bucket is tilted.

[0030] Optionally, the bucket tooth abnormality includes wear;

[0031] Determining whether the bucket teeth of the bucket are worn according to the recognition results of the multiple real-time bucket images includes:

[0032] Calculate the real-time area of ​​a bucket tooth based on the mask of the bucket tooth in each real-time bucket image;

[0033] Calculating an area ratio of the real-time area of ​​the bucket tooth to the preset area according to the real-time area of ​​the bucket tooth and the preset area;

[0034] Recording the number of real-time bucket images whose area ratio exceeds a fourth preset threshold;

[0035] When the number of real-time bucket images whose area ratio exceeds the fourth preset threshold value exceeds the fifth preset threshold value, it is determined that a tooth of the bucket is worn.

[0036] Optionally, the preset area is the bucket tooth area when the bucket teeth leave the factory or the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image.

[0037] In a second aspect, an embodiment of the present application provides a bucket tooth monitoring device for a bucket, comprising: an image acquisition module, an image processing module, a bucket recognition model, and an abnormality determination module, wherein:

[0038] The image acquisition module is used to obtain multiple bucket images when the bucket is in working state;

[0039] The image processing module is used to mark the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image, and generate a label file;

[0040] A bucket recognition model, the bucket recognition model being obtained by training and validating a neural network model based on the multiple bucket images and corresponding label files; the bucket recognition model being configured to recognize the multiple acquired real-time bucket images and output recognition results of the multiple real-time bucket images, the recognition results including recognition results of the overall structure of the bucket teeth in the real-time bucket images and recognition results of each bucket tooth in the real-time bucket images;

[0041] The abnormality determination module is used to determine whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images.

[0042] In a third aspect, an embodiment of the present application provides an excavator, comprising:

[0043] A bucket tooth integral structure, the bucket tooth integral structure comprising a plurality of bucket teeth;

[0044] And the bucket tooth monitoring device of the bucket described in the second aspect.

[0045] Based on the above content, the present application provides a bucket tooth monitoring method for a bucket, which first obtains multiple bucket images of the bucket in a working state, and then labels the overall structure of the bucket teeth in each bucket image and each bucket tooth included in the overall structure of the bucket teeth, and generates a label file. Based on the multiple bucket images and the corresponding label files, the neural network model is trained and verified to obtain a trained bucket recognition model. The multiple real-time bucket images obtained are input into the bucket recognition model for recognition, and the recognition results of the multiple real-time bucket images are output. It is judged whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images. In this way, by using the trained bucket recognition model to recognize the bucket images in a working state in real time and monitor the recognition results, it is possible to accurately identify whether the bucket teeth of the bucket are abnormal, eliminate safety hazards at the construction site, and reduce the accident rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A schematic flow chart of a bucket tooth monitoring method for a bucket provided in an embodiment of the present application;

[0048] Figure 2 This is a diagram showing the recognition results of a bucket tooth monitoring method for a bucket provided in an embodiment of the present application;

[0049] Figure 3 A schematic flow chart of a bucket tooth monitoring method for a bucket provided in an embodiment of the present application;

[0050] Figure 4 A schematic flow chart of a bucket tooth monitoring method for a bucket provided in an embodiment of the present application;

[0051] Figure 5 A schematic flow chart of a bucket tooth monitoring method for a bucket provided in an embodiment of the present application;

[0052] Figure 6 This is a structural block diagram of a bucket tooth monitoring device for a bucket provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, technical or scientific terms used in the embodiments of this specification should have the same ordinary meaning as those understood by persons of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not denote any order, quantity, or importance, but are provided solely to avoid confusion between constituent elements.

[0054] Unless the context requires otherwise, throughout this specification, the term "plurality" means "at least two," and "including" is to be interpreted as open and inclusive, meaning "including, but not limited to." Throughout this specification, the terms "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with the embodiment or example is included in at least one embodiment or example of this specification. The schematic representations of these terms do not necessarily refer to the same embodiment or example.

[0055] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0056] Application Overview

[0057] The materials that excavators need to excavate on a daily basis are mainly soil, coal, silt, and pre-loosened soil and rock. During the excavation process, the excavator needs to perform preliminary crushing on these materials, and then load and unload them onto the transport truck. The truck then loads the materials onto equipment such as crushers for further processing. The main working parts of the excavator are the bucket teeth. Due to the accumulated working hours and harsh working conditions, the bucket teeth are easily damaged to varying degrees, and in severe cases, the bucket teeth may even fall off. The fallen bucket teeth may fall into the unexcavated materials and be loaded and unloaded into the truck during the subsequent excavation process. The bucket teeth may also fall directly into the truck during the loading and unloading process, causing the fallen bucket teeth to eventually enter the crusher and other equipment, causing damage to the equipment.

[0058] In existing technology, only each bucket tooth is labeled to obtain a predicted frame for each tooth. This predicted frame is then processed to create a tooth mask image. The difference or ratio between the tooth heights in the mask image is used to determine whether a tooth is missing in the current frame. If the cumulative number of missing teeth reaches a set threshold within a specified time period, the tooth mask image is considered missing. This resulting tooth mask image has a random height in each frame and does not perfectly match the tooth. Therefore, accurately determining whether a tooth is missing based on tooth height differences is difficult.

[0059] To solve the above technical problems, the present application provides a bucket tooth monitoring method, device, and excavator. The method manually labels the overall bucket tooth structure and each bucket tooth included in the overall bucket tooth structure in multiple bucket images and generates a label file. Based on the multiple bucket images and the corresponding label files, the neural network model is trained and verified to obtain a trained bucket recognition model. The multiple real-time bucket images obtained are input into the bucket recognition model to obtain recognition results of the multiple real-time bucket images. The bucket teeth of the bucket are judged to be abnormal based on the recognition results of the multiple real-time bucket images. In this way, by using the trained bucket recognition model to recognize bucket images in a working state in real time and monitor the recognition results, it is possible to accurately identify whether the bucket teeth of the bucket are abnormal, eliminate safety hazards at the construction site, and reduce the accident rate.

[0060] The bucket tooth monitoring method of a bucket provided in the embodiments of this specification will be exemplarily described below with reference to the accompanying drawings.

[0061] Exemplary Methods

[0062] like Figure 1 As shown, in this exemplary embodiment, a bucket tooth monitoring method is provided, which may include the following steps:

[0063] S101: Acquire multiple bucket images of the bucket in a working state.

[0064] The image acquisition device is pre-installed at a specific position of the excavator, such as the top of the cab or under the boom or other fixed position. The image acquisition device may include a camera, a video camera, a surveillance camera or other equipment that can be used to take images or record videos. They are not listed here one by one. Under the premise of not exceeding the core idea of ​​this application, they also fall within the scope of protection of this application.

[0065] This application takes the image acquisition device as an example, which is a vehicle-mounted surveillance camera. The vehicle-mounted surveillance camera can obtain real-time video of the bucket in the working state, and by analyzing the video, obtain multiple bucket images in the working state.

[0066] In some embodiments of the present application, the bucket image is preprocessed, and the preprocessing includes data cleaning and data enhancement. The data enhancement includes simulating rainy, foggy, dusty, and exposed scenes.

[0067] Data cleaning: Remove damaged bucket image samples and select appropriate bucket images to retain. The number of positive and negative samples should be as balanced as possible for the next step.

[0068] Over a period of time, the bucket images obtained by shooting and analyzing videos will most likely only cover one scenario, such as sunny, cloudy, or rainy days. In order to cover various extreme scenarios and improve the generalization ability of deep learning, data enhancement technology is used on the basis of the obtained bucket images to increase the amount of bucket image data. The features extracted from the processed images will be different from those of the original images, which will improve the training effect of the model and make the model more robust.

[0069] Simulating rainy scenes: Generate Gaussian noise in the original bucket image, simulate the shape of raindrops through a diagonal matrix, simulate the tilt angle of raindrops by rotating the diagonal matrix, and adjust the threshold parameter to control the number of raindrops.

[0070] Simulate fog and dust scenes: Set the concentration and brightness of fog and dust, calculate the transmittance based on the Euclidean distance between the center point of the bucket image and each pixel, and perform weighted processing on the bucket image based on the distance.

[0071] Simulate exposure scenarios: Calculate the center and radius of the image, determine the enhanced central area, use the distance from the pixel to the center as the weight, amplify the pixel value in the central area, reduce the pixel value in the edge area, and highlight the details of the exposed area.

[0072] In some embodiments, tilt sensors are respectively installed on the excavator's boom, arm, and bucket. After the excavator is just started, before obtaining multiple bucket images of the bucket in the working state, the excavator's boom, arm, and bucket are controlled to automatically adjust to the target position by obtaining the value of the tilt sensor. The target position is the position where the on-board monitoring camera can monitor all bucket teeth, and then starts to obtain multiple bucket images of the bucket in the working state.

[0073] S102: Label the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image, and generate a label file.

[0074] Depending on the size of the bucket, the number of teeth in the bucket is also different. This application takes the bucket with 4 teeth as an example. When dividing the current working status of the excavator according to the different shapes of the bucket in the acquired bucket image, the bucket image is labeled to obtain a label file corresponding to the bucket image.

[0075] When it is determined from the bucket image that the working state of the excavator is digging, the bucket image is labeled as digging.

[0076] When it is determined from the bucket image that the working state of the excavator is unloading, the bucket image is marked as unloading.

[0077] When it is determined from the bucket image that the working state of the excavator is loading, the bucket image is labeled "loading".

[0078] When it is determined from the bucket image that there is occlusion and not all bucket teeth can be seen, the bucket image is marked as undetected.

[0079] When it is judged from the bucket image that the working state of the excavator is in the reset state, that is, the excavator is in the state from the completion of excavation to the next excavation, and there is no obstruction to see all the bucket teeth, all four bucket teeth are regarded as the bucket tooth overall structure, that is, the bucket tooth overall structure includes all the bucket teeth. The bucket tooth overall structure in each bucket image is manually labeled, and each bucket tooth included in the bucket tooth overall structure is manually labeled to obtain a label file, and the label file is matched with the bucket image.

[0080] In some embodiments of the present application, the specific steps of marking the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image in S102 may include:

[0081] When the bucket image determines that the excavator's working state is in the reset state, that is, the excavator is in the state from the end of excavation to the next excavation, and there is no obstruction to the view of all bucket teeth, the bucket tooth structure and each bucket tooth included in the bucket tooth structure in each bucket image are manually labeled. The bucket tooth structure and each bucket tooth included in the bucket tooth structure are selected using a labeling box according to the category name, and the category number and the coordinate information of the labeling box are generated. The category number corresponds to the category name. The label file contains information after the bucket image is labeled, such as all the labeling box details, the category number, the bucket tooth category name, the horizontal and vertical coordinates of the labeling box center point, and the width and height of the labeling box. The relative position of the center point in the bucket image is calculated based on the horizontal and vertical coordinates of the labeling box center point, and the ratio of the width and height of the labeling box to the width and height of the entire bucket image is calculated based on the width and height of the labeling box.

[0082] A bucket tooth consists of a tip and a root. If a tooth in the bucket image has a tip, the tooth category is "tip." If a tooth in the bucket image does not have a tip, the tooth category is "root," indicating that the tooth is missing. Each selected tooth is assigned a category number based on its position from left to right. The category number corresponds to its category name. For example, 1 represents the first tooth from left to right, and its category is "tip." For example, 2 represents the second tooth from left to right, and its category is "root."

[0083] When the bucket and bucket teeth account for a relatively low proportion in the bucket image and the surrounding environment accounts for a relatively high proportion, it is not conducive to the recognition of the bucket teeth, which will reduce the recognition accuracy and waste the detection performance of the model. In some embodiments, a fixed area of ​​the marked bucket image is cut out to obtain a cutout bucket image, and the annotations in the bucket image are offset according to the relative position relationship between the cutout bucket image and the bucket image.

[0084] The bucket image is labeled by recording its position on the bucket image. Cropping a fixed region of the bucket image means the original labeling frame position is no longer applicable to the cropped image. Based on the cropped fixed region information, the relative position between the cropped bucket image and the bucket image can be calculated. The labeling frame can then be offset based on this positional relationship, generating a new label file.

[0085] S103: Based on the multiple bucket images and the corresponding label files, the neural network model is trained and verified to obtain a trained bucket recognition model.

[0086] The above-mentioned multiple pairs of bucket images and the corresponding label files are used as data sets and divided into training set, validation set and test set. The division can be carried out in a fixed ratio, such as 8:1:1 for the training set, validation set and test set. A suitable deep learning algorithm such as the target detection algorithm and the instance segmentation algorithm is selected. After adjusting the parameters, the neural network model is trained with the training set to obtain the preliminary trained bucket recognition model and the weight file of the model parameters. The preliminary trained bucket recognition model is verified and optimized with the validation set. The trained model weight file is quantized and accelerated to adapt to the edge computing device. A fixed period can be set for the validation set. The validation set is used to verify the effect of the model in each period and to update the parameters of the model. The test set is used to evaluate the model to obtain the trained bucket recognition model.

[0087] S104: Inputting the acquired multiple real-time bucket images into a bucket recognition model for recognition, and outputting recognition results of the multiple real-time bucket images.

[0088] The bucket video captured in real time by the vehicle-mounted surveillance camera is parsed to obtain multiple real-time bucket images. The multiple real-time bucket images obtained are input into the trained bucket recognition model for recognition. The bucket recognition model will output the recognition results of the multiple real-time bucket images.

[0089] When the bucket recognition model identifies that the working state of the excavator in the real-time bucket image is digging, the recognition result is digging.

[0090] When the bucket recognition model identifies that the working state of the excavator in the real-time bucket image is unloading, the recognition result is unloading.

[0091] When the bucket recognition model identifies that the working state of the excavator in the real-time bucket image is loading, the recognition result is loading.

[0092] When the bucket recognition model detects that there is occlusion in the real-time bucket image and all bucket teeth cannot be seen, the recognition result is undetect.

[0093] When the bucket recognition model identifies that the working state of the excavator in the real-time bucket image is in the reset state, that is, the excavator is in the state from the completion of excavation to the next excavation, and there is no occlusion and all bucket teeth can be seen, the recognition result includes the recognition result of the overall structure of the bucket teeth in the real-time bucket image and the recognition result of each bucket tooth in the real-time bucket image, such as Figure 2 shown.

[0094] S105: Determine whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images.

[0095] The overall tooth structure recognition results generate a predicted frame for the tooth structure within the real-time bucket image. The individual tooth recognition results generate a predicted frame for each tooth within the real-time bucket image. Based on the overall tooth structure recognition results and the individual tooth recognition results within the real-time bucket image, the bucket teeth are determined to be abnormal. Abnormal bucket teeth include missing, tilted, and worn teeth.

[0096] In some embodiments of the present application, multiple bucket images of the bucket in a working state are obtained, the overall structure of the bucket teeth in each bucket image and each bucket tooth included in the overall structure of the bucket teeth are respectively labeled, and a label file is generated. Based on the multiple bucket images and the corresponding label files, a neural network model is trained and verified to obtain a trained bucket recognition model. The multiple real-time bucket images obtained are input into the bucket recognition model for recognition, and the recognition results of the multiple real-time bucket images are output. It is determined whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images.

[0097] Through the trained bucket recognition model, the bucket image in the working state is recognized in real time and the recognition results are monitored. It can accurately identify whether the bucket teeth are abnormal, eliminate safety hazards at the construction site, and reduce the accident rate.

[0098] In some embodiments of the present application, Figure 3 As shown, the step of determining whether the bucket teeth of the bucket are missing based on the recognition results of the multiple real-time bucket images may specifically include:

[0099] S301: Match the predicted frame of the bucket tooth overall structure in each real-time bucket image with the predicted frame of each bucket tooth to determine a matching result.

[0100] In some embodiments of the present application, Figure 4 As shown, the predicted frame of the bucket tooth overall structure in each real-time bucket image is matched with the predicted frame of each bucket tooth. The specific steps of determining the matching result include:

[0101] S3011: Determine a first predicted missing bucket tooth position according to a predicted frame of the bucket tooth overall structure in the real-time bucket image.

[0102] The multiple real-time bucket images obtained are input into the bucket recognition model for recognition. The bucket recognition model uses the target detection algorithm to output the recognition result of each real-time bucket image. The recognition result of each real-time bucket image includes the recognition result of the overall structure of the bucket tooth and the recognition result of each bucket tooth. The recognition result of the overall structure of the bucket tooth in the real-time bucket image is to generate a prediction frame of the overall structure of the bucket tooth in the real-time bucket image and obtain the information of the prediction frame. The information of the prediction frame includes the coordinate information and category name of the upper left corner and lower right corner of the prediction frame, as well as the confidence obtained by using the target detection algorithm, such as Figure 2 The confidence value range is between 0 and 1. The larger the confidence value, the more reliable the prediction result. According to the prediction frame of the overall structure of the bucket tooth in the real-time bucket image, the information of whether the bucket tooth is missing can be obtained. When the bucket tooth is missing, the first predicted position of the missing bucket tooth is obtained, such as Figure 2 Miss4 means that the missing bucket tooth is the fourth one from the left to the right, that is, the fourth bucket tooth is missing.

[0103] S3012: Determine a second predicted position of the missing bucket tooth according to the predicted frame of each bucket tooth in the real-time bucket image.

[0104] The recognition result of each bucket tooth in the real-time bucket image is to generate a prediction box for each bucket tooth in the real-time bucket image and obtain the prediction box information. The prediction box information includes the coordinate information and category name of the upper left corner and lower right corner of the prediction box. According to the prediction box of each bucket tooth in the real-time bucket image, information on whether the bucket tooth is missing can be obtained. When the bucket tooth is missing, the second predicted position of the missing bucket tooth is obtained.

[0105] In some embodiments, the specific steps of determining the position of the second predicted missing bucket tooth based on the predicted frame of each bucket tooth in the real-time bucket image may include:

[0106] The bucket teeth in the real-time bucket image are sorted according to the horizontal coordinate of the upper left corner of the prediction box, and the information of the bucket tooth prediction box is judged in turn. When the category name in the prediction box information is tooth root, the second predicted position of the missing bucket tooth is output. The second predicted position of the missing bucket tooth is the position of the tooth root.

[0107] S3013: When the first predicted position of the missing bucket tooth is consistent with the second predicted position of the missing bucket tooth, the matching result is determined to be a successful match.

[0108] A real-time bucket image is input into the bucket recognition model for recognition. The bucket recognition model then outputs the recognition results for this real-time bucket image. The recognition results include the recognition results for the overall bucket tooth structure and the recognition results for each bucket tooth. Only when the recognition results for the overall bucket tooth structure and the recognition results for each bucket tooth both include the location of the missing tooth are they matched. When the first predicted missing tooth location matches the second predicted missing tooth location, the match is determined to be successful, and the successfully matched image is saved. Otherwise, when the first predicted missing tooth location does not match the second predicted missing tooth location, the match is determined to be unsuccessful, and the unsuccessful matching image is saved for later verification. Real-time bucket images in which the recognition results for the overall bucket tooth structure and the recognition results for each bucket tooth both include the location of the missing tooth are matched sequentially.

[0109] Normally, the recognition result for a bucket tooth is either the root or the tip. The sum of the number of tips and roots equals the number of teeth in the current bucket. If the sum of the number of tips and roots is less than the number of teeth in the current bucket, it means that the bucket image is incomplete or at least one tooth is not recognized. In this case, the recognition result of the bucket image is not matched to prevent misjudgment due to incomplete bucket images.

[0110] S302: When the matching result shows that the number of successful matches exceeds a first preset threshold, it is determined that the bucket teeth of the bucket are missing.

[0111] A first preset threshold is set in advance, the number of times the real-time bucket image matching is completed is accumulated, and the number of times the position of the first predicted missing bucket tooth in the real-time bucket image is consistent with the position of the second predicted missing bucket tooth is recorded, that is, the number of successful matches is the matching result. When the matching result is that the number of successful matches of the same bucket tooth exceeds the first preset threshold, it is determined that the bucket tooth is missing, and an alarm is issued, and the staff performs a shutdown operation.

[0112] In some embodiments of the present application, Figure 5 As shown, the step of determining whether the bucket teeth of the bucket are tilted according to the recognition results of the multiple real-time bucket images may specifically include:

[0113] S501: Calculate the real-time width of a bucket tooth based on the predicted frame of the bucket tooth in each real-time bucket image.

[0114] The bucket recognition model uses an instance segmentation algorithm to output recognition results for each real-time bucket image. This recognition result is an image with a predicted box generated for each bucket in the real-time bucket image. This generates the predicted box information and a mask for each bucket in the real-time bucket image. The mask accurately covers every pixel in the bucket. For example, based on the predicted box for a specific bucket tooth in a real-time bucket image, the real-time width of that tooth is calculated. The specific tooth can be any tooth, and the real-time width of each tooth is calculated sequentially.

[0115] S502: Calculating a width ratio of the real-time width of a bucket tooth to the factory width according to the real-time width of the bucket tooth and the factory width of the bucket tooth.

[0116] Record the factory width of the bucket teeth in advance. This factory width does not necessarily refer to the width measured immediately after delivery; it can be the width of a new bucket tooth of the same model. Calculate the ratio of each tooth's real-time width to the factory width based on the recorded real-time width and factory width. Calculate the width ratio of each tooth.

[0117] S503: Record the number of real-time bucket images whose width ratio exceeds a second preset threshold.

[0118] A second preset threshold is pre-set as a criterion for determining bucket tooth tilt. The ratio of the real-time width of each bucket tooth to the factory width is sequentially compared with the second preset threshold. When the width ratio exceeds the second preset threshold, the corresponding real-time bucket image is recorded, and the number of real-time bucket images is counted.

[0119] S504: When the number of real-time bucket images whose width ratio exceeds the second preset threshold value exceeds the third preset threshold value, it is determined that a certain bucket tooth of the bucket is tilted.

[0120] A third preset threshold is set in advance as another criterion for judging the tilt of the bucket tooth. When the number of real-time bucket images of a bucket tooth whose width ratio exceeds the second preset threshold exceeds the third preset threshold, it is judged that the bucket tooth is tilted.

[0121] Specifically, the bucket teeth whose width ratio exceeds the second preset threshold are not necessarily the same bucket teeth. For example, the third preset threshold is 20, the number of real-time bucket images whose first bucket tooth width ratio exceeds the second preset threshold is 2, and the number of real-time bucket images whose second bucket tooth width ratio exceeds the second preset threshold is 30. At this time, it is judged that the second bucket tooth is tilted.

[0122] Furthermore, more than one bucket tooth may be tilted in the same real-time bucket image, meaning that the width ratio of more than one bucket tooth exceeds the second preset threshold. In this case, the number of real-time bucket images in which the width ratio of each bucket tooth exceeds the second preset threshold is recorded. If the number of real-time bucket images in which the width ratio of a particular bucket tooth exceeds the second preset threshold exceeds a third preset threshold, the bucket tooth is determined to be tilted and an alarm is issued.

[0123] In some embodiments of the present application, Figure 6 As shown, the step of determining whether the bucket teeth of the bucket are worn according to the recognition results of the multiple real-time bucket images may specifically include:

[0124] S601: Calculate the real-time area of ​​a bucket tooth based on the mask of the bucket tooth in each real-time bucket image.

[0125] The area of ​​the bucket tooth is calculated based on the outer contour of the mask. This application uses Green's formula to mathematically calculate the area of ​​the area enclosed by a closed curve in the plane. Green's formula converts the complex integral in the two-dimensional plane area into a simple integral on the curve contour, thereby efficiently calculating the area of ​​the closed area. The formula is as follows:

[0126]

[0127] Specifically for area calculation, the contourArea function uses the discrete form of Green's theorem and approximates continuous integrals through mathematical summation. The contribution of each pair of adjacent points is calculated and accumulated through a specific formula to finally obtain the total area of ​​the closed contour.

[0128] Specifically, the algorithm details of the contourArea function involve "directed edges". Each edge provides a directed area contribution, and the direction of the edge (clockwise or counterclockwise) will affect the positive and negative sign of the area. Traverse the points on the contour, for each pair of adjacent points (x[i], y[i]) and (x[i+1], y[i+1]), the corresponding directed area contribution is (x[i]*y[i]-x[i+1]*y[i+1]) / 2. The area of ​​the entire contour is the sum of the directed area contributions of all these small triangles. In this process, if the contour is clockwise, the result is positive, and if it is counterclockwise, the result is negative. Finally, the contourArea function returns the absolute value of the area to ensure that the actual area size is obtained. The calculation formula is as follows:

[0129]

[0130] S602: Calculating an area ratio of the real-time area of ​​a bucket tooth to the preset area according to the real-time area of ​​the bucket tooth and the preset area.

[0131] The preset area is recorded in advance, and the area ratio of the real-time area of ​​each bucket tooth to the preset area is calculated based on the recorded real-time area of ​​each bucket tooth and the preset area. The area ratios of the four bucket teeth are obtained respectively.

[0132] In some embodiments of the present application, the preset area is the bucket tooth area when the bucket teeth leave the factory or the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image.

[0133] When the preset area is the bucket tooth area when the bucket tooth leaves the factory, the bucket tooth area when the bucket tooth leaves the factory is calculated in advance, and the above area ratio is the ratio of the real-time area of ​​a certain bucket tooth to the factory area.

[0134] When the preset area is the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image, the real-time area of ​​each bucket tooth is calculated in turn according to the bucket tooth mask, and the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image is compared. The above area ratio is the ratio of the real-time area of ​​a certain bucket tooth to the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image.

[0135] S603: Record the number of real-time bucket images whose area ratio exceeds a fourth preset threshold.

[0136] A fourth preset threshold is set as a criterion for determining bucket tooth wear. The ratio of the real-time area of ​​each bucket tooth to the preset area is sequentially compared with the fourth preset threshold. When the area ratio exceeds the fourth preset threshold, the corresponding real-time bucket image is recorded, and the number of real-time bucket images is counted.

[0137] S604: When the number of real-time bucket images whose area ratio exceeds the fourth preset threshold value exceeds the fifth preset threshold value, it is determined that a tooth of the bucket is worn.

[0138] A fifth preset threshold is set in advance as another criterion for judging bucket tooth wear. When the number of real-time bucket images in which a bucket tooth area ratio exceeds the fourth preset threshold exceeds the fifth preset threshold, it is determined that a bucket tooth of the bucket is worn.

[0139] Specifically, the bucket teeth whose area ratio exceeds the fourth preset threshold are not necessarily the same bucket teeth. For example, the fifth preset threshold is 20, the number of real-time bucket images in which the area ratio of the first bucket tooth exceeds the second preset threshold is 2, and the number of real-time bucket images in which the area ratio of the second bucket tooth exceeds the fourth preset threshold is 30. At this time, it is judged that the second bucket tooth is worn.

[0140] Furthermore, more than one tooth in the same real-time bucket image may be worn, meaning that the area ratio of more than one tooth exceeds the fourth preset threshold. In this case, the number of real-time bucket images in which the area ratio of each tooth exceeds the fourth preset threshold is recorded. If the number of real-time bucket images in which the area ratio of a particular tooth exceeds the second preset threshold exceeds a fifth preset threshold, the bucket is judged to be worn and an alarm is issued.

[0141] Exemplary devices

[0142] In some embodiments of the present application, a bucket tooth monitoring device is provided, comprising: an image acquisition module 601, an image processing module 602, a bucket recognition model 603 and an abnormality determination module 604, wherein:

[0143] An image acquisition module 601 is used to acquire multiple bucket images of the bucket in a working state;

[0144] An image processing module 602 is used to label the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image and generate a label file;

[0145] Bucket recognition model 603, which is obtained by training and verifying a neural network model based on multiple bucket images and corresponding label files; Bucket recognition model 603 is used to recognize multiple real-time bucket images and output recognition results for the multiple real-time bucket images, including recognition results of the overall structure of the bucket teeth in the real-time bucket images and recognition results of each bucket tooth in the real-time bucket images;

[0146] The abnormality determination module 604 is configured to determine whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images.

[0147] In some embodiments of the present application, the image processing module 602 is further used to preprocess the bucket image, where the preprocessing includes data cleaning and data enhancement, and the data enhancement includes simulating rainy, foggy, dusty, and exposed scenes.

[0148] In some embodiments of the present application, the image processing module 602 is also used to intercept a fixed area of ​​the annotated bucket image to obtain a intercepted bucket image, and offset the annotations in the bucket image according to the relative positional relationship between the intercepted bucket image and the bucket image. The bucket tooth monitoring device provided in this embodiment belongs to the same application concept as the bucket tooth monitoring method provided in the above embodiments of the present application, and can execute the bucket tooth monitoring method provided in any of the above embodiments of the present application, and has the corresponding functional units and beneficial effects for executing the bucket tooth monitoring method. For technical details not fully described in this embodiment, please refer to the specific processing content of the bucket tooth monitoring method provided in the above embodiments of the present application, and will not be repeated here.

[0149] Exemplary excavator

[0150] In some embodiments of the present application, an excavator is provided, comprising: a bucket tooth integral structure, the bucket tooth integral structure comprising a plurality of bucket teeth, and a bucket tooth monitoring device for the bucket.

[0151] In some embodiments, the image acquisition module 601 in the bucket tooth monitoring device is installed on top of the cab to capture images of the bucket during the entire working state of the excavator. This not only reduces the impact of excavator vibration on the data acquisition equipment, but also reduces the impact of the surrounding environment.

[0152] Exemplary electronic devices

[0153] In some embodiments of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above bucket tooth monitoring methods is implemented.

[0154] Specifically, the electronic device may further include a bus, a communication interface, an input device, and an output device. The processor, memory, communication interface, input device, and output device are interconnected via a bus. The bus may include a path for transmitting information between various components of the computer system.

[0155] The processor may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0156] The processor may include a main processor, and may also include a baseband chip, a modem, etc.

[0157] The memory stores a computer program that executes the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include program code, and the program code includes computer operating instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash, etc.

[0158] The computer program may be written in any combination of one or more programming languages ​​to implement the operations of the embodiments of this specification, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Input devices may include devices that receive data and information input by a user, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0160] Output devices may include means that allow information to be output to a user, such as display screens, printers, speakers, and the like.

[0161] The communication interface may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0162] The processor executes the program stored in the memory and calls other devices, which can be used to implement each step of the bucket tooth monitoring method provided in the above embodiment of the present application.

[0163] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by instructing the relevant hardware using a computer program, and the program can be stored in a computer-readable storage medium, such as a read-only memory. Alternatively, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software functional modules. The present disclosure is not limited to any particular combination of hardware and software.

[0165] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or highly formal sense unless expressly defined as such herein.

[0166] The above is an illustration of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims. It should be understood that the above is an illustration of the present disclosure and should not be considered as limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A bucket tooth monitoring method, characterized in that: include: Acquire multiple bucket images of the bucket in a working state; Label the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image, and generate a label file; Based on the multiple bucket images and corresponding label files, a neural network model is trained and verified to obtain a trained bucket recognition model; Inputting the acquired multiple real-time bucket images into a bucket recognition model for recognition, and outputting recognition results of the multiple real-time bucket images, the recognition results including a recognition result of the overall structure of the bucket teeth in the real-time bucket images and a recognition result of each bucket tooth in the real-time bucket images; determining whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images; The step of marking the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image includes: Select the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure using a marking box according to the category name, and generate a category serial number, wherein the category serial number corresponds to the category name; The recognition result of the real-time bucket image includes: a prediction frame of the overall structure of the bucket tooth in the real-time bucket image, and a prediction frame of each bucket tooth in the real-time bucket image; The bucket teeth abnormality of the bucket includes tilting; Determining whether the bucket teeth of the bucket are tilted according to the recognition results of the multiple real-time bucket images includes: Calculate the real-time width of a bucket tooth based on the predicted frame of a bucket tooth in each real-time bucket image; Calculating a width ratio of the real-time width of the bucket tooth to the factory width according to the real-time width of the bucket tooth and the factory width of the bucket tooth; Recording the number of real-time bucket images whose width ratio exceeds a second preset threshold; When the number of real-time bucket images whose width ratio exceeds the second preset threshold value exceeds the third preset threshold value, it is determined that a certain bucket tooth of the bucket is tilted.

2. The bucket tooth monitoring method of claim 1, wherein: The method further comprises: The bucket image is preprocessed, and the preprocessing includes data cleaning and data enhancement. The data enhancement includes simulating rainy, foggy, dusty, and exposed scenes.

3. The bucket tooth monitoring method of claim 1, wherein: The bucket teeth of the bucket may be missing. Determining whether bucket teeth of the bucket are missing according to recognition results of the multiple real-time bucket images includes: Matching the predicted frame of the bucket tooth overall structure in each real-time bucket image with the predicted frame of each bucket tooth to determine a matching result; When the matching result is that the number of successful matches exceeds a first preset threshold, it is determined that the bucket teeth of the bucket are missing.

4. The bucket tooth monitoring method of claim 3, wherein: Match the predicted frame of the bucket tooth overall structure in each real-time bucket image with the predicted frame of each bucket tooth to determine the matching result, including: Determine a first predicted missing bucket tooth position according to a predicted frame of the bucket tooth overall structure in the real-time bucket image; Determining a position of a second predicted missing bucket tooth based on a predicted box of each bucket tooth in the real-time bucket image; and When the first predicted position of the missing bucket tooth is consistent with the second predicted position of the missing bucket tooth, the matching result is determined to be a successful match.

5. The bucket tooth monitoring method of claim 1, wherein: The bucket teeth abnormalities include wear; Determining whether the bucket teeth of the bucket are worn according to the recognition results of the multiple real-time bucket images includes: Calculate the real-time area of ​​a bucket tooth based on the mask of the bucket tooth in each real-time bucket image; Calculating an area ratio of the real-time area of ​​the bucket tooth to the preset area according to the real-time area of ​​the bucket tooth and the preset area; Recording the number of real-time bucket images whose area ratio exceeds a fourth preset threshold; When the number of real-time bucket images whose area ratio exceeds the fourth preset threshold value exceeds the fifth preset threshold value, it is determined that a tooth of the bucket is worn.

6. The bucket tooth monitoring method of claim 5, wherein: The preset area is the bucket tooth area when the bucket teeth leave the factory or the bucket tooth area of ​​the bucket tooth with the largest area in the real-time bucket image.

7. A bucket tooth monitoring device for a bucket, characterized in that: The device includes: an image acquisition module, an image processing module, a bucket recognition model and an abnormality determination module, wherein: The image acquisition module is used to obtain multiple bucket images when the bucket is in working state; The image processing module is configured to label the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image, and generate a label file; the labeling of the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure in each bucket image includes: selecting the bucket tooth overall structure and each bucket tooth included in the bucket tooth overall structure using a labeling box according to the category name, and generating a category serial number, wherein the category serial number corresponds to the category name; A bucket recognition model, the bucket recognition model being obtained by training and validating a neural network model based on the multiple bucket images and corresponding label files; the bucket recognition model being configured to recognize the multiple acquired real-time bucket images and output recognition results of the multiple real-time bucket images, the recognition results including recognition results of the overall structure of the bucket teeth in the real-time bucket images and recognition results of each bucket tooth in the real-time bucket images; The abnormality determination module is used to determine whether the bucket teeth of the bucket are abnormal based on the recognition results of the multiple real-time bucket images; the recognition results of the real-time bucket images include: the prediction frame of the overall structure of the bucket teeth in the real-time bucket image, and the prediction frame of each bucket tooth in the real-time bucket image; the abnormality of the bucket teeth of the bucket includes tilting; determining whether the bucket teeth of the bucket are tilted based on the recognition results of the multiple real-time bucket images includes: calculating the real-time width of a certain bucket tooth based on the prediction frame of a certain bucket tooth in each real-time bucket image; calculating the width ratio of the real-time width of the certain bucket tooth to the factory width based on the real-time width of the certain bucket tooth and the factory width of the certain bucket tooth; recording the number of real-time bucket images whose width ratio exceeds a second preset threshold; when the number of real-time bucket images whose width ratio exceeds the second preset threshold exceeds a third preset threshold, determining that a certain bucket tooth of the bucket is tilted.

8. An excavator, characterized in that: include: A bucket tooth integral structure, the bucket tooth integral structure comprising a plurality of bucket teeth; And the bucket tooth monitoring device of claim 7.

Images