Excavator loading counting method and device based on machine vision, terminal and medium

Abstract

The invention relates to the technical field of machine vision, and discloses an excavator loading counting method and device based on machine vision, a terminal and a medium, and the method comprises the steps: obtaining a current frame image collected according to a preset frame rate; inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; when the detection result comprises a target image, extracting a loading vehicle image from the target image; inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; and counting the loading quantity of the excavator according to the number identification result of the loading vehicle. The automatic identification of the loading vehicle and the automatic statistics of the loading quantity of the excavator are realized, the complexity and error-prone property of manual counting are avoided, the working efficiency of a mine field is improved, and the workload of operators is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a method, device, terminal and medium for counting loading of an excavator based on machine vision. Background Art

[0002] At present, open-pit mines mostly use excavators to load ore into loading trucks for transportation to designated locations. The mine environment is usually complex, the operating conditions are relatively harsh, there are a large number of excavators and loading trucks, and they need to cooperate with each other to complete the work effectively. In this working environment, how to count the number of loading trucks of each excavator in real time and accurately has become an important issue in mine management.

[0003] However, existing statistical methods mainly rely on manual counting to count the number of excavators loaded, which not only increases the workload of operators, but also easily leads to low statistical efficiency and human errors. Therefore, how to improve the accuracy and real-time performance of excavator loading statistics has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, terminal and medium for counting loading of an excavator based on machine vision, so as to solve the problem of low accuracy and real-time performance of the existing counting of loading of an excavator.

[0005] In a first aspect, a method for counting loading of an excavator based on machine vision is provided, comprising: Obtain the current frame image captured at a preset frame rate; Inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; When the detection result includes a target image, extracting a loading vehicle image from the target image; Inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; According to the loader number identification result, the number of loader loads by the excavator is counted.

[0006] In one embodiment, the target image is an image including a bucket and a loader; When the detection result includes a target image, extracting a loading vehicle image from the target image comprises: When the detection result includes the target image, extracting a first bucket image from the target image, and acquiring a previous frame image; Determining whether the previous frame image includes a second bucket image; If the previous frame image includes the second bucket image, extracting the second bucket image from the previous frame image; Inputting the first bucket image and the second bucket image into a pre-trained Siamese network model to obtain a similarity between outputs of the pre-trained Siamese network model; The loading vehicle image is extracted from the target image according to the similarity level.

[0007] In one embodiment, extracting the loading vehicle image from the target image according to the similarity degree includes: When the similarity is less than a preset threshold, extracting the loading vehicle image from the target image; When the degree of similarity is greater than or equal to the preset threshold, the next frame of image is acquired for processing.

[0008] In one embodiment, the detection result also includes the status of the loading vehicle; When the similarity is less than a preset threshold, extracting the loading vehicle image from the target image includes: When the similarity is less than the preset threshold, obtaining the state of the loading vehicle; Determining whether the loading vehicle is fully loaded; If the state of the loading vehicle is full, extracting the loading vehicle image from the target image; If the state of the loading vehicle is not full, the next frame of image is acquired for processing.

[0009] In one embodiment, counting the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number includes: Determining whether the loading vehicle number recognition result includes the first loading vehicle number; If the loading vehicle number recognition result includes the first loading vehicle number, obtaining the second loading vehicle number of the loading vehicle recorded last time; comparing the first loading vehicle number with the second loading vehicle number; When the first loading vehicle number and the second loading vehicle number are the same, acquiring the next frame of image for processing; When the first loading vehicle number and the second loading vehicle number are different, the number of loading vehicles loaded by the excavator is counted, and the second loading vehicle number of the loading vehicle recorded last time is updated as the current first loading vehicle number; If the loading vehicle number recognition result does not include the first loading vehicle number, extracting features from the loading vehicle image using a target algorithm to obtain first loading vehicle features; The number of vehicles loaded by the excavator is counted according to the first loader characteristic.

[0010] In one embodiment, the step of extracting features from the loading vehicle image using a target algorithm to obtain a first loading vehicle feature includes: Performing FAST corner point detection on the loading vehicle image to locate key corner points in the loading vehicle image; The ORB descriptor of the key corner point is extracted, and the ORB descriptor is used as the first loading vehicle feature.

[0011] In one embodiment, counting the number of vehicles loaded by the excavator according to the first loader feature includes: Get the second loading vehicle feature of the last recorded loading vehicle; comparing the first loader characteristic with the second loader characteristic; When the first loading vehicle feature and the second loading vehicle feature are the same, acquiring the next frame of image for processing; When the first loader feature is different from the second loader feature, the number of vehicles loaded by the excavator is counted, and the second loader feature of the loader recorded last time is updated as the current first loader feature.

[0012] In a second aspect, a device for counting loading of an excavator based on machine vision is provided, comprising: An acquisition module, used to acquire a current frame image acquired at a preset frame rate; A first input module, used to input the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; an extraction module, for extracting a loading vehicle image from the target image when the detection result includes the target image; A second input module is used to input the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; A statistics module is used to count the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number.

[0013] In a third aspect, an intelligent terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the excavator loading counting method based on machine vision as described in the first aspect is implemented.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the excavator loading counting method based on machine vision described in the first aspect is implemented.

[0015] In one solution implemented by the above-mentioned machine vision-based excavator loading counting method, device, terminal and medium, the method includes: obtaining a current frame image acquired at a preset frame rate; inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; when the detection result includes a target image, extracting a loading vehicle image from the target image; inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; and counting the number of loading vehicles of the excavator according to the loading vehicle number recognition result. In this embodiment, by acquiring images at a preset frame rate and using a pre-trained target detection model, the loading vehicle image can be accurately located, the detection accuracy and efficiency can be improved, and the loading vehicle number can be identified by a pre-trained optical character recognition model, thereby realizing automatic identification of the loading vehicle and automatic counting of the number of loading vehicles of the excavator, avoiding the tediousness and error-proneness of manual counting, improving the working efficiency of the mine, and reducing the workload of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 Schematic diagram of an application environment of an excavator loading counting method based on machine vision in one embodiment of the present invention; Figure 2 is a flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 3 is another flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 4 is another flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 5 is another flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 6is another flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 7 is another flow chart of a method for counting loading vehicles of an excavator based on machine vision in one embodiment of the present invention; Figure 8 is a schematic diagram of an excavator loading counting device based on machine vision in one embodiment of the present invention; Fig. 9 is a schematic diagram of an intelligent terminal in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The excavator loading counting method based on machine vision provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the excavator loading counting method based on machine vision is applied in an excavator loading counting system based on machine vision, and the excavator loading counting system based on machine vision includes: Figure 1 The camera device and the smart terminal shown in the figure communicate with the camera device through a network to implement the excavator loading counting method based on machine vision. The camera device is installed on the excavator, and the smart terminal can include but is not limited to various personal computers, laptops, smart phones, tablet computers and excavator vehicles.

[0020] In order to facilitate understanding of the embodiments of the present invention, the target detection model, the twin network model and the optical character recognition model involved in the present invention are first explained as follows: Target detection model: A neural network algorithm used to identify specified objects in an image and determine the bounding box of the object. The specific type of the target detection model can be a two-stage target detection model, such as R-CNN, Fast R-CNN, SPP-NET, etc., or a single-stage target detection model, such as SSD, YOLO, RetinaNet, etc., or a target detection model with a self-attention mechanism, such as DETR, Deformable DETR, Dynamic DETR, etc. The present invention can use any target detection model for bucket and loader detection, which does not constitute a limitation of the present invention.

[0021] Twin network model: a neural network model used to judge the similarity between two images. Among them, the specific type of the twin network model can be a twin network model based on a Siamese network, which processes two images separately with the help of two identical sub-networks with shared parameters to extract features, and then judges the similarity based on the feature vector distance; it can also be a twin network model based on contrastive learning, which learns similarity by comparing positive and negative sample pairs, bringing the features of the positive sample pair closer and the features of the negative sample pair farther away; it can also be a twin network model based on a triplet network, which uses the relationship between the anchor image, the positive sample image and the negative sample image for training and learning to achieve the judgment of the similarity of the two images. The present invention can use any twin network model to compare the similarity of the buckets of the two frames of images before and after, and this does not constitute a limitation of the present invention.

[0022] Optical character recognition model (OCR model): a neural network model used to detect the position of characters in an image and recognize the characters. The specific type of the optical character recognition model can be an OCR model based on template matching, an OCR model based on statistical feature recognition, or a neural network OCR model based on a convolutional neural network (CNN). The present invention can use any OCR model, and this does not constitute a limitation of the present invention.

[0023] A method for counting loading vehicles for an excavator based on machine vision provided by the present invention will be described in detail through the following specific embodiments.

[0024] In one embodiment, if Figure 2 As shown in the figure, a method for counting loading vehicles of excavators based on machine vision is provided. Figure 1 The following steps are used as an example to illustrate the smart terminal in FIG. 1 : S201: Acquire a current frame image captured at a preset frame rate.

[0025] In this embodiment, the preset frame rate refers to a preset image acquisition frequency, that is, the number of frames of images collected from the camera device per minute. For example, the preset frame rate is set to 10 frames per minute. The preset frame rate can be set according to actual detection requirements and image algorithm processing speed, and is not limited here.

[0026] In some embodiments, a camera device installed on the excavator can be used to capture video images of the excavator at work in real time, and then the corresponding current frame image can be cropped from the real-time captured video image at a preset frame rate; or, the current frame image can be directly captured by the camera device at a preset frame rate; or, the current frame image can be captured by other methods, which are not limited here.

[0027] The camera device involved here covers various types, for example, various common cameras (for example, ordinary cameras or infrared night vision cameras), cameras, and other terminals with photo or video functions are all within the protection scope of the present invention. Preferably, the camera device can be installed in the upper center of the excavator cab glass.

[0028] It should be noted that this installation method is only an example among many feasible solutions and does not constitute a substantial restriction on the present invention as a whole, as long as the installation position of the camera device adopted can ensure the effective collection of image information of the excavator working scene and meet the subsequent image analysis and processing requirements.

[0029] S202: Input the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model.

[0030] In this embodiment, the detection results may include the target image, the status of the loader, and other images, etc., or may not include the target image, which is not a limitation here. Among them, the target image refers to an image that includes both the excavator bucket and the loader, that is, the target image refers to an image that includes the bucket and the loader; the status of the loader includes full or not full; other images may be images taken by the excavator during idling, for example, only an image of the excavator bucket, or only an image of the loader, or an image of the excavator bucket in a suspended non-operating position, with no loader or other work-related objects around the excavator. The description of the above detection results is only for example and does not constitute a limitation of the present invention.

[0031] In some embodiments, after the current frame image is acquired according to step S201, the current frame image can be directly input into a pre-trained target detection model to obtain the detection result corresponding to the current frame image output by the pre-trained target detection model. This process can quickly obtain the corresponding detection result output by the pre-trained target detection model for the current frame image, which is direct and simple and can quickly advance the subsequent process; the current frame image can also be pre-processed to obtain the pre-processed current frame image, wherein the pre-processing of the current frame image can include grayscale processing, noise reduction processing, and normalization processing of the current frame image, which are not limited here; further, the pre-processed current frame image is input into a pre-trained target detection model to obtain the detection result corresponding to the current frame image output by the pre-trained target detection model. This process improves the adaptability and stability of the pre-trained target detection model to various images, and improves the reliability and accuracy of the final detection result.

[0032] It should be understood that the above two methods are only examples, and the present invention may also undergo other processing, as long as the detection result corresponding to the current frame image can be output by the trained target detection model, which is not limited here.

[0033] As an example, assuming that a deep convolutional neural network model is used, after the current frame image is input, the convolution layer of the trained target detection model will extract features from the current frame image. Then, the fully connected layer integrates and judges the features. For example, when an excavator bucket and a loader are detected in the current frame image, their positions in the current frame image will be determined to generate a target image including the two, and the state of the loader will be further analyzed to determine whether it is full. It should be understood that the bucket and loader on the target image will be selected by the target frame for subsequent recognition and counting operations.

[0034] It should be noted that when multiple loaders are identified at the same time, only the loaders with the largest target frame area are retained, and the target frames of the remaining loaders are deleted. For example, the trained target detection model detects three loaders in the picture, which are marked as loaders A, loaders B and loaders C. Among them, the target frame area of ​​loader A is 50 square units, the target frame area of ​​loader B is 35 square units, and the target frame area of ​​loader C is 60 square units. According to the rules, the target frame of loader C will be retained, and the target frames of loaders A and loaders B will be deleted, so as to reduce the problem of multiple loaders in the mine, resulting in inaccurate loading counts of the current excavator, which does not constitute a limitation of the present invention.

[0035] In one embodiment, the pre-trained object detection model is trained in the following manner: S202A: Obtain video data of the excavator at work.

[0036] In some embodiments, a camera device may be used to collect video data of an excavator at work, wherein the video data includes but is not limited to video data of an excavator at work under different lighting conditions, different excavators, and different working environments.

[0037] It should be understood that the description of the camera device can refer to the description of the above step S201 embodiment, and in order to avoid repetition, it will not be repeated here.

[0038] S202B, storing the video data as images, and annotating and preprocessing the images to obtain a training image set and a test image set.

[0039] In some embodiments, the acquired video data of the excavator at work can be first extracted and stored as images according to a specific frame rate. Subsequently, these images are annotated, with the bucket turned down and the loader in different states (full or not full) located in front of the excavator as the key targets, and the rectangular frame is used to accurately define the position range in the image and assign the corresponding category label. After the annotation is completed, the image is preprocessed, for example, grayscale processing, noise reduction processing and normalization processing are performed on the image. Further, the processed images are divided into a training image set and a test image set according to a certain ratio, wherein the training image set is used for the subsequent training of the target detection model so that the target detection model can learn the characteristic patterns and category judgment basis of the bucket and the loader, and the test image set is used to evaluate and verify the effect of the trained target detection model, so as to determine whether the trained target detection model meets the expected performance standards and requirements. It should be noted that the above is only an example and does not constitute a limitation of the present invention.

[0040] S202C, training the target detection model based on the training image set to obtain a trained target detection model; S202D. Based on the test image set, test the trained object detection model to obtain a first test result.

[0041] In some embodiments, after processing from step S202A to step S202B, a training image set and a test image set for training a target detection model are obtained; further, the target detection model can be first trained with the training image set to obtain a trained target detection model; then, the trained target detection model is tested with the test image set to obtain a first test result; finally, it is determined whether the first test result meets the first preset requirement, for example, whether the accuracy rate reaches the target accuracy rate, and whether the recall rate reaches the target recall rate, so as to determine whether the first test result meets the first preset requirement. If the first test result meets the first preset requirement, step S202E is executed; if the first test result does not meet the first preset requirement, step S202F is executed.

[0042] As an example, suppose the training image set contains 10,000 images, each labeled with a loader, bucket, and loader status. These images are fed into the object detection model for training. During the training process, the object detection model gradually adjusts its weights and parameters with the goal of accurately locating the loader, bucket, and loader status given a new image.

[0043] After the training is completed, the trained object detection model will be tested using the test image set. Assume that the test image set contains 1,000 images, and the scenes of these images are slightly different from those of the training image set, for example, images of a loader and a bucket under different lighting conditions or at different angles. Further, the performance of the trained object detection model on the test image set is tested to obtain a first test result. Next, it is determined whether the first test result meets the first preset requirement.

[0044] S202E: If the first test result meets the first preset requirement, use the trained target detection model as a pre-trained target detection model; S202F. If the first test result does not meet the first preset requirement, adjust the model parameters of the trained target detection model and / or increase the training image set, and repeat steps S202C to S202E until the first test result meets the first preset requirement.

[0045] In some embodiments, if the first test result meets the first preset requirement, it indicates that the trained target detection model performs well in the detection of the test image set, and its indicators such as the recognition accuracy of the bucket and the loader, and the accuracy of judging the state of the loader, all meet the expected standards. At this time, the target detection model after the above training is used as a pre-trained target detection model. If the first test result does not meet the first preset requirement, it indicates that the trained target detection model cannot meet the expected standard. At this time, the model parameters of the trained target detection model can be adjusted, for example, the model parameters such as the learning rate and the number of iterations can be adjusted, and / or the training data set can be increased. Subsequently, steps S202C to S202E are repeated again until the first test result meets the first preset requirement.

[0046] It should be noted that the above model training process is only an example of the present invention. Specifically, the pre-trained target detection model of the present invention can be obtained by training in other ways, which does not constitute a limitation of the present invention.

[0047] S203: When the detection result includes a target image, extract a loading vehicle image from the target image.

[0048] In some embodiments, after obtaining the detection result outputted in step S202, it is first determined whether the detection result includes the target image. When the detection result includes the target image, the loading vehicle image is extracted from the target image. As an example, the target image can be cropped according to the detected loading vehicle target frame to obtain the loading vehicle image. When the detection result does not include the target image, the next frame of the image is obtained for processing. By first determining whether the detection result includes the target image, the accuracy of the excavator loading count can be effectively improved.

[0049] As an example, assume that the detection result includes a target image, that is, an excavator bucket and a loader. Further, an operation is performed according to the target frame information about the loader in the detection result. This target frame accurately defines the position and range of the loader in the target image. For example, the coordinate information of the target frame is (x1, y1, x2, y2), which represents the coordinates of the upper left corner vertex (x1, y1) and the lower right corner vertex (x2, y2) of the loader image in the entire target image.

[0050] Then, the target image is cropped according to the target frame. Through image processing software or related programming code (such as using the cropping function of the OpenCV library in Python), the corresponding image area is cut out from the entire target image according to the coordinate range of the target frame. After such cropping, the loader image can be obtained.

[0051] S204: Input the loading truck image into a pre-trained optical character recognition model to obtain a loading truck number recognition result output by the pre-trained optical character recognition model.

[0052] In this embodiment, the loader number recognition result includes recognizing the loader number, or not recognizing the loader number, that is, the loader number recognition result includes the first loader number, or the loader number recognition result does not include the first loader number.

[0053] The loader number refers to a code or serial number used to uniquely identify different loader trucks. The loader number can be in various forms, including pure digital numbers such as 001, 002, 003, and so on; or a combination of letters and numbers, such as A01, B02, etc., which are not limited here.

[0054] It should be understood that the loader number should be marked on the loader body and the camera device can capture it. The specific marking position is not limited in the present invention.

[0055] In some embodiments, after the loading vehicle image is cropped in step S203, the cropped loading vehicle image can be further input into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result corresponding to the loading vehicle image output by the pre-trained optical character recognition model. By recognizing the loading vehicle number through another model, it is possible to more accurately recognize and extract the number characters that may exist in the loading vehicle image, thereby obtaining a loading vehicle number recognition result.

[0056] It should be noted that in this process, the current frame image can also be cropped according to the loader target frame in the target image to obtain the loader image, and the current frame image can be further cropped to obtain the loader image for recognition, which is not limited here. Among them, when the loader image cropped from the current frame image is recognized, the cropped loader image can be directly recognized, or the cropped loader image can be preprocessed, and then the preprocessed loader image is recognized, which is not limited here.

[0057] In one embodiment, the pre-trained optical character recognition model is trained in the following manner: S204A, obtaining a training loader image set and a test loader image set; S204B, training the optical character recognition model based on the training loader image set to obtain a trained optical character recognition model; S204C: Based on the test loader image set, test the trained optical character recognition model to obtain a second test result.

[0058] S204D, if the second test result meets the second preset requirement, using the trained optical character recognition model as a pre-trained optical character recognition model; S204E: If the second test result does not meet the second preset requirement, adjust the model parameters of the trained optical character recognition model and / or increase the training loader image set, and repeat steps S204B to S204C until the second test result meets the second preset requirement.

[0059] In some embodiments, an optical character recognition model is trained using a training loader image set, tested using a test loader image set, and a decision is made based on whether the test results meet preset requirements whether to adopt the trained optical character recognition model or adjust model parameters and increase training data until the requirements are met. This can significantly improve the accuracy of the model in recognizing characters in loader images and ensure that the model ultimately adopted has efficient target detection capabilities.

[0060] It should be understood that the specific process of obtaining the pre-trained optical character recognition model in the above steps S204A to S204E can refer to the embodiment description of obtaining the pre-trained target detection model in steps S202A to S202F, and will not be repeated here to avoid repetition.

[0061] S205: Count the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number.

[0062] In some embodiments, after obtaining the loader number recognition result, it is necessary to first determine whether the loader number recognition result includes the loader number; if the loader number recognition result includes the loader number, the loader number obtained by this recognition is compared with the loader number recorded last time. When it is detected that the loader number has changed, it is determined that the excavator has completed a new loading operation, so the loading quantity is increased by one; if the loader number remains unchanged, it means that the excavator is still loading the same loader, and the loading quantity remains unchanged.

[0063] This process can update the excavator's loading operation statistics in real time, significantly improving operation efficiency and statistical accuracy. It is also applicable to a variety of complex environments such as mines and docks, facilitating subsequent data analysis and scheduling optimization.

[0064] As an example, suppose that in the first frame image, the number of the loader detected is "001", the system records it as the current loader, and initializes the number of loading of the excavator to 1. When processing the next frame image, the number of the loader detected is "002". At this time, it is judged that the current loader number is inconsistent with the number "001" recorded last time. According to this change, it is determined that the current excavator has completed a new loading operation, so the number of loading is increased by 1 and the number "002" is recorded. If the number "001" is detected again in the subsequent frame image, the above logic will be repeated to ensure that the statistics of multiple loadings are accurate. For example, in the continuous operation process, if the number sequence is detected in sequence as "001", "002", "001" and "003", the final number of loading of the excavator is 4, which can accurately reflect the number of loading of the loader. In this way, the number of loading of the excavator can be updated in real time, and the alternating operation scenes of multiple loaders can be effectively handled, thereby ensuring the accuracy and completeness of the statistical results.

[0065] To sum up, in a solution provided by an embodiment of the present invention, by capturing images at a preset frame rate and using a pre-trained target detection model, the loader image can be accurately located, the detection accuracy and efficiency can be improved, and the loader number can be identified through a pre-trained optical character recognition model, thereby realizing automatic identification of the loader and automatic counting of the number of excavators loaded, avoiding the tediousness and error-proneness of manual counting, improving the work efficiency of the mine, and reducing the workload of the operator.

[0066] In one embodiment, if Figure 3 As shown, in step S203, the target image is an image including a bucket and a loader; That is, when the detection result includes a target image, extracting the loading vehicle image from the target image includes the following steps: S2031. When the detection result includes the target image, extracting a first bucket image from the target image, and acquiring a previous frame image; S2032, determining whether the previous frame image includes a second bucket image; S2033, if the previous frame image includes the second bucket image, extracting the second bucket image from the previous frame image; S2034, inputting the first bucket image and the second bucket image into a pre-trained twin network model to obtain a similarity degree of output of the pre-trained twin network model; S2035, extracting the loading vehicle image from the target image according to the similarity degree; S2036: If the previous frame image does not include the second bucket image, obtain the next frame image for processing.

[0067] In this embodiment, the degree of similarity refers to the similarity measurement value between two images calculated by the twin network model.

[0068] The pre-trained Siamese network model consists of two identical neural networks that share weights, process two input images separately, and evaluate the similarity of the images by calculating the distance between the feature vectors of the two images. For example, the similarity of the two images is calculated by Euclidean distance or cosine similarity.

[0069] The Euclidean distance calculation formula is as follows: in, It represents the feature vector of the first bucket image; It represents the feature vector of the second bucket image; and It represents the first of these two eigenvectors. The weight of the dimension, is the dimension of the feature vector.

[0070] In practical applications, if the calculated Euclidean distance is small, it means that the two images are relatively close in the feature space, indicating that the two images are similar, and the bucket is considered to be in a non-dumped state; if the distance is large, it means that there is a large difference between the two images, indicating that the two images are not similar, and the bucket is considered to be in a dumped state.

[0071] The cosine similarity calculation formula is as follows: in, It represents the dot product of two eigenvectors; are the norms of Lian Gege’s eigenvectors respectively.

[0072] In practical applications, the value range of cosine similarity is [-1, 1]. The closer the value is to 1, the more similar the two images are, and the bucket is considered to be in a non-dumped state; the closer the value is to -1, the more dissimilar the two images are, and the bucket is considered to be in a dumped state.

[0073] As an example, assume that the feature vector of the first bucket image is , the feature vector of the second bucket image is The similarity between the first bucket image and the second bucket image is calculated using the Euclidean distance calculation formula as follows: .

[0074] It should be understood that the above is merely an example of calculating the degree of similarity of a model and does not constitute a limitation of the present invention.

[0075] In some embodiments, when it is determined that the detection result includes the target image, the first bucket image is first extracted from the target image, and the previous frame image is obtained. Next, it is determined whether the bucket image is detected in the previous frame image. If the previous frame image includes the second bucket image, the second bucket image is extracted from the previous frame image, and the first bucket image and the second bucket image are simultaneously input into the pre-trained twin network model. The pre-trained twin network model will extract the feature vector of each image and calculate the similarity between the two feature vectors to determine the similarity between the two images, thereby determining whether the loader image is extracted from the target image based on the similarity, as described in steps S20351 to S20352. If the previous frame image does not include the second bucket image, the next frame image is obtained for processing, or, when the detection result does not include the target image, the next frame image is obtained for processing.

[0076] It should be understood that the above is only an example and does not constitute a limitation of the present invention.

[0077] In one embodiment, the pre-trained twin network model is trained in the following manner: S2034A, obtaining a training bucket image set and a test bucket image set; S2034B, training the twin network model based on the training bucket image set to obtain a trained twin network model; S2034C. Based on the test bucket image set, test the trained twin network model to obtain a third test result.

[0078] S2034D. If the third test result meets the third preset requirement, the trained twin network model is used as the pre-trained twin network model; S2034E. If the third test result does not meet the third preset requirement, adjust the model parameters of the trained twin network model and / or increase the training bucket image set, and repeat steps S2034B to S2034C until the second test result meets the second preset requirement.

[0079] In some embodiments, a training bucket image set and a test bucket image set containing different bucket states are obtained. Next, the twin network model is trained using the training bucket image set to generate a preliminarily trained twin network model. Then, the trained model is tested based on the test bucket image set to obtain a third test result. If the test result meets the preset accuracy requirements (for example, the recognition accuracy rate reaches 85% or more), the trained twin network model is considered valid and is used as a pre-trained twin network model. If the test result fails to meet the requirements, the model parameters are adjusted or the training set samples are increased, and retraining and testing are performed until the test results meet the accuracy standards.

[0080] This process ensures that the trained twin network model can accurately judge the similarity of bucket images, thereby supporting subsequent loading and counting processing.

[0081] It should be understood that the specific process of obtaining the pre-trained twin network model in the above steps S2034A to S2034E can also refer to the embodiment description of obtaining the pre-trained target detection model in steps S202A to S202F. In order to avoid repetition, it will not be repeated here.

[0082] In one embodiment, if Figure 3 As shown, step S2035, that is, extracting the loading vehicle image from the target image according to the similarity, includes the following steps: S20351, when the similarity is less than a preset threshold, extracting the loading vehicle image from the target image; S20352: When the degree of similarity is greater than or equal to the preset threshold, obtain the next frame of image for processing.

[0083] In some embodiments, after obtaining the similarity between the first bucket image and the second bucket image, further, the working state of the excavator bucket is judged according to the similarity and the preset threshold. If the similarity is less than the preset threshold, it indicates that the current bucket is in the state of dumping ore, and the image of the loading vehicle is cropped from the target image. Otherwise, it indicates that the current bucket is not in the state of dumping ore, so as to avoid the situation where the excavator is idling and taking images of the loading vehicle that is not loaded by itself, effectively eliminating the misjudgment caused by the excavator idling and taking pictures of irrelevant loading vehicles, and greatly improving the accuracy of operation status monitoring.

[0084] The above process helps the system distinguish different working states of the bucket by judging the changes in the bucket image, so as to perform subsequent loader counting and status monitoring tasks more accurately.

[0085] As an example, assuming that the preset threshold is set to 0.85, when processing a certain frame of image, the similarity calculated by the twin network model is 0.75, which is lower than the preset threshold of 0.85, indicating that the bucket is in the state of dumping ore. At this time, the loader image is extracted from the frame of image for subsequent processing. If the similarity is 0.90 in the next frame of image, it indicates that the bucket is not in the state of dumping ore, and the next frame of image is directly obtained for processing. In this way, unnecessary image processing is avoided, and the processing efficiency of the system is improved.

[0086] It should be understood that the preset threshold is a critical value preset by the user, which can be specifically adjusted according to different application scenarios and requirements, and is not limited here. The above embodiment is only an example and does not constitute a limitation of the present invention.

[0087] In one embodiment, if Figure 4 As shown, in step S20351, the detection result also includes the state of the loading vehicle; That is, when the similarity is less than a preset threshold, extracting the loading vehicle image from the target image includes the following steps: S20351A, when the similarity is less than the preset threshold, obtaining the state of the loading vehicle; S20351B, determining whether the loading vehicle is fully loaded; S20351C, if the state of the loading vehicle is full, extracting the loading vehicle image from the target image; S20351D: If the loading vehicle is not full, obtain the next frame of image for processing.

[0088] In this embodiment, the detection result also includes the state of the loading vehicle, wherein the state of the loading vehicle includes being fully loaded or not fully loaded.

[0089] In some embodiments, when the similarity between the first bucket image and the second bucket image is less than a preset threshold, it indicates that the bucket is in a dumping state. At this time, the state of the loading vehicle output by the pre-trained target detection model is obtained, and it is determined whether the state of the loading vehicle is full. If the state of the loading vehicle is full, the loading vehicle image is extracted from the target image for subsequent statistics and processing. If the state of the loading vehicle is not full, it is considered that the loading vehicle has not yet reached the counting condition, so the next frame of image is obtained for processing to avoid invalid image extraction and loading quantity statistics.

[0090] This process ensures that the image of the loading vehicle is extracted and the number of loaded vehicles is accurately counted only after the loading vehicle has completed loading, thereby improving the efficiency and accuracy of the system.

[0091] In one embodiment, if Figure 5 As shown, step S205, that is, counting the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number, includes the following steps: S2051, determining whether the loading vehicle number recognition result includes the first loading vehicle number; S2052: If the loading vehicle number recognition result includes the first loading vehicle number, obtaining the second loading vehicle number of the loading vehicle recorded last time; S2053, comparing the first loading vehicle number with the second loading vehicle number; S2054: when the first loading vehicle number and the second loading vehicle number are the same, obtaining the next frame of image for processing; S2055: When the first loading vehicle number and the second loading vehicle number are different, count the number of loading vehicles loaded by the excavator, and update the second loading vehicle number of the last recorded loading vehicle to the current first loading vehicle number; S2056: If the loading vehicle number recognition result does not include the first loading vehicle number, extract features from the loading vehicle image using a target algorithm to obtain first loading vehicle features; S2057. Count the number of vehicles loaded by the excavator according to the characteristics of the first loader.

[0092] In this embodiment, the target algorithm refers to an algorithm used to extract features from a loader image, including but not limited to FAST corner detection, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0093] In some embodiments, after obtaining the loading vehicle number recognition result outputted in step S204, it is further necessary to determine whether the loading vehicle number recognition result includes the first loading vehicle number, so as to avoid the situation where the loading vehicle number recognition fails due to stains during operation of the loading vehicle.

[0094] If the loader number recognition result includes the first loader number, the second loader number of the last recorded loader is obtained. Further, the first loader number can be compared with the second loader number through a matching algorithm; if the loader number in the current frame image is consistent with the loader number recorded last time, that is, the first loader number is consistent with the second loader number, it indicates that the same loader is full and has not left, so the next frame image is obtained for processing, and the number of excavator loading is not increased. If the loader number in the current frame image is different from the loader number recorded last time, that is, the first loader number is inconsistent with the second loader number, it is considered that this is a new loader, and the number of excavator loading is counted once, and the second loader number of the last recorded loader is updated to the current first loader number. If the first loader number is not identified in a certain frame image, the loader image is feature extracted by the target algorithm to obtain the first loader feature, and based on the first loader feature, it is determined whether to count the number of excavator loading.

[0095] Through the above methods, the system's adaptability to different situations is enhanced, making the entire loading operation management more scientific and reasonable. It can play a strong supporting role in both the evaluation of excavator operating efficiency and the optimization of loading vehicle allocation, reduce decision-making errors and resource waste caused by data errors, and effectively avoid repeated counting of the same loading vehicle that has not left the site, ensuring that the loading quantity statistics are accurate and correct, providing a reliable basis for subsequent production scheduling, resource allocation and workload accounting.

[0096] In one embodiment, if Figure 6 As shown, in step S2056, that is, using the target algorithm to extract features from the loading vehicle image to obtain the first loading vehicle feature, the following steps are included: S2056A, performing FAST corner point detection on the loading vehicle image to locate key corner points in the loading vehicle image; S2056B, extracting the ORB descriptor of the key corner point, and using the ORB descriptor as the first loader feature.

[0097] In this embodiment, FAST (Features from Accelerated Segment Test) is a commonly used corner detection algorithm, which detects corners in an image by checking the brightness changes around pixels. These corners usually contain unique information in the image and help distinguish different images.

[0098] As an example, when performing FAST corner detection on a loader image, a suitable corner detection threshold can be set, such as a grayscale difference threshold of 20, and a judgment is made within a neighborhood with a circle radius of 3 pixels centered on the pixel. When there are 9 consecutive pixels around a pixel whose grayscale difference is greater than 20, the pixel is determined to be a key corner. For example, multiple key corners such as the edge of the loader compartment and the corner of the cab are located.

[0099] Next, the ORB algorithm is used to extract the ORB descriptors of these key corner points. For example, for the key corner points at the edge of the car body, the ORB descriptor includes the coordinate position of the key corner point in the image, the scale of the area where it is located, and the direction information relative to the overall image, etc. The extracted ORB descriptor is used as the first loading vehicle feature. In one embodiment, if Figure 7 As shown, step S2057, that is, counting the number of vehicles loaded by the excavator according to the first loader characteristics, includes the following steps: S2057A, obtaining the second loading vehicle feature of the loading vehicle recorded last time; S2057B, comparing the first loading vehicle feature with the second loading vehicle feature; S2057C: when the first loading vehicle feature and the second loading vehicle feature are the same, obtaining the next frame of image for processing; S2057D: When the first loader characteristic is different from the second loader characteristic, count the number of loader loadings by the excavator, and update the second loader characteristic of the last recorded loader as the current first loader characteristic.

[0100] In some embodiments, first, obtain the second loading vehicle feature of the last recorded loading vehicle. This feature can be the loading vehicle feature obtained by the aforementioned FAST corner point detection and ORB descriptor extraction, which is used to identify the feature of the last loading vehicle. Then, the first loading vehicle feature extracted in step S2056 is compared with the second loading vehicle feature. The purpose of this step is to confirm whether it is the same loading vehicle. If the first loading vehicle feature and the second loading vehicle feature are the same, it means that the loading vehicle currently identified is the same vehicle as the last recorded one. At this time, the next frame image is directly obtained for processing without updating the number of loading vehicles. On the contrary, if the first loading vehicle feature and the second loading vehicle feature are not the same, it means that the loading vehicle currently identified is not the same vehicle as the last recorded one. At this time, the number of loading vehicles of the excavator is counted, the current is marked as a new loading operation, and the second loading vehicle feature of the last recorded loading vehicle is updated to the current first loading vehicle feature.

[0101] This method can further determine whether it is the same loader by feature comparison when the pre-trained optical character recognition model cannot recognize the loader number, thereby effectively reducing the occurrence of errors in the statistical number of excavator loading vehicles and improving the accuracy of the statistics.

[0102] For example, in a mine operation, if the feature of the loader identified in the previous frame is A and the feature of the loader identified in the current frame is B, and the features of the two are different, it means that they are different loader trucks. At this time, the number of excavator loading vehicles will be increased and the record will be updated. If the loader feature of the current frame image is consistent with the loader feature of the last record, it means that they are the same loader truck. At this time, the next frame image will be obtained for processing to avoid repeated statistics.

[0103] It should be noted that the above is only an example and does not constitute a limitation of the present invention.

[0104] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0105] In a second aspect, a device for counting the loading of an excavator based on machine vision is provided, and the device for counting the loading of an excavator based on machine vision corresponds one to one with the method for counting the loading of an excavator based on machine vision in the above-mentioned embodiment. Figure 8 As shown, the excavator loading counting device based on machine vision includes an acquisition module 101, a first input module 102, an extraction module 103, a second input module 104 and a statistical module 105. The detailed description of each functional module is as follows: An acquisition module 101 is used to acquire a current frame image captured at a preset frame rate; A first input module 102 is used to input the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; An extraction module 103, configured to extract a loading vehicle image from the target image when the detection result includes the target image; A second input module 104 is used to input the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; The statistics module 105 is used to count the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number.

[0106] In one embodiment, the target image is an image including a bucket and a loader; In one embodiment, the extraction module 103 is specifically used to: When the detection result includes the target image, extracting a first bucket image from the target image, and acquiring a previous frame image; Determining whether the previous frame image includes a second bucket image; If the previous frame image includes the second bucket image, extracting the second bucket image from the previous frame image; Inputting the first bucket image and the second bucket image into a pre-trained Siamese network model to obtain a similarity between outputs of the pre-trained Siamese network model; The loading vehicle image is extracted from the target image according to the similarity level.

[0107] In one embodiment, the extraction module 103 is further configured to: When the similarity is less than a preset threshold, extracting the loading vehicle image from the target image; When the degree of similarity is greater than or equal to the preset threshold, the next frame of image is acquired for processing.

[0108] In one embodiment, the detection result also includes the status of the loading vehicle; The extraction module 103 is further specifically used for: When the similarity is less than the preset threshold, obtaining the state of the loading vehicle; Determining whether the loading vehicle is fully loaded; If the state of the loading vehicle is full, extracting the loading vehicle image from the target image; If the state of the loading vehicle is not full, the next frame of image is acquired for processing.

[0109] In one embodiment, the statistics module 105 is specifically used for: Determining whether the loading vehicle number recognition result includes the first loading vehicle number; If the loading vehicle number recognition result includes the first loading vehicle number, obtaining the second loading vehicle number of the loading vehicle recorded last time; comparing the first loading vehicle number with the second loading vehicle number; When the first loading vehicle number and the second loading vehicle number are the same, acquiring the next frame of image for processing; When the first loading vehicle number and the second loading vehicle number are different, the number of loading vehicles loaded by the excavator is counted, and the second loading vehicle number of the loading vehicle recorded last time is updated as the current first loading vehicle number; If the loading vehicle number recognition result does not include the first loading vehicle number, extracting features from the loading vehicle image using a target algorithm to obtain first loading vehicle features; The number of vehicles loaded by the excavator is counted according to the first loader characteristic.

[0110] In one embodiment, the statistics module 105 is further configured to: Performing FAST corner point detection on the loading vehicle image to locate key corner points in the loading vehicle image; The ORB descriptor of the key corner point is extracted, and the ORB descriptor is used as the first loading vehicle feature.

[0111] In one embodiment, the statistics module 105 is further configured to: Get the second loading vehicle feature of the last recorded loading vehicle; comparing the first loader characteristic with the second loader characteristic; When the first loading vehicle feature and the second loading vehicle feature are the same, acquiring the next frame of image for processing; When the first loader feature is different from the second loader feature, the number of vehicles loaded by the excavator is counted, and the second loader feature of the loader recorded last time is updated as the current first loader feature.

[0112] For the specific definition of the excavator loading counting device based on machine vision, please refer to the definition of the excavator loading counting method based on machine vision above, which will not be repeated here. Each module in the above-mentioned excavator loading counting device based on machine vision can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the smart terminal in the form of hardware, or can be stored in the memory of the smart terminal in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0113] In a third aspect, a smart terminal is provided. The smart terminal may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The intelligent terminal includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the intelligent terminal is used to store data required for executing an excavator loading counting method based on machine vision. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for counting excavator loading based on machine vision is implemented.

[0114] In one embodiment, a smart terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain the current frame image captured at a preset frame rate; Inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; According to the detection result, extracting a loading vehicle image from the current frame image; Inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; According to the loader number identification result, the number of loader loads by the excavator is counted.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain the current frame image captured at a preset frame rate; Inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; According to the detection result, extracting a loading vehicle image from the current frame image; Inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; According to the loader number identification result, the number of loader loads by the excavator is counted.

[0116] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0117] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0118] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for counting loading of excavators based on machine vision, characterized in that: include: Obtain the current frame image captured at a preset frame rate; Inputting the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; When the detection result includes a target image, extracting a loading vehicle image from the target image; Inputting the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; The number of vehicles loaded by the excavator is counted according to the loader number identification result.

2. The method for counting loading of an excavator based on machine vision according to claim 1, characterized in that: The target image is an image including a bucket and a loader; When the detection result includes a target image, extracting a loading vehicle image from the target image comprises: When the detection result includes the target image, extracting a first bucket image from the target image, and acquiring a previous frame image; Determining whether the previous frame image includes a second bucket image; If the previous frame image includes the second bucket image, extracting the second bucket image from the previous frame image; Inputting the first bucket image and the second bucket image into a pre-trained Siamese network model to obtain a similarity between outputs of the pre-trained Siamese network model; The loading vehicle image is extracted from the target image according to the similarity level.

3. The method for counting loading of an excavator based on machine vision as claimed in claim 2, characterized in that: The step of extracting the loading vehicle image from the target image according to the similarity level includes: When the similarity is less than a preset threshold, extracting the loading vehicle image from the target image; When the degree of similarity is greater than or equal to the preset threshold, the next frame of image is acquired for processing.

4. The method for counting loading of an excavator based on machine vision as claimed in claim 3, characterized in that: The detection result also includes the status of the loading vehicle; When the similarity is less than a preset threshold, extracting the loading vehicle image from the target image includes: When the similarity is less than the preset threshold, obtaining the state of the loading vehicle; Determining whether the loading vehicle is fully loaded; If the state of the loading vehicle is full, extracting the loading vehicle image from the target image; If the state of the loading vehicle is not full, the next frame of image is acquired for processing.

5. The method for counting loading of an excavator based on machine vision as claimed in claim 1, characterized in that: The counting of the number of vehicles loaded by the excavator according to the result of the identification of the loading vehicle number comprises: Determining whether the loading vehicle number recognition result includes the first loading vehicle number; If the loading vehicle number recognition result includes the first loading vehicle number, obtaining the second loading vehicle number of the loading vehicle recorded last time; comparing the first loading vehicle number with the second loading vehicle number; When the first loading vehicle number and the second loading vehicle number are the same, acquiring the next frame of image for processing; When the first loading vehicle number and the second loading vehicle number are different, the number of loading vehicles loaded by the excavator is counted, and the second loading vehicle number of the loading vehicle recorded last time is updated as the current first loading vehicle number; If the loading vehicle number recognition result does not include the first loading vehicle number, extracting features from the loading vehicle image using a target algorithm to obtain first loading vehicle features; The number of vehicles loaded by the excavator is counted according to the first loader characteristic.

6. The method for counting loading of an excavator based on machine vision as claimed in claim 5, characterized in that: The extracting features of the loading vehicle image by using a target algorithm to obtain a first loading vehicle feature includes: Performing FAST corner point detection on the loading vehicle image to locate key corner points in the loading vehicle image; The ORB descriptor of the key corner point is extracted, and the ORB descriptor is used as the first loading vehicle feature.

7. The method for counting loading of an excavator based on machine vision as claimed in claim 5, characterized in that: The counting of the number of vehicles loaded by the excavator according to the first loader feature includes: Get the second loading vehicle feature of the last recorded loading vehicle; comparing the first loader characteristic with the second loader characteristic; When the first loading vehicle feature and the second loading vehicle feature are the same, acquiring the next frame of image for processing; When the first loader feature is different from the second loader feature, the number of vehicles loaded by the excavator is counted, and the second loader feature of the loader recorded last time is updated as the current first loader feature.

8. An excavator loading counting device based on machine vision, characterized in that: include: An acquisition module, used to acquire a current frame image acquired at a preset frame rate; A first input module, used to input the current frame image into a pre-trained target detection model to obtain a detection result output by the pre-trained target detection model; an extraction module, for extracting a loading vehicle image from the target image when the detection result includes the target image; A second input module is used to input the loading vehicle image into a pre-trained optical character recognition model to obtain a loading vehicle number recognition result output by the pre-trained optical character recognition model; A statistics module is used to count the number of vehicles loaded by the excavator according to the identification result of the loading vehicle number.

9. An intelligent terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the excavator loading counting method based on machine vision as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for counting loading equipment for an excavator based on machine vision as described in any one of claims 1 to 7 is implemented.

Images