Counterweight detection method, device and equipment for operation equipment and storage medium

Abstract

The invention discloses a counterweight detection method and device for operation equipment, the operation equipment and a storage medium, and relates to the technical field of mechanical engineering. The method comprises the following steps: acquiring a region-of-interest image of a counterweight region; inputting the image into a balancing weight detection model to obtain the number of balancing weights in the image and the position information of each balancing weight in the image; cutting the image into a plurality of sub-images according to the position information; inputting the plurality of sub-images into a counterweight specification detection model to obtain an identification result and a confidence coefficient of a counterweight block specification identifier of each sub-image; determining the category number of the recognition results of which the confidence degree meets a confidence degree threshold value; determining uniform specification identification information according to the number of the identification results; and the total balance weight of the operation equipment is determined according to the number of the balancing weights and the specification identification information of the balancing weights. According to the method, any number of balancing weights can be identified without adding a calibration object, total balance weight detection errors caused by electromagnetic interference can be avoided, and the operation safety of operation equipment is improved.

Description

Technical Field

[0001] The present application relates to the technical field of mechanical engineering, and particularly relates to a counterweight detection method, a processor, a device, a working device and a storage medium for a working device. Background Art

[0002] During the engineering operation process, correctly setting the counterweight is crucial for the safety, performance and reliability of the working device, which directly affects the operation efficiency and working quality of the device. At present, the main method for counting counterweight technology is an automatic counterweight identification method based on radio frequency technology. In this method, radio frequency tags with ID numbers burned are pasted at designated positions on each counterweight module in advance. When the device needs counterweight, the radio frequency host automatically identifies the ID number of the counterweight module, calculates the counterweight weight and transmits it to the vehicle-mounted host through the CAN bus. The radio frequency host receives the CAN information returned by the vehicle-mounted host and verifies whether it meets the counterweight requirements. If the counterweight is overloaded, an alarm message is sent through the CAN. Due to the stability and durability of the radio frequency tags being affected by environmental humidity, temperature and electromagnetic field interference, there will be problems with signal transmission and reception between the radio frequency tags and the radio frequency host in the construction environment, resulting in incorrect calculation of the counterweight weight. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a counterweight detection method, a processor, a device, a working device and a storage medium for a working device, so as to solve the technical problem that there are problems with signal transmission and reception between the radio frequency tag and the radio frequency host due to interference in the construction environment in the prior art, resulting in incorrect calculation of the counterweight weight.

[0004] To achieve the above purpose, the first aspect of the present application provides a counterweight detection method for a working device, including:

[0005] Obtain an image of the region of interest, where the region of interest includes the counterweight area of the working device;

[0006] Input the image into the counterweight block detection model to obtain the number of counterweight blocks in the image and the position information of each counterweight block in the image;

[0007] Cut the image into multiple sub-images according to the position information;

[0008] Input the multiple sub-images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight blocks;

[0009] Determine the total counterweight of the working device according to the number of counterweight blocks and the specification identification information of the counterweight blocks;

[0010] Wherein, inputting the multiple sub-images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight blocks includes:

[0011] Input multiple sub-images into the counterweight specification detection model to obtain the recognition results and confidence levels of the counterweight block specifications for each sub-image;

[0012] Determine the number of categories of the recognition results whose confidence levels meet the confidence level threshold;

[0013] Determine the unified specification identification information based on the number of recognition results.

[0014] In the embodiments of the present application, determining the unified specification identification information based on the number of recognition results includes:

[0015] When the number of categories of the recognition results is equal to 1, determine the recognition result as the unified specification identification information; when the number of categories of the recognition results is equal to 2, determine the recognition result with the higher confidence level among the recognition results whose confidence levels meet the confidence level threshold as the unified specification identification information; when the number of categories of the recognition results is greater than 2, determine the recognition result with the largest number of identical recognition results among the recognition results whose confidence levels meet the confidence level threshold as the unified specification identification information.

[0016] In the embodiments of the present application, input the image into the counterweight block detection model to obtain the number of counterweight blocks in the image and the position information of each counterweight block in the image, including: according to the image, through the counterweight block detection model, determine the pre-recognized counterweight block positions in the image and the category confidence levels of the pre-recognized counterweight blocks; count the number of pre-recognized counterweight blocks whose category confidence levels exceed the category confidence level threshold to determine the number of counterweight blocks in the image; determine the positions of the pre-recognized counterweight blocks whose category confidence levels exceed the category confidence level threshold in the image as the position information of each counterweight block in the image.

[0017] In the embodiments of the present application, after obtaining the image, the method further includes: obtaining the first clustering center coordinates and the second clustering center coordinates in the image, where the first clustering center coordinates and the second clustering center coordinates are used to represent the center positions of the counterweight blocks on both sides of the operation device preliminarily determined in the image; determining the clustering results of each counterweight block according to the position information of each counterweight block and the first clustering center coordinates and the second clustering center coordinates; determining the number of counterweight blocks on both sides of the operation device according to the clustering results of each counterweight block; determining the counterweights on both sides of the operation device according to the unified specification identification information and the number of counterweight blocks on both sides.

[0018] In the embodiments of the present application, the counterweight detection method for the operation device further includes: processing each sub-image based on the sub-pixel neural network algorithm to obtain multiple resolution-enhanced images; inputting the multiple sub-images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight blocks, including: inputting the multiple resolution-enhanced images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight blocks.

[0019] In the embodiments of the present application, the calibration of the training images of the counterweight specification detection model is the multi-character overall identification of the counterweight specification.

[0020] In the embodiments of the present application, the counterweight specification detection model is trained through the following steps: obtaining an original counterweight block image sample set; cropping each original counterweight block image in the original counterweight block image sample set to obtain a plurality of cropped images including counterweight specification characters; respectively calibrating the counterweight specifications of each cropped image according to the multi-character overall identifications of multiple preset counterweight specifications to obtain a plurality of calibrated images; and training a pre-trained character detection model based on the plurality of calibrated images to obtain the counterweight specification detection model.

[0021] In the embodiments of the present application, the counterweight block detection model is trained through the following steps: obtaining an original counterweight block image sample set; calibrating the counterweight block borders of each original counterweight block image in the original counterweight block image sample set to obtain a calibrated counterweight block image sample set; performing image data enhancement processing on the calibrated counterweight block images in the calibrated counterweight block image sample set to obtain a counterweight block enhanced image sample set; and training a pre-trained image detection model based on the counterweight block enhanced image sample set to obtain the counterweight block detection model.

[0022] In the embodiments of the present application, the image data enhancement processing includes image flipping, rotation, occlusion, and contrast adjustment.

[0023] The second aspect of the present application provides a processor, which is configured to implement the counterweight detection method for a working device according to the first aspect of the present application when executing instructions.

[0024] The third aspect of the present application provides a counterweight detection device for a working device, including: a camera configured to capture an image of an area of interest, where the area of interest includes the counterweight area of the working device; and a processor according to the second aspect of the present application, connected to the camera and used to receive the image.

[0025] The fourth aspect of the present application provides a working device, including a lifting device; a counterweight device for adjusting the number of counterweights to balance the lifting device; and a counterweight detection device for a working device according to the third aspect of the present application.

[0026] The fifth aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to cause a machine to execute the counterweight detection method for a working device according to the first aspect of the present application.

[0027] Through the above technical solution, by adopting the processing method of cascading multiple image recognition and detection models, the recognition of the number of counterweights and the recognition of specifications are divided into two parts and implemented by two models. Thus, it is possible to simultaneously detect counterweights and counterweight specification characters with a large scale difference, and further realize the detection of the total counterweight of the working equipment. Thereby, any number of counterweights can be recognized without adding and configuring additional calibration objects, and the error in the detection of the total counterweight caused by environmental electromagnetic interference can be avoided. Therefore, the safety of the operation of the working equipment can be improved.

[0028] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation part. Brief Description of the Drawings

[0029] The drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the following specific implementation manners, they are used to explain the embodiments of the present application, but do not constitute a limitation to the embodiments of the present application. In the drawings:

[0030] Figure 1 Schematically shows a flowchart of a method for detecting the counterweight of a working equipment according to an embodiment of the present application;

[0031] Figure 2 Schematically shows a flowchart of a method for detecting the counterweight balance of a working equipment according to an embodiment of the present application;

[0032] Figure 3 Schematically shows a structural diagram of a device for detecting the counterweight of a working equipment according to an embodiment of the present application. Detailed Description of the Embodiments

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be understood that the specific implementation manners described herein are only used to explain and illustrate the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0034] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present application, the directional indications are only used to explain the relative position relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly.

[0035] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, such descriptions of "first", "second", etc. are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0036] The existing automatic counterweight identification method based on radio frequency technology realizes the identification of radio frequency tags by the radio frequency host through radio communication between the radio frequency host and the radio frequency tags, so as to determine the counterweight specifications corresponding to the radio frequency tags. Therefore, the stability of the radio frequency host's identification of counterweight specifications and the service life of the radio frequency tags are easily affected by electromagnetic interference in the environment. Moreover, if new counterweights are added to the equipment, new radio frequency tags need to be added and installed, so its marginal cost increases linearly. Therefore, the embodiments of the present application provide a counterweight detection method for working equipment based on image recognition technology, so as to realize the counterweight detection of working equipment by adopting a technical direction different from that of radio frequency technology. This method can realize the real-time detection of the quantity and specifications of counterweight blocks of various shapes and sizes by installing a camera at a suitable position on the vehicle body and by means of an image processing algorithm, and no additional calibration objects are required. For the region of interest to be identified, by means of the specification identification on the counterweight block, the characters of the specification identification are recognized to obtain the counterweight specifications. Combining the counterweight block detection model and the counterweight specification detection model to obtain the quantity and weight of the counterweight blocks, so as to determine the total counterweight of the working equipment's counterweight. This solution can collect images of the counterweight area of the working equipment through the rear camera in the monitoring system of the working equipment. When this method is actually executed, it can be executed by integrating it into the processor built in the working equipment, or by an industrial computer connected by a network.

[0037] As Figure 1 shown, the embodiments of the present application provide a counterweight detection method for working equipment. Taking the execution of this method in the processor built in the working equipment as an example, this method may include the following steps:

[0038] S102. Obtain an image of the region of interest, where the region of interest includes the counterweight area of the working equipment;

[0039] S104. Input the image into the counterweight block detection model to obtain the quantity of counterweight blocks in the image and the position information of each counterweight block in the image;

[0040] S106. Cut the image into multiple sub-images according to the position information;

[0041] Input multiple sub-images into a counterweight specification detection model to obtain unified specification identification information of the counterweight blocks;

[0042] S114. Determine the total counterweight of the working equipment according to the number of counterweight blocks and the specification identification information of the counterweight blocks;

[0043] Among them, inputting multiple sub-images into a counterweight specification detection model to obtain unified specification identification information of the counterweight blocks includes:

[0044] S108. Input multiple sub-images into a counterweight specification detection model to obtain the recognition results and confidence levels of the counterweight block specification identifications of each sub-image;

[0045] S110. Determine the number of categories of the recognition results whose confidence levels meet the confidence level threshold;

[0046] S112. Determine the unified specification identification information according to the number of recognition results.

[0047] The image obtained by the counterweight detection method for a working equipment provided by an embodiment of the present application is a region of interest image including the counterweight area of the working equipment. Then, the number of counterweight blocks in the image and the position information of each counterweight block in the image are detected and determined through a counterweight block detection model. Thus, the image is segmented according to the position information to obtain multiple sub-images. Thereby, the number of counterweight blocks in the counterweight area of the working equipment is determined simultaneously, and the input for the subsequent counterweight specification detection model is obtained. Since the counterweight blocks of the working equipment adopt a unified specification, the counterweight specification detection model can determine the unified specification of each counterweight block in the counterweight area of the working equipment based on multiple sub-images. Thus, the total counterweight of the working equipment is determined according to the unified specification and the number of counterweight blocks. Therefore, the counterweight detection method for a working equipment provided by an embodiment of the present application adopts a processing method of cascading multiple image recognition detection models, divides the recognition of the number of counterweight blocks and the specification recognition into two parts, and is implemented through two models. Thereby, the technical effect of simultaneously detecting counterweight blocks and counterweight specification characters with a large scale difference can be achieved, and further, the detection of the total counterweight of the working equipment can be realized. Based on this, the counterweight detection method for a working equipment provided by an embodiment of the present application can identify any number of counterweight blocks without adding and configuring additional calibration objects, and can avoid errors in total counterweight detection caused by environmental electromagnetic interference. Therefore, the safety of the operation of the working equipment can be improved.

[0048] Understandably, the working equipment may include various types of cranes, or other excavators, loaders, tower cranes, etc. with counterweights. The image of the region of interest in step S102 can be captured by the on-vehicle camera of the working equipment. The on-vehicle camera can be selected as a short-focus camera to capture the counterweight region 10 meters away from the rear of the vehicle. The counterweight detection model in step S104 and the counterweight specification detection model in step S108 can be, for example, the yolov8 model, or other versions of the yolo model, neural network model. The counterweight specification identification in step S108 can be, for example, the specification characters marked on the counterweight, such as 10t (t is the unit of ton), 5t or 15t. The recognition result of the counterweight specification identification is the corresponding specification character. The recognition results of the counterweight specification identifications determined by the counterweight specification detection model each have their corresponding confidence levels. The confidence level can be, for example, any value within the closed interval from 0 to 1. The confidence level threshold in step S110 can be, for example, a pre-selected confidence level such as 0.6, 0.7 or 0.8.

[0049] Specifically, step S108 may include:

[0050] Inputting multiple sub-images into the counterweight specification detection model respectively to obtain the recognition results and confidence levels of the counterweight specification identifications of each sub-image respectively;

[0051] In the case where the counterweight specification identifications of the sub-images recognized by the counterweight specification detection model are multiple, determine the counterweight specification identification with the highest confidence level as the recognition result of the counterweight specification identification of the sub-image.

[0052] Specifically, step S110 may include:

[0053] In the case where the confidence level of the recognition result of the current sub-image is greater than or equal to the confidence level threshold, and the recognition result of the current sub-image is different from the recognition results of the sub-images that have been detected and whose confidence levels are greater than or equal to the confidence level threshold, increment the number of categories of the recognition result by one.

[0054] Thus, the number of categories of different recognition results whose confidence levels meet the confidence level threshold can be counted, and based on this number of categories, the unified specification identification information of the counterweight can be determined.

[0055] In some embodiments of the present application, to determine the unified specification identification information of the counterweight, step S112 may include:

[0056] In the case where the number of categories of the recognition result is equal to 1, determine the recognition result as the unified specification identification information;

[0057] In the case where the number of categories of the recognition result is equal to 2, determine the recognition result with the greater confidence level among the recognition results whose confidence levels meet the confidence level threshold as the unified specification identification information;

[0058] When the number of categories of the recognition results is greater than 2, the recognition result with the largest number of identical recognition results among the recognition results whose confidence levels meet the confidence level threshold is determined as the unified specification identification information.

[0059] Based on the above steps, when the number of categories of the recognition results is equal to 1, it is equivalent to that the counterweight specification identifiers corresponding to multiple sub-images recognized by the counterweight specification detection model are the same, so that the unified specification identification information of multiple counterweight blocks in the counterweight area of the working equipment can be directly determined. When the number of categories of the recognition results is equal to 2, it is equivalent to that the counterweight specification identifiers corresponding to multiple sub-images recognized by the counterweight specification detection model are divided into two types. Therefore, to determine the unified specification identification information, the recognition result with a larger confidence level is determined as the unified specification identification information. When the number of categories of the recognition results is greater than 2, it is equivalent to that the counterweight specification identifiers corresponding to multiple sub-images recognized by the counterweight specification detection model are divided into more than three types. In this case, since the number of identical recognition results corresponding to each type of recognition result is different, that is, the number of sub-images corresponding to each type of recognition result is different, the recognition result with the largest number of identical recognition results can be selected as the unified specification identification information. Thus, in the case where the recognition results are relatively scattered, the recognition result with the largest number obtained from multiple independent recognitions is used as the unified specification identification information.

[0060] As Figure 2 shown, in some embodiments of the present application, to implement counterweight balance detection, after obtaining the image of the region of interest in step S102 and obtaining the position information of each counterweight block in the image in step S104, the counterweight detection method for the working equipment may further include:

[0061] S202. Obtain the first cluster center coordinates and the second cluster center coordinates in the image, where the first cluster center coordinates and the second cluster center coordinates are used to represent the center of the counterweight blocks on both sides of the working equipment preliminarily determined in the image;

[0062] S204. Determine the clustering results of each counterweight block according to the position information of each counterweight block, and the first cluster center coordinates and the second cluster center coordinates;

[0063] S206. Determine the number of counterweight blocks on both sides of the working equipment according to the clustering results of each counterweight block;

[0064] S208. Determine the counterweights on both sides of the working equipment according to the unified specification identification information and the number of counterweight blocks on both sides.

[0065] In the counterweight area of the working device, multiple counterweight blocks will be distributed and placed in the left and right counterweight sub-areas, such as the position distribution of the superlift counterweights of medium and large tonnage cranes and tower cranes. Therefore, based on the above steps, the centers of the left and right counterweight sub-areas can be first determined through the obtained first clustering center coordinates and second clustering center coordinates, and then according to the position information of each counterweight block and the distance between the two center coordinates, the counterweight sub-area where each counterweight block is located can be determined, so as to determine the clustering result of each counterweight block. Thus, the number of counterweight blocks in the left and right counterweight sub-areas can be known, and combined with the unified specification identification information obtained above, the counterweight of the left and right counterweight sub-areas can be obtained, so as to determine the counterweight balance state of the working device and realize counterweight balance detection.

[0066] In some embodiments of the present application, since the first clustering center coordinates and the second clustering center coordinates may not accurately reflect the centers of the counterweight blocks on both sides of the working device when initially obtained. Therefore, determining the clustering result of each counterweight block according to the position information of each counterweight block, as well as the first clustering center coordinates and the second clustering center coordinates, may include adaptive updating of the clustering center:

[0067] According to the position information of the counterweight block, the first clustering center coordinates and the second clustering coordinates, determine the first distance between the counterweight block and the first clustering center coordinates and the second distance between the counterweight block and the second clustering coordinates;

[0068] Select the minimum value among the first distance and the second distance, and use the clustering coordinates corresponding to the minimum value as the clustering result of the counterweight block;

[0069] Update the clustering center coordinates corresponding to the clustering result of the counterweight block according to the position information of the counterweight block.

[0070] Specifically, updating the clustering center coordinates corresponding to the clustering result of the counterweight block according to the position information of the counterweight block may, for example, take the average value of the position information and the clustering center coordinates, or the clustering center coordinates move towards the position information with a fixed step size.

[0071] Through the above adaptive updating steps, the first clustering center coordinates and the second clustering center coordinates will be adjusted due to the position information of multiple counterweight blocks and gradually approach the centers of the counterweight blocks on both sides of the working device.

[0072] In some embodiments of the present application, to accurately identify the counterweight specification identification in step S108, after obtaining multiple sub-images in step S106, the counterweight detection method for the working device provided by the embodiments of the present application may further include:

[0073] Process each sub-image based on the sub-pixel neural network algorithm to obtain multiple resolution-enhanced images;

[0074] Input multiple sub-images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight block, including:

[0075] Input multiple resolution-enhanced images into the counterweight specification detection model to obtain the unified specification identification information of the counterweight block.

[0076] Correspondingly, step S108 includes:

[0077] Input multiple resolution-enhanced images into the counterweight specification detection model to obtain the recognition results and confidence levels of the counterweight block specifications of each resolution-enhanced image.

[0078] After processing the sub-images through the sub-pixel neural network algorithm, the obtained resolution-enhanced images have better image quality compared to their sub-images. The sub-pixel neural network algorithm can learn and obtain the weight parameters of the upsampling filter, so that the obtained upsampling filter can smooth the interpolation process during upsampling, remove high-frequency artifacts, and thus maintain the image quality to achieve the super-resolution reconstruction of the sub-images, thereby obtaining resolution-enhanced images.

[0079] It can be understood that the sub-pixel neural network algorithm can be implemented through the ESPCN (Efficient Sub-Pixel Convolutional Network) algorithm. The ESPCN algorithm can directly perform feature extraction on the input low-resolution sub-images, and then use the sub-pixel convolutional layer for the extracted deep feature maps, thereby realizing the processing of the sub-images by the sub-pixel neural network algorithm.

[0080] In some embodiments of the present application, the calibration of the training images of the counterweight specification detection model is the multi-character overall identification of the counterweight specification.

[0081] Since the counterweight block specifications are fixed, the common counterweight block weight characters can be regarded as a whole for calibration (that is, when calibrating "10t", it is directly marked as "10t" instead of marking each character separately as "1", "0", "t"). This method can integrate the information of all characters, improve the discriminability of the character as a whole, and improve the recognition accuracy.

[0082] In some embodiments of the present application, the counterweight specification detection model is trained through the following steps:

[0083] Obtain the original counterweight block image sample set;

[0084] Crop each original counterweight block image in the original counterweight block image sample set to obtain multiple cropped images including counterweight specification characters;

[0085] Calibrate the weight specifications of each cropped image according to the multi-character overall identification of multiple preset weight specifications to obtain multiple calibrated images;

[0086] Train a pre-trained character detection model based on multiple calibrated images to obtain a weight specification detection model.

[0087] Based on the above steps, the original weight block image can be cropped on the basis of the original weight block image sample set to obtain a cropped image with weight specification characters, so that the pre-trained character detection model can be trained based on the cropped image.

[0088] It can be understood that each original weight block image in the original weight block image sample set can be obtained by a camera on the working device photographing the weight block in the weight area of the working device.

[0089] In some embodiments of the present application, cropping each original weight block image in the original weight block image sample set to obtain multiple cropped images including weight specification characters may include:

[0090] Crop the original weight block image to obtain multiple preliminary cropped images;

[0091] In the case where the preliminary cropped image includes weight specification characters, save the preliminary cropped image as a cropped image including weight specification characters;

[0092] In the case where the preliminary cropped image does not include the characters in the weight specification, add the weight specification characters to the preliminary cropped image;

[0093] Save the preliminary cropped image after adding the weight specification characters as a cropped image including weight specification characters.

[0094] Based on the above steps, the sample set for training the pre-trained character detection model can be enriched, thereby improving the accuracy of the weight specification detection model in identifying the weight specification.

[0095] In some embodiments of the present application, the weight block detection model is trained through the following steps:

[0096] Obtain the original weight block image sample set;

[0097] Calibrate the weight block borders of each original weight block image in the original weight block image sample set to obtain a calibrated weight block image sample set;

[0098] Perform image data enhancement processing on the calibrated weight block images in the calibrated weight block image sample set to obtain a weight block enhanced image sample set;

[0099] Enhance the pre-trained image detection model based on counterweight blocks to obtain a counterweight block detection model. Image data enhancement processing can enrich the number of samples based on the calibrated counterweight block images, obtaining a counterweight block enhanced image sample set with more samples, thereby improving the training effect of the pre-trained image detection model.

[0100] In some embodiments of the present application, the pre-trained image detection model can be, for example, a pre-trained image recognition model. The above training step fine-tunes the parameters of the image recognition model by training the pre-trained image detection model with the counterweight block enhanced image sample set, so that the image recognition model can accurately recognize the counterweight blocks.

[0101] In some embodiments of the present application, the image data enhancement processing can include image flipping, rotation, occlusion, and contrast adjustment.

[0102] Based on the above image data enhancement processing means, the image samples in the counterweight block enhanced image sample set can be made more complex, enabling the image recognition model to still recognize the contour and position of the counterweight blocks under conditions such as low brightness, incomplete counterweight block images, and varying angles of the counterweight blocks relative to the camera.

[0103] The overall process of the counterweight detection method for the working equipment provided by the embodiments of the present application will be exemplarily described below.

[0104] A wide-angle camera can be configured at the rear of the working equipment. This wide-angle camera is used to capture images of the counterweight area of the working equipment and send the images to a switch via the network card of the working equipment through a local area network. The switch sends the images to an industrial computer through the local area network. The industrial computer executes the counterweight detection method for the working equipment and then outputs the detection result of the counterweight through a display.

[0105] Before using the counterweight detection method for the working equipment to perform counterweight detection, it is first necessary to train the pre-trained character detection model and the pre-trained image detection model. After training, the obtained counterweight specification detection model and counterweight block detection model are optimized through TensorRT to enable them to perform optimally on the GPU of the industrial computer.

[0106] Record the optimized counterweight specification detection model and counterweight block detection model in the industrial computer. Take images of the counterweight area through the camera on the working equipment to obtain images of the region of interest, thereby executing steps S102 to S114 to obtain the total counterweight detection result of the working equipment; and determine the counterweights on both sides of the working equipment based on the clustering result of the counterweight blocks, thereby realizing counterweight balance detection. The detection results of the total counterweight and counterweight balance can be displayed through a display.

[0107] In summary, the counterweight detection method for a work device provided by the embodiments of the present application is applicable to counterweights of various shapes and sizes, and is not limited to a specific installation environment or counterweight shape. As long as the counterweight can be captured by the camera, counterweight detection can be achieved, so the applicable range is wider. This method does not require an additional calibration object, and can identify and count the counterweights in real time, and can record almost simultaneously when the counterweights are added or removed. This method realizes counterweight detection based on image recognition, can directly convert the image of the counterweight into digital information, which is convenient for subsequent data processing and analysis. Compared with radio frequency technology, it does not require an additional database and data processing system, reducing the complexity of the system and management costs.

[0108] The embodiments of the present application also provide a processor, which is configured to implement the counterweight detection method for a work device according to any one of the above embodiments when executing instructions.

[0109] The embodiments of the present application also provide a counterweight detection device for a work device, including: a camera and the above-mentioned processor. The camera is configured to capture an image of the region of interest, and the region of interest includes the counterweight area of the work device. The processor is connected to the camera and is used to receive the image to execute the counterweight detection method for a work device according to any one of the above embodiments.

[0110] As Figure 3 shown, in some embodiments of the present application, the counterweight detection device for a work device may further include: a switch 320 and a display 340. The camera 310 is connected to the processor 330 through the switch 320, and the processor 330 is connected to the display.

[0111] It can be understood that the camera 310 and the switch 320 can be connected through a network, the switch 320 and the processor 330 can be connected through a network, the processor 330 can be, for example, the built-in graphics processor of the working condition machine, and the display 340 is used to display the counterweight detection result obtained by the processor 330.

[0112] The embodiments of the present application also provide a work device, including a lifting device, a counterweight device, and the above-mentioned counterweight detection device for a work device. The counterweight device is used to adjust the counterweight quantity to balance the lifting device. The counterweight detection device for a work device acquires an image of the counterweight area of the counterweight device.

[0113] It can be understood that the device can be, for example, a hydraulic boom, and the counterweight device can be, for example, a hydraulically controlled counterweight system.

[0114] The embodiments of the present application also provide a machine-readable storage medium, on which instructions are stored, and the instructions are used to cause a machine to execute the counterweight detection method for a work device according to any one of the above embodiments.

[0115] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0117] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0119] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0120] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A counterweight detection method for operating equipment, characterized in that: include: Acquire an image of a region of interest, wherein the region of interest includes a counterweight area of ​​the working equipment; Inputting the image into a counterweight detection model to obtain the number of counterweights in the image and the position information of each counterweight in the image; Cutting the image into a plurality of sub-images according to the position information; Inputting the plurality of sub-images into a counterweight specification detection model to obtain uniform specification identification information of the counterweight block; Determining the total counterweight of the operating equipment according to the number of the counterweight blocks and the specification identification information of the counterweight blocks; The multiple sub-images are input into the counterweight specification detection model to obtain the uniform specification identification information of the counterweight block, including: Inputting the plurality of sub-images into a counterweight specification detection model to obtain a recognition result and a confidence level of a counterweight specification identifier of each sub-image; Determine the number of categories of recognition results whose confidence meets the confidence threshold; The unified specification identification information is determined according to the number of the recognition results.

2. The method according to claim 1, characterized in that The determining the uniform specification identification information according to the number of the recognition results includes: When the number of categories of the recognition result is equal to 1, determining the recognition result as the unified specification identification information; When the number of categories of the recognition results is equal to 2, the recognition result with a greater confidence among the recognition results whose confidences meet the confidence threshold is determined as the unified specification identification information; When the number of categories of the recognition results is greater than 2, the recognition result having the largest number of identical recognition results among the recognition results whose confidences satisfy the confidence threshold is determined as the unified specification identification information.

3. The method according to claim 1, characterized in that Inputting the image into a counterweight detection model to obtain the number of counterweights in the image and position information of each counterweight in the image includes: According to the image, determining the pre-identified counterweight position in the image and the category confidence of the pre-identified counterweight by using the counterweight detection model; Counting the number of the pre-identified counterweight blocks whose category confidence exceeds the category confidence threshold, so as to determine the number of the counterweight blocks in the image; The position of each of the pre-identified counterweight blocks whose category confidence exceeds the category confidence threshold in the image is determined as the position information of each of the counterweight blocks in the image.

4. The method according to claim 1, characterized in that: After acquiring the image, the method further includes: Acquire first cluster center coordinates and second cluster center coordinates in the image, wherein the first cluster center coordinates and the second cluster center coordinates are used to express the centers of the counterweight blocks on both sides of the working equipment preliminarily determined in the image; Determining a clustering result of each of the counterweight blocks according to the position information of each of the counterweight blocks, as well as the first cluster center coordinates and the second cluster center coordinates; Determining the number of counterweights on both sides of the operating equipment according to the clustering results of each counterweight; The counterweights on both sides of the operating equipment are determined according to the uniform specification identification information and the number of counterweight blocks on both sides.

5. The method according to claim 1, characterized in that Also includes: Processing each of the sub-images based on a sub-pixel neural network algorithm to obtain a plurality of resolution-enhanced images; The step of inputting the plurality of sub-images into a counterweight specification detection model to obtain uniform specification identification information of the counterweight block includes: The multiple resolution enhanced images are input into a counterweight specification detection model to obtain uniform specification identification information of the counterweight block.

6. The method according to claim 1, characterized in that The training images of the weight specification detection model are calibrated as a multi-character overall identification of the weight specification.

7. The method according to claim 6, characterized in that The weight specification detection model is trained through the following steps: Obtaining an original counterweight image sample set; Cropping each original counterweight image in the original counterweight image sample set to obtain a plurality of cropped images including counterweight specification characters; Calibrate the weight specification of each of the cropped images according to multiple multi-character overall marks of preset weight specifications to obtain multiple calibrated images; A pre-trained character detection model is trained based on a plurality of the calibrated images to obtain the weight specification detection model.

8. The method according to claim 1, characterized in that The counterweight detection model is trained by the following steps: Obtaining an original counterweight image sample set; Calibrate the counterweight frame of each original counterweight image in the original counterweight image sample set to obtain a calibrated counterweight image sample set; Performing image data enhancement processing on the calibrated counterweight images in the calibrated counterweight image sample set to obtain a counterweight enhanced image sample set; A pre-trained image detection model is trained based on the counterweight enhanced image sample set to obtain the counterweight detection model.

9. The method according to claim 8, characterized in that The image data enhancement processing includes image flipping, rotation, occlusion, and contrast adjustment.

10. A processor, characterized in that: The device is configured to implement the counterweight detection method for working equipment according to any one of claims 1 to 9 when executing the instructions.

11. A counterweight detection device for operating equipment, characterized in that: include: a camera configured to capture an image of an area of ​​interest, the area of ​​interest including a counterweight area of ​​the working equipment; as well as The processor of claim 10, connected to the camera and configured to receive the image.

12. An operating device, characterized in that: include, A counterweight device, used to adjust the number of counterweights for balancing the operating equipment; A counterweight detection device for working equipment as claimed in claim 11.

13. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing the machine to execute the counterweight detection method for working equipment according to any one of claims 1 to 9.

Images