Grab bucket control method and system, electronic equipment and computer program product

Abstract

The invention discloses a grab bucket control method and system, electronic equipment and a computer program product, and relates to the field of image processing, and the grab bucket control method comprises the steps that after a grab bucket grabs materials, an image of the grab bucket is shot, and the image comprises the grab bucket and the materials grabbed by the grab bucket; the image is input into a preset deep learning model, so that the preset deep learning model identifies an image area of the material grabbed by the grab bucket and predicts the weight of the current material grabbed by the grab bucket according to the identified image area; obtaining the weight of the current material grabbed by the grab bucket and output by the preset deep learning model; and according to the comparison result of the current material weight and the weight critical value, the operation mode of the grab bucket is adjusted, and the weight critical value is the rated lifting weight of the grab bucket. The preset deep learning model is used for identifying the shot image, whether the weight of the current material grabbed by the grab bucket has an overload risk or not is judged in advance before the grab bucket is lifted, so that the number of times that the steel wire rope operates when exceeding the designed safe load is reduced, and the service life and safety of the steel wire rope are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a grab control method, system, electronic equipment and computer program product. Background Art

[0002] Grab ship unloader is a mechanical device used to grab and unload materials from ships. The grab on the grab ship unloader grabs the materials from the ship cabin to the bulk material conveyor belt at the dock. Grab overload protection is an important part of ensuring the safe operation of the grab ship unloader. When the grab ship unloader grabs materials, the grab will stop for a short time. After the grab grab grabs the materials, the wire rope is tightened to lift the grab to transport the materials.

[0003] At present, the grab overload protection of grab ship unloader is mainly realized by the tension sensor on the grab wire rope. However, the tension sensor can only measure the weight of the material grabbed by the grab when the grab is lifted to determine whether the grab is overloaded. At this time, the wire rope may have exceeded its designed safety load. If the wire rope exceeds its designed safety load for many times, the service life and safety of the wire rope will be greatly reduced. Summary of the invention

[0004] In view of the above problems, the present application provides a grab control method, system, electronic device and computer program product to achieve the purpose of improving the service life and safety of the wire rope. The specific scheme is as follows:

[0005] A first aspect of the present application provides a grab bucket control method, the grab bucket control method comprising:

[0006] After the grab grabs the material, an image of the grab is photographed, wherein the image includes the grab and the material grabbed by the grab;

[0007] Inputting the image into a preset deep learning model so that the preset deep learning model identifies the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket based on the identified image area;

[0008] Obtaining the current weight of the material grabbed by the grab bucket output by the preset deep learning model;

[0009] The operation mode of the grab bucket is adjusted according to the comparison result between the current material weight and the weight critical value, and the weight critical value is the rated lifting weight of the grab bucket.

[0010] In a possible implementation, the preset deep learning model identifies the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket according to the identified image area, including:

[0011] The preset deep learning model identifies the image area, determines the material category of the material and the area ratio of the image area in the image area inside the grab bucket;

[0012] The material weight corresponding to the area ratio and the material category is obtained, and the queried weight is determined as the current material weight.

[0013] In a possible implementation, adjusting the operation mode of the grab bucket according to the comparison result between the current material weight and the weight threshold value includes:

[0014] If the current material weight is less than the first multiple of the weight threshold value, the grab bucket is lifted at a first speed;

[0015] If the current material weight is not less than the weight of the first multiple of the weight critical value and less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at a second speed;

[0016] If the current material weight is not less than the second multiple of the weight threshold value, the lifting of the grab bucket is stopped, the grabbing force of the grab bucket is reduced to reduce the material grabbed by the grab bucket, and the grab bucket is lifted again after the grabbing force of the grab bucket is reduced;

[0017] The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

[0018] In a possible implementation, the method further includes:

[0019] Before the grab grabs the material, photographing a material pile image of the grabbed material pile;

[0020] Inputting the material pile image into the preset deep learning model, so that the preset deep learning model identifies the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the identified material pile area;

[0021] Obtaining the material category and the current stacking angle output by the preset deep learning model;

[0022] If the current stacking angle is not less than the angle critical value, the grabbing position of the grab bucket is not adjusted. If the current stacking angle is less than the angle critical value, the grabbing position of the grab bucket is adjusted. The angle critical value corresponds to the material category.

[0023] A second aspect of the present application provides a grab bucket control system, the grab bucket control system comprising:

[0024] A first imaging unit, used for capturing an image of the grab bucket after the grab bucket grabs the material, wherein the image includes the grab bucket and the material grabbed by the grab bucket;

[0025] A first recognition unit, configured to input the image into a preset deep learning model, so that the preset deep learning model recognizes an image area of ​​the material grabbed by the grab bucket and predicts a current weight of the material grabbed by the grab bucket based on the recognized image area;

[0026] A first acquisition unit, used to obtain the current weight of the material grabbed by the grab bucket output by the preset deep learning model;

[0027] The first adjustment unit is used to adjust the operation mode of the grab bucket according to the comparison result of the current material weight and the weight critical value, and the weight critical value is the rated lifting weight of the grab bucket.

[0028] In a possible implementation, the preset deep learning model is specifically configured as follows:

[0029] The preset deep learning model identifies the image area, determines the material category of the material and the area ratio of the image area in the image area inside the grab bucket;

[0030] The material weight corresponding to the area ratio and the material category is obtained, and the queried weight is determined as the current material weight.

[0031] In a possible implementation, the first adjustment unit is specifically configured as follows:

[0032] If the current material weight is less than the first multiple of the weight threshold value, the grab bucket is lifted at a first speed;

[0033] If the current material weight is not less than the weight of the first multiple of the weight critical value and less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at a second speed;

[0034] If the current material weight is not less than the second multiple of the weight threshold value, suspending the lifting of the grab bucket, reducing the grabbing force of the grab bucket to reduce the material grabbed by the grab bucket, and then lifting the grab bucket after reducing the grabbing force of the grab bucket;

[0035] The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

[0036] In a possible implementation, the grab control system further includes a second image unit, a second recognition unit, a second acquisition unit, and a second adjustment unit:

[0037] The second imaging unit is used to capture a material pile image of the grabbed material pile before the grab grab grabs the material;

[0038] The second recognition unit is used to input the material pile image into the preset deep learning model, so that the preset deep learning model recognizes the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the recognized material pile area;

[0039] The second acquisition unit is used to acquire the material category and the current stacking angle output by the preset deep learning model;

[0040] The second adjustment unit is used for not adjusting the grabbing position of the grab bucket if the current stacking angle is not less than the angle critical value, and for adjusting the grabbing position of the grab bucket if the current stacking angle is less than the angle critical value, and the angle critical value corresponds to the material category.

[0041] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the grab bucket control method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0044] A fourth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the grab control method of the first aspect or any implementation of the first aspect.

[0045] By means of the above technical solution, the present application provides a grab control method, system, electronic device and computer program product. After the grab grabs the material, the method takes a grab image and inputs it into a preset deep learning model. The deep learning model predicts the current weight of the material grabbed by the grab, and adjusts the operation mode of the grab by comparing the current material weight with the weight threshold. The method uses a preset deep learning model to identify the captured image, and before the grab is lifted, it determines in advance whether there is an overload risk for the current material weight grabbed by the grab, so as to reduce the number of times the wire rope operates beyond its designed safe load, and effectively improve the life and safety of the wire rope. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A schematic flow chart of a grab bucket control method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a material stacking angle provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a grab control system provided in an embodiment of the present application;

[0050] Figure 4 A hardware structure block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0052] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0053] The terms "first", "second", etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0054] Grab bucket ship unloader is a kind of equipment specially used for grabbing and unloading materials in the ship cabin, which is widely used in ports, shipyards and water buildings. Grab bucket ship unloader grabs materials with a grab bucket, and lifts the grab bucket by tightening the wire rope to realize material transportation. Therefore, grab bucket overload protection is an important part to ensure the safe operation of grab bucket ship unloader. The existing common grab bucket overload protection mainly senses the weight of the material when the grab bucket grabs the material through the tension sensor on the wire rope, so as to judge whether the weight of the material grabbed by the grab bucket is overloaded, and realize the overload protection of the grab bucket.

[0055] However, the above grab bucket overload protection method has an obvious disadvantage. When the grab bucket unloader grabs materials in the cabin, the grab bucket will briefly contact the material surface. At this time, the grab bucket is briefly placed on the material surface. The material surface has a certain support force on the grab bucket, and the wire rope has not yet begun to tighten and lift the grab bucket. The wire rope may be loose. Therefore, the weight of the material sensed by the tension sensor on the wire rope is not the weight of the material actually grabbed by the grab bucket. When the grab bucket is lifted by the tightened wire rope, the tension sensor on the wire rope can accurately sense the weight of the material grabbed by the grab bucket, thereby judging whether the grab bucket is overloaded. If the grab bucket is overloaded, the grab bucket overload alarm will be issued. Therefore, the existing common grab bucket overload protection can only issue an overload alarm after the wire rope is tightened, and at this time the wire rope is already tightened, and the tension of the wire rope may have exceeded the designed safety load due to overload. If the wire rope is operated for multiple times exceeding its designed safety load, it will accelerate the aging of the wire rope and greatly reduce the service life and safety of the wire rope.

[0056] In order to solve the above problems, the embodiment of the present application provides a grab control method. Before the grab is lifted, the method takes a picture of the material grabbed by the grab, and through the image recognition of the preset deep learning model, it predicts the weight of the material grabbed by the grab in advance before the grab is lifted, effectively reducing the number of times the wire rope operates when it exceeds the designed safety load, and effectively improving the service life and safety of the wire rope. The grab control method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 1 , Figure 1 A flowchart of a grab control method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a grab bucket control method provided in an embodiment of the present application may include steps S10 to S13, and these steps are described in detail below.

[0058] S10. After the grab grabs the material, an image of the grab is photographed, where the image includes the grab and the material grabbed by the grab.

[0059] Among them, the present application can install a shooting device on the beam of the grab ship unloader to obtain images of the grab grab grabbing materials. Of course, the installation position of the shooting device can be unlimited, and it is sufficient to clearly shoot the grab and the materials grabbed in the grab. Specifically, the shooting device can be a high-definition camera. The shooting device in this embodiment can continuously track each working state of the grab when the grab ship unloader is running. Furthermore, after the grab grab grabs the material, this embodiment can use the image directly taken by the shooting device as the image of the grab that needs to be analyzed, or can obtain a video of the grab grabbing the material by continuously recording the process of the grab grab grabbing the material through the shooting device, and can obtain a frame of the grab grab grabbing the material from the video as the image of the grab that needs to be analyzed.

[0060] When the grab bucket of the grab ship unloader is grabbing materials, the high-definition camera can continuously capture the status of the grab bucket and the materials inside the grab bucket, and transmit the captured images to the intelligent control system of the grab ship unloader for subsequent processing.

[0061] S11, inputting the image into a preset deep learning model, so that the preset deep learning model recognizes the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket according to the recognized image area;

[0062] S12. Obtain the current weight of the material grabbed by the grab bucket output by the preset deep learning model.

[0063] The image area of ​​the material may refer to the image area occupied by the material grabbed by the grab bucket in the image. The preset deep learning model is a neural network model trained for the weight prediction of the grabbed material in this embodiment.

[0064] Specifically, in this embodiment, grab images of multiple grabs grabbing materials of different weights can be first obtained and the corresponding weight can be annotated for each grab image, which can be used as a training data set and input into the deep learning model for training. After the training is completed, the grab is used to grab different amounts of materials multiple times, and the tension sensor on the wire rope is used to sense the actual weight of the materials grabbed by the grab, and the grab image of the grab is photographed and input into the trained deep learning model for prediction. The weight prediction result of the deep learning model is compared with the actual sensing result of the tension sensor on the corresponding wire rope to continuously correct the parameters of the deep learning model and improve the prediction accuracy of the deep learning model. When the error between the prediction result of the deep learning model and the sensing result of the actual tension sensor is within a controllable range, the parameter correction of the deep learning model is stopped to obtain the preset deep learning model in this embodiment. In this embodiment, the parameters of the deep learning model are corrected by the actual sensing result of the tension sensor, which can further improve the prediction accuracy of the preset deep learning model and ensure the accuracy of the grab overload protection.

[0065] Furthermore, there may be multiple categories of materials grabbed by the grab bucket, and because the density of materials of different categories is different, different materials of the same volume may have different weights. Therefore, when marking the weight in the grabbed image, this embodiment may also mark the material category of the grabbed material in the grabbed image, so that the trained deep model can not only identify the material category based on the grabbed image, but also further predict the material weight based on the material category, thereby improving the accuracy of weight prediction.

[0066] When the intelligent control system obtains the image taken by the camera, it can be input into the trained preset deep learning model for image recognition to predict the weight of the material grabbed by the grab in the image.

[0067] Specifically, the image recognition process of the preset deep learning model in this embodiment can be shown as steps 1 and 2:

[0068] Step 1: Preset the deep learning model to identify the image area, determine the material category and the area ratio of the image area in the grab bucket;

[0069] Step 2: Obtain the material weight corresponding to the area ratio and material category, and determine the queried weight as the current material weight.

[0070] Among them, the internal image area of ​​the grab bucket may refer to the original image area occupied by the internal space area of ​​the grab bucket when the grab bucket is empty. The preset deep learning model can determine the image area occupied by the material by recognizing the image, and calculate the area ratio of the material in the internal space area of ​​the grab bucket according to the image area occupied by the material and the original image area occupied by the internal space area of ​​the grab bucket. If the grabbed material has multiple gaps, the gaps also belong to the internal space area of ​​the grab bucket, not the image area occupied by the material. The current material weight is: the weight of the material being grabbed in the grab bucket predicted by the preset deep learning model based on the grabbed image.

[0071] Due to the different densities of materials of different categories, even with the same volume, the weight may be different. Therefore, even with the same area ratio, the corresponding material weights of materials of different categories may be different. For example, the density of category A and category B is different. The area ratio of materials of category A is 50%, and the corresponding material weight is 10 kg, while the area ratio of materials of category B is also 50%, but the corresponding material weight is 15 kg. Therefore, the preset deep learning model in this embodiment can first identify the material category to which the material in the image area belongs, and then determine the material weight corresponding to both the area ratio and the material category as the current material weight.

[0072] Furthermore, in order to improve the accuracy of prediction, this embodiment uses a large amount of multi-category training data when training the preset deep learning model, so that each area proportion under each material category can correspond to a material weight. For example, under a material category, 50% of the area proportion can correspond to a material weight, and 51% of the area proportion can correspond to a material weight. Of course, the preset deep learning model can also adopt a range correspondence method. For example, under a material category, 51%-55% of the area proportion can correspond to a material weight, and 56%-60% of the area proportion can correspond to a material weight. Of course, the range correspondence method has a lower accuracy in weight prediction than the single correspondence method.

[0073] S13. Adjust the operation mode of the grab bucket according to the comparison result between the current material weight and the weight critical value, where the weight critical value is the rated lifting weight of the grab bucket.

[0074] Among them, when the intelligent control system obtains the current material weight output by the deep learning model, it can compare the current material weight with the weight critical value of the grab bucket to determine whether the grab bucket is currently overloaded, thereby adjusting the operation mode of the grab bucket. Specifically, in this embodiment, based on the comparison result of the current material weight and the weight critical value, the adjustment logic of the operation mode of the grab bucket can be as follows:

[0075] If the current material weight is less than the first multiple of the weight critical value, the grab bucket is lifted at the first speed;

[0076] If the current material weight is not less than the weight of the first multiple of the weight critical value and is less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at the second speed;

[0077] If the current material weight is not less than the second multiple of the weight critical value, the lifting of the grab bucket is suspended, the grabbing force of the grab bucket is reduced to reduce the material grabbed by the grab bucket, and the grab bucket is lifted again after the grabbing force of the grab bucket is reduced;

[0078] The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

[0079] The first multiple and the second multiple are multiples determined according to the actual application data of the grab bucket, and their values ​​may be different according to different grab buckets. In this embodiment, the first multiple may be 1.1, and the second multiple may be 1.2. The first speed is the normal lifting speed when the grab bucket grabs materials, and the second speed is a lifting speed lower than the normal lifting speed.

[0080] When the current material weight is less than 1.1 times of the weight critical value, since the material weight only exceeds the critical value by a small amount at this time, in order to save time and improve the efficiency of the grab, the grab can be lifted at a normal lifting speed without adjusting the amount of material grabbed.

[0081] When the current material weight is not less than 1.1 times the weight critical value and less than 1.2 times the weight critical value, in order to improve the efficiency of the grab and reduce damage to the wire rope, the grab's lifting speed can be limited and the grab can be lifted at a speed lower than the normal lifting speed (slow speed).

[0082] When the current material weight is not less than 1.2 times the weight threshold, the grab bucket is considered to be overloaded. Even a slow lift can cause significant damage to the wire rope. Then, a grab bucket overload alarm can be issued to remind the control personnel to suspend lifting the grab bucket or automatically suspend lifting the grab bucket. The grab bucket can be controlled to release some materials to reduce the weight of the materials before lifting. After the grab bucket releases some materials, the image of the grab bucket can be taken again to predict the weight of the materials grabbed by the grab bucket at this time, and the grab bucket can be directly lifted or the weight of the materials can be further reduced according to the weight of the materials.

[0083] The embodiment of the present application provides a grab control method. After the grab grabs the material, the grab grab captures the image and inputs it into a preset deep learning model. The deep learning model predicts the current weight of the material grabbed by the grab, and adjusts the operation mode of the grab by comparing the current material weight with the weight threshold. The method uses a preset deep learning model to identify the captured image, and before the grab is lifted, it determines in advance whether the current weight of the material grabbed by the grab has an overload risk, so as to reduce the number of times the wire rope operates beyond its designed safe load, and effectively improve the life and safety of the wire rope.

[0084] In a possible implementation, before the grab grabs the material, the grab control method of this embodiment may further include the following steps:

[0085] Step 3: photographing a material pile image of the grabbed material pile;

[0086] Step 4: inputting the material pile image into a preset deep learning model, so that the preset deep learning model identifies the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the identified material pile area;

[0087] Step 5: Obtain the material category and current stacking angle output by the preset deep learning model;

[0088] Step 6: If the current stacking angle is not less than the angle critical value, the grabbing position of the grab bucket will not be adjusted. If the current stacking angle is less than the angle critical value, the grabbing position of the grab bucket will be adjusted. The angle critical value corresponds to the material category.

[0089] Among them, when the grab bucket is grabbing materials, the material surface may collapse, causing the grab bucket to be buried by the material, resulting in the wire rope being tightened and overloaded. Therefore, before the grab bucket grabs materials, this embodiment can also take an image of the material pile, and the intelligent control system and the preset deep learning model can jointly determine whether the material pile has the risk of collapse, thereby determining the material grabbing position of the grab bucket.

[0090] Since the grab bucket generally starts grabbing materials from the middle of the material pile, the materials around the middle position move from both sides to the lower position, causing the stacking angles at both ends of the material pile to gradually decrease. Figure 2 As shown, the stacking angles of the material pile on the left are a and b. After the grab bucket digs out part of the material, the materials on both sides slide down to both sides and become the material pile on the right. Comparing a and b of the left and right material piles, it can be obviously concluded that a and b of the right material pile are smaller than a and b of the left material pile.

[0091] Of course, when the grab bucket selects any position of the material pile to grab the material, the surrounding materials of the grabbed part of the material pile can still be affected by the grab bucket and move, so that the stacking angle of at least one end of the material pile is affected. The smaller the stacking angle, the more likely the material moves to the sides, and the material at the grabbed position loses the support of some materials, and is more likely to move to the sides or the hollow position of the grabbed position, so it is easy to bury the grab bucket that is grabbing the material. Therefore, this embodiment can identify the stacking angle of the material pile in the image through a preset deep learning model, and the intelligent control system can determine the collapse risk of the current material pile.

[0092] Furthermore, since different categories of materials have different sizes of particles and different fluidity, the accumulation angles when they collapse under the influence of the grab bucket may be different. For example, there are differences in the size of coal and sand particles, and the fluidity of sand is higher than that of coal. When the accumulation angle of the coal pile is reduced to 35 degrees, the fluidity of the coal pile is poor, so it collapses at the grasped position. However, since the fluidity of sand is better, when the accumulation angle of the sand pile is reduced to below 30 degrees, the grasped position of the sand pile collapses. Therefore, in addition to outputting the recognized accumulation angle, the preset deep learning model also needs to send the recognized material category along with the accumulation angle to the intelligent control system, and the intelligent control system determines the angle critical value to be compared with the accumulation angle according to the material category. Furthermore, this embodiment can collect a large number of images of multi-category material piles at different shooting angles, and mark the accumulation angles at each shooting angle as a training data set, thereby training the accumulation angle recognition ability of the deep learning model.

[0093] Specifically, when the current stacking angle is not less than the angle critical value, it means that the grabbed position of the material pile does not have the risk of collapse, and the grab can continue to grab materials at this grab position, so the grab position of the grab does not need to be adjusted; when the current stacking angle is less than the angle critical value, it means that the grabbed position of the material has the risk of collapse, so the grab position needs to be adjusted to reduce the risk of the grab being buried. Among them, the grab can be controlled to use the original grabbed position as the reference position, and re-select a suitable new grab position around the reference position (front, back, left, and right positions are all acceptable) to continue grabbing materials.

[0094] Furthermore, due to the influence of the wind speed in the surrounding environment, the grab bucket posture may be unstable when grabbing materials, the grab bucket's grabbing position is not fixed, and it has a certain impact, which may increase the risk of collapse of the material surface or cause direct collapse. The current grab bucket ship unloader often ignores the protection of the grab bucket posture. Therefore, the intelligent control system in this embodiment can also continuously track and capture the grab bucket posture during the material collection process through the shooting equipment. If the grab bucket posture is found to be unstable, an early warning can be issued in time to adjust the grab bucket posture to prevent the grab bucket from overturning, and then grab the material when the grab bucket is stable.

[0095] A grab bucket control method provided in an embodiment of the present application is introduced above, and a system applying the grab bucket control method will be introduced below.

[0096] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a grab control system provided in an embodiment of the present application. Figure 3 As shown, the grab control system may include:

[0097] The first imaging unit 100 is used to capture an image of the grab bucket after the grab bucket grabs the material, wherein the image includes the grab bucket and the material grabbed by the grab bucket;

[0098] A first recognition unit 110 is used to input the image into a preset deep learning model so that the preset deep learning model recognizes the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket based on the recognized image area;

[0099] A first acquisition unit 120 is used to obtain the current weight of the material grabbed by the grab bucket output by a preset deep learning model;

[0100] The first adjustment unit 130 is used to adjust the operation mode of the grab bucket according to the comparison result between the current material weight and the weight critical value, and the weight critical value is the rated lifting weight of the grab bucket.

[0101] In a possible implementation, the preset deep learning model can be specifically configured as:

[0102] The preset deep learning model identifies the image area, determines the material category and the area ratio of the image area in the grab bucket;

[0103] Get the material weight corresponding to the area ratio and material category, and determine the queried weight as the current material weight.

[0104] In a possible implementation, the first adjustment unit may be specifically configured as follows:

[0105] If the current material weight is less than the first multiple of the weight critical value, the grab bucket is lifted at the first speed;

[0106] If the current material weight is not less than the weight of the first multiple of the weight critical value and is less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at the second speed;

[0107] If the current material weight is not less than the second multiple of the weight critical value, the lifting of the grab bucket is suspended, the grabbing force of the grab bucket is reduced to reduce the material grabbed by the grab bucket, and the grab bucket is lifted again after the grabbing force of the grab bucket is reduced;

[0108] The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

[0109] In a possible implementation, the grab control system may further include a second image unit, a second recognition unit, a second acquisition unit, and a second adjustment unit:

[0110] The second imaging unit is used to capture the image of the material pile before the grab grabs the material;

[0111] A second recognition unit is used to input the material pile image into a preset deep learning model, so that the preset deep learning model recognizes the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the recognized material pile area;

[0112] A second acquisition unit is used to acquire the material category and current stacking angle output by a preset deep learning model;

[0113] The second adjustment unit is used for not adjusting the grabbing position of the grab bucket if the current stacking angle is not less than the angle critical value, and adjusting the grabbing position of the grab bucket if the current stacking angle is less than the angle critical value, and the angle critical value corresponds to the material category.

[0114] The present application also provides an electronic device in an embodiment. Figure 4As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0116] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0117] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any grab control method provided in the embodiment of the present application.

[0118] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any grab control method provided in the embodiment of the present application.

[0119] It should also be noted that the system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0120] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0121] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0122] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0123] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0125] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A grab bucket control method, characterized in that: The grab bucket control method comprises: After the grab grabs the material, an image of the grab is photographed, wherein the image includes the grab and the material grabbed by the grab; Inputting the image into a preset deep learning model so that the preset deep learning model identifies the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket based on the identified image area; Obtaining the current weight of the material grabbed by the grab bucket output by the preset deep learning model; The operation mode of the grab bucket is adjusted according to the comparison result between the current material weight and the weight critical value, and the weight critical value is the rated lifting weight of the grab bucket.

2. The grab bucket control method according to claim 1, characterized in that: The preset deep learning model identifies the image area of ​​the material grabbed by the grab bucket and predicts the current weight of the material grabbed by the grab bucket according to the identified image area, including: The preset deep learning model identifies the image area, determines the material category of the material and the area ratio of the image area in the image area inside the grab bucket; The material weight corresponding to the area ratio and the material category is obtained, and the queried weight is determined as the current material weight.

3. The grab bucket control method according to claim 1, characterized in that: The step of adjusting the operation mode of the grab bucket according to the comparison result between the current material weight and the weight critical value includes: If the current material weight is less than the first multiple of the weight threshold value, the grab bucket is lifted at a first speed; If the current material weight is not less than the weight of the first multiple of the weight critical value and less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at a second speed; If the current material weight is not less than the second multiple of the weight threshold value, the lifting of the grab bucket is stopped, the grabbing force of the grab bucket is reduced to reduce the material grabbed by the grab bucket, and the grab bucket is lifted again after the grabbing force of the grab bucket is reduced; The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

4. The grab bucket control method according to claim 1, characterized in that: Also includes: Before the grab grabs the material, photographing a material pile image of the grabbed material pile; Inputting the material pile image into the preset deep learning model, so that the preset deep learning model identifies the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the identified material pile area; Obtaining the material category and the current stacking angle output by the preset deep learning model; If the current stacking angle is not less than the angle critical value, the grabbing position of the grab bucket is not adjusted. If the current stacking angle is less than the angle critical value, the grabbing position of the grab bucket is adjusted. The angle critical value corresponds to the material category.

5. A grab bucket control system, characterized in that: The grab control system comprises: A first imaging unit is used to capture an image of the grab bucket after the grab bucket grabs the material, wherein the image includes the grab bucket and the material grabbed by the grab bucket; A first recognition unit, configured to input the image into a preset deep learning model, so that the preset deep learning model recognizes an image area of ​​the material grabbed by the grab bucket and predicts a current weight of the material grabbed by the grab bucket based on the recognized image area; A first acquisition unit, used to obtain the current weight of the material grabbed by the grab bucket output by the preset deep learning model; The first adjustment unit is used to adjust the operation mode of the grab bucket according to the comparison result of the current material weight and the weight critical value, and the weight critical value is the rated lifting weight of the grab bucket.

6. The grab bucket control system according to claim 5, characterized in that: The preset deep learning model is specifically configured as follows: The preset deep learning model identifies the image area, determines the material category of the material and the area ratio of the image area in the image area inside the grab bucket; The material weight corresponding to the area ratio and the material category is obtained, and the queried weight is determined as the current material weight.

7. The grab bucket control system according to claim 5, characterized in that: The first adjustment unit is specifically configured as follows: If the current material weight is less than the first multiple of the weight threshold value, the grab bucket is lifted at a first speed; If the current material weight is not less than the weight of the first multiple of the weight critical value and less than the weight of the second multiple of the weight critical value, the grab bucket is lifted at a second speed; If the current material weight is not less than the second multiple of the weight threshold value, the lifting of the grab bucket is stopped, the grabbing force of the grab bucket is reduced to reduce the material grabbed by the grab bucket, and the grab bucket is lifted again after the grabbing force of the grab bucket is reduced; The first speed is faster than the second speed, and the first multiple is smaller than the second multiple.

8. The grab bucket control system according to claim 5, characterized in that: The grab control system further includes a second image unit, a second recognition unit, a second acquisition unit and a second adjustment unit: The second imaging unit is used to capture a material pile image of the grabbed material pile before the grab grab grabs the material; The second recognition unit is used to input the material pile image into the preset deep learning model, so that the preset deep learning model recognizes the material pile area in the material pile image, and predicts the material type and current stacking angle of the material pile according to the recognized material pile area; The second acquisition unit is used to acquire the material category and the current stacking angle output by the preset deep learning model; The second adjustment unit is used for not adjusting the grabbing position of the grab bucket if the current stacking angle is not less than the angle critical value, and for adjusting the grabbing position of the grab bucket if the current stacking angle is less than the angle critical value, and the angle critical value corresponds to the material category.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the grab bucket control method as described in any one of claims 1 to 4.

10. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the grab control method according to any one of claims 1 to 4.

Images