Counterweight control method, device and equipment for engineering machinery and storage medium thereof

Abstract

The invention discloses a counterweight control method, device and equipment for engineering machinery and a storage medium thereof, and relates to the technical field of engineering machinery. The method comprises the following steps: segmenting and extracting a preliminary outline of a counterweight oil cylinder and a preliminary outline of a limiting clamping groove in a video frame based on an image recognition model; fitting a complete outline of the counterweight oil cylinder according to the preliminary outline of the counterweight oil cylinder and a common oil cylinder shape template; performing curve fitting on the initial contour of the limiting clamping groove to determine a complete contour of the limiting clamping groove; the distance between the counterweight oil cylinder and the limiting clamping groove is determined according to the complete contour of the counterweight oil cylinder and the complete contour of the limiting clamping groove; and the relative positions of the counterweight oil cylinder and the limiting clamping groove are adjusted according to the distance so as to realize counterweight hanging. According to the method, automatic mounting of the counterweight can be achieved in an image capturing and recognizing mode, and the adverse effect of electromagnetic interference signals in the environment on determination of the distance between the counterweight oil cylinder and the counterweight when the distance between the counterweight oil cylinder and the counterweight is determined through a radio device is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to a counterweight control method, device, equipment and storage medium for engineering machinery. Background Art

[0002] In the existing crane counterweight mounting method, the installation of the counterweight is based on the communication between radio devices to measure the distance between the counterweight connector and the counterweight in the adjustable counterweight, and the counterweight connector is connected to the counterweight according to the distance. However, the radio devices are susceptible to environmental interference, and as the distance between the radio devices increases, the interference signal becomes more obvious, so there is a problem of difficulty in determining the distance between the counterweight connector and the counterweight, which will hinder the successful mounting of the counterweight. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a counterweight control method, device, equipment and storage medium for engineering machinery, so as to solve the technical problem in the prior art that the radio device is easily affected by environmental interference and hinders the successful mounting of the counterweight.

[0004] In order to achieve the above-mentioned object, the first aspect of the present application provides a counterweight control method for an engineering machine, the engineering machine comprising:

[0005] Counterweight, with limit slot;

[0006] The counterweight cylinder is used to connect the counterweight through the limit slot;

[0007] A camera, used to obtain images of the limit slot and the counterweight cylinder;

[0008] Counterweight control methods include:

[0009] Obtain video frames of the counterweight cylinder and limit slot through the camera;

[0010] The preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame are extracted based on the image recognition model segmentation;

[0011] Fit the complete outline of the counterweight cylinder based on the preliminary outline of the counterweight cylinder and the common cylinder shape template;

[0012] Performing curve fitting on the preliminary contour of the limit slot to determine the complete contour of the limit slot;

[0013] Determine the distance between the counterweight cylinder and the limit slot according to the complete outline of the counterweight cylinder and the complete outline of the limit slot;

[0014] The relative positions of the counterweight cylinder and the limit slot are adjusted according to the spacing to achieve counterweight mounting.

[0015] In an embodiment of the present application, the counterweight control method also includes: obtaining a post-mounting video frame of the counterweight cylinder and the limit slot; determining the post-mounting spacing between the counterweight cylinder and the limit slot according to the post-mounting video frame; extracting the counterweight top plate contour and the remaining counterweight contours of the counterweight in the post-mounting video frame based on the image recognition model segmentation; performing curve fitting on the counterweight top plate contour and the remaining counterweight contours respectively to determine the lower boundary line of the counterweight top plate and the upper boundary line of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights; determining a first gap value between the counterweight top plate and the remaining counterweights according to the lower boundary line and the upper boundary line; when the post-mounting spacing exceeds a first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds a second distance threshold, a counterweight sinking alarm signal is issued, wherein the value of the spacing between the counterweight top plate and the remaining counterweights is the first gap value.

[0016] In an embodiment of the present application, the counterweight control method also includes: determining a second gap value between the counterweight top plate and the remaining counterweights based on the counterweight top plate contour and the remaining counterweight contours; wherein the value of the gap between the counterweight top plate and the remaining counterweights is the larger of the first gap value and the second gap value.

[0017] In an embodiment of the present application, the preliminary outline of the limiting card slot includes a set of card slot outline pixel coordinates in a video frame; curve fitting is performed on the preliminary outline of the limiting card slot to determine the complete outline of the limiting card slot, including: determining the complete outline of the limiting card slot by least squares curve fitting based on the card slot outline pixel coordinate set.

[0018] In an embodiment of the present application, a preliminary outline of the counterweight cylinder and a preliminary outline of the limit slot in a video frame are segmented and extracted based on an image recognition model, including: determining a region of interest in the video frame based on the image recognition model, the region of interest including an image of the counterweight cylinder and an image of the limit slot; segmenting and extracting the video frame of the region of interest to obtain a preliminary outline of the counterweight cylinder and a preliminary outline of the limit slot.

[0019] In the embodiment of the present application, the camera is a monocular camera.

[0020] In an embodiment of the present application, an image recognition model is obtained by the following steps: obtaining a training image set, wherein the training image set includes a plurality of counterweight area images, the counterweight area images including images of a limit slot, an image of a counterweight cylinder, an image of a counterweight top plate, and images of the remaining counterweights; annotating the contour range of the limit slot, the contour range of the counterweight cylinder, the contour range of the counterweight top plate, and the contour range of the counterweight cylinder in each counterweight area image in the training image set to obtain an annotated image of each counterweight area image; training a preset image segmentation and recognition Transformer model according to the training image set and each annotated image until a preset convergence condition is met to obtain a trained image segmentation and recognition Transformer model as an image recognition model; wherein the preset convergence condition includes: the number of training rounds reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than a preset loss value.

[0021] A second aspect of the present application provides a counterweight control device for engineering machinery, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and to implement a counterweight control method for engineering machinery provided according to any one of the above embodiments when executing the instructions.

[0022] The third aspect of the present application provides a counterweight control device for engineering machinery, including: a counterweight with a limit slot; a counterweight cylinder for connecting the counterweight through the limit slot; a camera for acquiring images of the counterweight cylinder and the limit slot; a mounting drive device for controlling the position of the counterweight cylinder relative to the counterweight; According to the second aspect of the present application, a counterweight control device for engineering machinery is provided, which is used to acquire images and control the mounting drive device.

[0023] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute a counterweight control method for an engineering machine provided according to any one of the above embodiments.

[0024] Through the above technical scheme, the counterweight control method for engineering machinery provided in the embodiment of the present application can realize automatic mounting of the counterweight by means of image capture and recognition, and avoids the adverse effects of electromagnetic interference signals in the environment on the determination of the distance between the counterweight cylinder and the counterweight when a radio device is used to determine the distance between the counterweight cylinder and the counterweight.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 A schematic diagram of a flow chart of a counterweight control method for engineering machinery according to an embodiment of the present application is schematically shown;

[0028] Figure 2 A schematic diagram of a flow chart of another counterweight control method for engineering machinery according to an embodiment of the present application is shown;

[0029] Figure 3 The structural block diagram of a counterweight control device for engineering machinery according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0032] If there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0033] The radio device used in the existing crane counterweight mounting method may include a radio generating device and a radio receiving device, one of which is set at the counterweight cylinder and the other is set at the counterweight, and is used to measure the distance between the counterweight connecting piece and the counterweight through radio signals. The on-board processor or the remote processor relative to the crane adjusts the relative position of the counterweight connecting piece and the counterweight according to the distance, thereby realizing the counterweight mounting. Since the counterweight cylinder and the counterweight are often metal parts, there may be complex electromagnetic interference signals in the construction environment. In the process of determining the relative position of the counterweight cylinder and the counterweight by a radio device, thereby controlling the counterweight cylinder and the counterweight to achieve mounting, the signal received by the radio receiving device not only includes the signal emitted by the radio generating device to express the relative position of the counterweight cylinder and the counterweight, but also receives interference signals, which will cause misjudgment when judging the distance between the counterweight cylinder and the counterweight according to the signal received by the radio receiving device. Moreover, as the distance between the counterweight cylinder and the counterweight increases, the intensity of the interference signal is stronger relative to the intensity of the signal generated by the radio generating device, so the misjudgment of the distance between the counterweight cylinder and the counterweight is more likely to occur, which will cause the crane counterweight mounting to fail or the counterweight cylinder and the counterweight to collide. Based on the analysis of the above-mentioned prior art, the embodiment of the present application provides a counterweight control method for engineering machinery, which determines the relative position between the limit slots of the counterweight cylinder and the counterweight by image capture and recognition to adjust the position of the counterweight cylinder and the counterweight, thereby realizing automatic mounting of the counterweight.

[0034] Figure 1 The following schematically shows a flow chart of a counterweight control method for engineering machinery according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a counterweight control method for engineering machinery, and the engineering machinery to which the method is applied includes: a counterweight, a counterweight cylinder, and a camera. The counterweight has a limit slot. The counterweight cylinder is used to connect the counterweight through the limit slot. The camera is used to obtain images of the limit slot and the counterweight cylinder.

[0035] The above-mentioned counterweight control method may include the following steps:

[0036] S102, obtaining video frames of the counterweight cylinder and the limit slot through a camera;

[0037] S104, extracting the preliminary outline of the counterweight cylinder in the video frame based on the image recognition model segmentation and

[0038] Preliminary outline of the limit slot;

[0039] S106, fitting a complete outline of the counterweight cylinder according to the preliminary outline of the counterweight cylinder and a common cylinder shape template;

[0040] S108, curve fitting is performed on the preliminary contour of the limit slot to determine the complete contour of the limit slot.

[0041] contour;

[0042] S110, determine the counterweight oil according to the complete outline of the counterweight cylinder and the complete outline of the limit slot

[0043] The distance between the cylinder and the limit slot;

[0044] S112, adjusting the relative positions of the counterweight cylinder and the limit slot according to the spacing to achieve counterweight mounting.

[0045] The counterweight control method for engineering machinery provided in the embodiment of the present application uses a camera to capture images of a limit slot and a counterweight cylinder, and obtains video frames of the counterweight cylinder and the limit slot based on the captured images, and further extracts the preliminary contour of the counterweight cylinder and the preliminary contour of the limit slot in the video frame through image recognition model segmentation. Because the image of the counterweight cylinder and / or the image of the limit slot in the video frame may be blocked and cannot be displayed in the preliminary contour, further contour processing is performed based on the preliminary contour of the counterweight cylinder and the preliminary contour of the limit slot determined above. The shape of the counterweight cylinder is generally regular, so the complete contour of the counterweight cylinder can be obtained by fitting the preliminary contour of the counterweight cylinder with a common cylinder shape template. In addition, for the limit slot whose contour regularity is difficult to represent through a common template, the complete contour of the limit slot is determined by fitting the preliminary contour of the limit slot through curve fitting. Then, the distance between the counterweight cylinder and the limit slot can be determined according to the complete outline of the counterweight cylinder and the complete outline of the limit slot, and the relative position of the counterweight cylinder and the limit slot can be adjusted according to the distance, so that the counterweight cylinder can be successfully extended into and fixed in the limit slot to realize the automatic mounting of the counterweight. Therefore, the counterweight control method for engineering machinery provided in the embodiment of the present application can realize the automatic mounting of the counterweight by image capture and recognition, and avoids the adverse effect of electromagnetic interference signals in the environment on the determination of the distance between the counterweight cylinder and the counterweight when a radio device is used to determine the distance between the counterweight cylinder and the counterweight.

[0046] It can be understood that the above-mentioned construction machinery can be, for example: various types of cranes, or other construction machinery such as excavators and loaders that realize counterweight mounting through counterweight cylinders and counterweight limit slots. And the camera included in the construction machinery can be used as a part of the crane and other machinery for construction, or as a combination of the above-mentioned machinery. The image acquired by the camera can be, for example, a plurality of video frames in a time sequence, so it can also be understood as a video image. The image recognition model in step S104 can be, for example, a convolutional neural network model, a yolo model, a Transformer model, and other neural network models that can be used for image recognition. In step S108, the curve fitting method for the preliminary contour of the limit slot may include least squares fitting and spline interpolation. In step S112, the spacing value of the distance between the counterweight cylinder and the limit slot can, for example, have a functional mapping relationship with the hydraulic control signal strength of the counterweight cylinder, and the relative positions of the counterweight cylinder and the limit slot at different times can be judged through multiple video frames, thereby realizing continuous adjustment of the relative positions of the counterweight cylinder and the limit slot until the counterweight is mounted, or the mounting fails within a preset time period and a failure alarm signal is generated.

[0047] Specifically, step S112 may include:

[0048] Determine the hydraulic control signal received by the counterweight cylinder according to the spacing and the preset spacing value-hydraulic control signal strength mapping relationship;

[0049] Determine a first distance that the counterweight cylinder extends toward the limit slot according to the hydraulic control signal;

[0050] After the counterweight cylinder extends the first distance, the distance between the counterweight cylinder and the limit slot and the corresponding hydraulic control signal are re-determined according to the current video frame of the counterweight cylinder and the limit slot after the first distance is extended.

[0051] In some embodiments of the present application, in order to monitor the sinking phenomenon of the counterweight after the counterweight is successfully mounted, the counterweight control method for engineering machinery provided in the embodiments of the present application may further include:

[0052] S202, obtaining a video frame of the counterweight cylinder and the limit slot after mounting;

[0053] S204, determining the post-mounting distance between the post-mounting counterweight cylinder and the limit slot according to the post-mounting video frame;

[0054] S206, extracting the contour of the counterweight top plate and the contours of the remaining counterweights in the mounted video frame based on the image recognition model segmentation;

[0055] S208, performing curve fitting on the contour of the counterweight top plate and the contours of the remaining counterweights, respectively, to determine the lower boundary line of the counterweight top plate and the upper boundary lines of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights;

[0056] S210, determining a first gap value between the counterweight top plate and the remaining counterweights according to the lower boundary line and the upper boundary line;

[0057] S212. When the spacing after mounting exceeds a first distance threshold, and the spacing between the counterweight top plate and the remaining counterweights exceeds a second distance threshold, a counterweight sinking alarm signal is issued, wherein the spacing between the counterweight top plate and the remaining counterweights is taken as a first gap value.

[0058] The counterweight includes the counterweight top plate and the remaining counterweights. When the remaining counterweights are tightly connected to the counterweight top plate, it can be determined that the counterweight has not sunk. When the remaining counterweights are far away from the counterweight top plate, if the counterweight cylinder is connected to the limit slots of the remaining counterweights, it means that the counterweight has not sunk. Only when there is a gap between the remaining counterweights and the top counterweight or the distance between the two is too far and the limit slots of the counterweight cylinder and the cylinder connected to the bottom counterweight fall off, it is determined that the counterweight has sunk abnormally. The counterweight control method for engineering machinery provided in the embodiment of the present application determines the post-mounting spacing between the counterweight cylinder and the limit slot, and extracts the counterweight top plate contour and the remaining counterweight contours based on image recognition model segmentation, and thereby determines the lower boundary line of the counterweight top plate and the upper boundary line of the remaining counterweights through curve fitting, thereby determining the first gap value between the counterweight top plate and the remaining counterweights, and whether the remaining counterweights and the counterweight top plate are tightly connected can be determined based on the first gap value, and at the same time, whether the counterweight cylinder and the limit slots of the remaining counterweights are connected is determined based on the post-mounting spacing determined in step S204, thereby automatically judging the counterweight sinking phenomenon, and when the post-mounting spacing exceeds the first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds the second distance threshold, it is judged that the counterweight has sunk and a counterweight sinking alarm signal is issued, so that subsequent devices can take corresponding measures or remind construction personnel to observe whether the counterweight is actually sinking.

[0059] It is understandable that the post-mounting spacing in step S204 can be achieved, for example, by Figure 1The number of counterweight blocks of the remaining counterweights may be, for example, single or multiple. In the case where the number of counterweight blocks of the remaining counterweights is multiple, it is only necessary to consider whether the counterweight cylinder is connected to the limit slot of the bottom counterweight in the remaining counterweights in the judgment of whether the counterweight cylinder is connected. The first distance threshold in step S212 can be determined according to the connection of the limit slot of the counterweight cylinder and the counterweight, so as to achieve the distance of the counterweight cylinder relative to the counterweight after mounting and the thickness of the single counterweight. For example, relative to the video frame after mounting, the first distance threshold can be set to the sum of the distance of the counterweight cylinder relative to the counterweight after mounting and the thickness of the single counterweight. The second distance threshold is used to determine whether there is a gap between the counterweight top plate and the remaining counterweights, so it can be set as small as possible, for example, the minimum non-zero spacing between the upper boundary line and the lower boundary line that can be identified by the image recognition model, or the size threshold set by the staff experience based on the size of the counterweight top plate and the counterweight block.

[0060] Furthermore, in order to better determine whether there is a gap between the counterweight top plate and the remaining counterweights, the counterweight control method for engineering machinery provided in the embodiment of the present application may also include:

[0061] The second gap value between the counterweight top plate and the remaining counterweights is determined according to the contour of the counterweight top plate and the contours of the remaining counterweights; wherein the value of the gap between the counterweight top plate and the remaining counterweights is the larger one of the first gap value and the second gap value.

[0062] Since the upper boundary line and the lower boundary line determined by curve fitting are often straight lines, the local gap between the counterweight top plate and the remaining counterweights caused by factors such as weight bending will be ignored. Therefore, it is further proposed to directly determine whether there is a local gap between the counterweight top plate and the remaining counterweights based on the contour of the counterweight top plate and the contour of the remaining counterweights, and represent the local gap by a second gap value. The largest gap value between the first gap value and the second gap value is selected as the evaluation index to indicate whether there is a gap between the counterweight top plate and the remaining counterweights.

[0063] It can be understood that the second gap value may be, for example, the maximum distance value between the identifiable pixels of the counterweight top plate contour and the remaining counterweight contours, when there are identifiable pixels along the height or thickness direction of the counterweight top plate and the remaining counterweights. This maximum distance value may be expressed, for example, as the number of pixels spaced between the identifiable pixels of the two in the video frame after mounting.

[0064] In some embodiments of the present application, the preliminary outline of the position-limiting card slot in step S104 includes a card slot outline pixel coordinate set in the video frame;

[0065] Step S108 may include:

[0066] According to the pixel coordinate set of the slot contour, the complete contour of the limit slot is determined by least square curve fitting.

[0067] It can be understood that the least square curve fitting can obtain a polynomial function for representing the complete outline of the card slot, and the independent variable and dependent variable of the function can be, for example, the horizontal coordinate value and the vertical coordinate value of the complete outline coordinate of the card slot, respectively.

[0068] Specifically, the preliminary polynomial of the least squares curve fitting can be:

[0069] y=a0+a1*x+a2*x 2 +…+a k *x k

[0070] Substitute the slot outline pixel coordinates into the above preliminary polynomial to obtain the matrix equation shown below:

[0071]

[0072] where (x1,y1)…(x n ,y n ) is the corresponding contour coordinate point in the slot contour pixel coordinate set.

[0073] Based on the above matrix equation, a0 to a k The values ​​of the coefficients of are obtained to obtain a polynomial function used to represent the complete profile of the card slot.

[0074] In some embodiments of the present application, step S206 may include:

[0075] Determine an area of ​​interest in the video frame based on an image recognition model, the area of ​​interest including an image of a counterweight cylinder and an image of a limit slot;

[0076] The video frames of the area of ​​interest are segmented and extracted to obtain the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot.

[0077] Since there may be multiple construction machines at the construction site, the images of the counterweight areas of other construction machines may be recorded in the video frame. Therefore, based on the image recognition model, the region of interest in the video frame is first determined to exclude the interference of the counterweight areas of other construction machines on the subsequent image processing steps.

[0078] It can be understood that there may be multiple areas including images of the counterweight cylinder and images of the limit slot in the video frame, and the area of ​​interest may be, for example, the largest area among the multiple areas mentioned above.

[0079] In some embodiments of the present application, the camera included in the engineering machinery is a monocular camera.

[0080] Therefore, the images obtained by the camera are multiple single video frames obtained in time sequence. A monocular camera is used to simultaneously perform mounting detection and counterweight sinking abnormality detection, and the device is simple. Through a single multi-classification instance segmentation model, the recognition and segmentation of multiple targets can be achieved, and the computing power requirement is not high, so it can be deployed on an embedded platform to achieve real-time detection, thereby meeting the computing speed requirements of real-time detection.

[0081] Furthermore, in some embodiments of the present application, the image recognition model is obtained by the following steps:

[0082] Acquire a training image set, wherein the training image set includes a plurality of counterweight area images, and the counterweight area images include images of a limit slot, images of a counterweight cylinder, images of a counterweight top plate, and images of other counterweights;

[0083] Annotate the contour range of the limit slot, the contour range of the counterweight cylinder, the contour range of the counterweight top plate, and the contour range of the counterweight cylinder in each counterweight area image in the training image set to obtain an annotation image of each counterweight area image;

[0084] The preset image segmentation recognition Transformer model is trained according to the training image set and each annotated image until a preset convergence condition is met, so as to obtain the trained image segmentation recognition Transformer model as the image recognition model;

[0085] Among them, the preset convergence conditions include: the training round reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than the preset loss value.

[0086] The image recognition model obtained by the above training method can determine whether there are counterweight slots, counterweight cylinders, counterweight top plates and other counterweights in a video frame based on a single video frame image, and determine the contours of the above objects in the video frame. Therefore, the image recognition model is a single multi-classification instance segmentation model to reduce the model's computing and storage resource requirements.

[0087] In addition, the training image set may also include images of the counterweight area in different weather and time, so that the image recognition model can also recognize the contours of the counterweight slot, counterweight cylinder, counterweight top plate and other counterweights during the day, at night and in different weather conditions, thereby improving the stability of mounting detection and counterweight sinking abnormality detection.

[0088] Taking a crane as an example, a complete exemplary description of the counterweight control method for engineering machinery provided in the embodiment of the present application is given below:

[0089] First, the monocular camera is installed at a fixed position in the counterweight area of ​​the crane to ensure that the counterweight cylinder and the limit slots on the bottom counterweight connecting the counterweight cylinder are within the camera's field of view. The instance segmentation models of multiple categories of the quantized single model are converted and deployed to the embedded system through TensorRT to complete the integrated device of the counterweight automatic mounting and sinking abnormality detection algorithm.

[0090] based on Figure 1 The counterweight control method for engineering machinery shown in the figure realizes automatic mounting detection of the counterweight and completes the connection between the counterweight cylinder and the limit slot.

[0091] Then based on Figure 2 The counterweight control method for engineering machinery shown realizes counterweight sinking abnormality detection and sends a counterweight sinking alarm signal when counterweight sinking is detected.

[0092] In summary, the counterweight control method for engineering machinery provided in the embodiment of the present application adopts a multi-class instance segmentation model algorithm. The main segmentation targets include the counterweight cylinder, the limit slot connected to the cylinder on the bottom counterweight, the counterweight, the top steel plate in the counterweight area and the remaining counterweights for segmentation and extraction. A deep learning instance segmentation model is used to perform rough extraction of each target, which can mainly solve the robustness of target segmentation and extraction in various complex scenes (strong light scenes, rainy scenes, night scenes) and enhance the adaptability of the algorithm to various scenes.

[0093] Using a single model (but being able to identify and segment multiple categories at the same time) can reduce the model's computing resource requirements, enable the model to be deployed on an embedded system platform more effectively, reduce resource consumption, and enable real-time simultaneous detection of counterweight mounting and counterweight sinking anomalies.

[0094] The deep learning model is used to extract the target, and the interference of segmentation noise is reduced through post-processing fitting and filtering of some traditional algorithms, thereby improving the accuracy of contour extraction of each target.

[0095] The extraction of the region of interest can limit the interference of targets on the surrounding cranes on the targets on the current crane. After extracting the corresponding area on the crane, the model is segmented and extracted, which helps to increase the stability of the algorithm.

[0096] The present application also provides a counterweight control device for engineering machinery, comprising: a memory and a processor. The memory is configured to store instructions. The processor is configured to call instructions from the memory and implement the counterweight control method for engineering machinery provided in any of the above embodiments when executing the instructions.

[0097] like Figure 3As shown, an embodiment of the present application also provides a counterweight control device for engineering machinery, including: a counterweight, a counterweight cylinder 310, a camera 320, a mounting drive device 330, and a counterweight control device 340 for engineering machinery. The counterweight has a limit slot. The counterweight cylinder 310 is used to connect the counterweight through the limit slot. The camera 320 is used to obtain images of the counterweight cylinder 310 and the limit slot. The mounting drive device 330 is used to control the position of the counterweight cylinder 310 relative to the counterweight. The counterweight control device for engineering machinery provided in any of the above embodiments is used to obtain images and control the mounting drive device 330.

[0098] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned counterweight control method for engineering machinery.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0100] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0104] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0105] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0106] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0107] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A counterweight control method for engineering machinery, characterized in that: The engineering machinery includes: Counterweight, with limit slot; A counterweight oil cylinder, used for connecting the counterweight through the limit slot; A camera, used to obtain images of the limit slot and the counterweight cylinder; The counterweight control method comprises: Acquire video frames of the counterweight cylinder and the limit slot through the camera; Extracting the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame based on the image recognition model segmentation; Fitting the complete outline of the counterweight cylinder according to the preliminary outline of the counterweight cylinder and a common cylinder shape template; Performing curve fitting on the preliminary contour of the limit slot to determine the complete contour of the limit slot; Determine the distance between the counterweight cylinder and the limit slot according to the complete outline of the counterweight cylinder and the complete outline of the limit slot; The relative positions of the counterweight cylinder and the limit slot are adjusted according to the spacing to achieve counterweight mounting.

2. The counterweight control method according to claim 1, characterized in that: Also includes: Obtaining a video frame of the counterweight cylinder and the limit slot after mounting; Determine the post-mounting distance between the counterweight cylinder and the limit slot after mounting according to the post-mounting video frame; Segment and extract the counterweight top plate contour of the counterweight and the remaining counterweight contours of the counterweight in the mounted video frame based on the image recognition model; Performing curve fitting on the contour of the counterweight top plate and the contours of the remaining counterweights respectively to determine the lower boundary line of the counterweight top plate and the upper boundary line of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights; Determine a first gap value between the counterweight top plate and the remaining counterweights according to the lower boundary line and the upper boundary line; When the spacing after mounting exceeds the first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds the second distance threshold, a counterweight sinking alarm signal is issued, wherein the spacing between the counterweight top plate and the remaining counterweights is taken as the first gap value.

3. The counterweight control method according to claim 2, characterized in that: Also includes: Determine a second gap value between the counterweight top plate and the remaining counterweights according to the counterweight top plate profile and the remaining counterweight profiles; Wherein, the interval between the counterweight top plate and the remaining counterweights is set to the larger one of the first gap value and the second gap value.

4. The counterweight control method according to claim 1, characterized in that: The preliminary outline of the position-limiting card slot includes a card slot outline pixel coordinate set in the video frame; Performing curve fitting on the preliminary contour of the limit slot to determine the complete contour of the limit slot includes: According to the slot outline pixel coordinate set, the complete outline of the position limiting slot is determined by least squares curve fitting.

5. The counterweight control method according to claim 1, characterized in that: The segmentation and extraction of the preliminary contour of the counterweight cylinder and the preliminary contour of the limit slot in the video frame based on the image recognition model includes: Determine a region of interest in the video frame based on the image recognition model, wherein the region of interest includes an image of the counterweight cylinder and an image of the limit slot; The video frames of the region of interest are segmented and extracted to obtain a preliminary outline of the counterweight cylinder and a preliminary outline of the limit slot.

6. The counterweight control method according to claim 2, characterized in that: The camera is a monocular camera.

7. The counterweight control method according to claim 6, characterized in that: The image recognition model is obtained by the following steps: Acquire a training image set, wherein the training image set includes a plurality of counterweight area images, and the counterweight area images include an image of the limit slot, an image of the counterweight cylinder, an image of the counterweight top plate, and images of the remaining counterweights; Annotating the contour range of the limit slot, the contour range of the counterweight cylinder, the contour range of the counterweight top plate and the contour range of the counterweight cylinder in each of the counterweight area images in the training image set to obtain an annotated image of each of the counterweight area images; Training a preset image segmentation and recognition Transformer model according to the training image set and each of the annotated images until a preset convergence condition is met, so as to obtain a trained image segmentation and recognition Transformer model as the image recognition model; Among them, the preset convergence condition includes: the training round reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than a preset loss value.

8. A counterweight control device for engineering machinery, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the counterweight control method for an engineering machine according to any one of claims 1 to 7 when executing the instructions.

9. A counterweight control device for engineering machinery, characterized in that: include: Counterweight, with limit slot; A counterweight oil cylinder, used for connecting the counterweight through the limit slot; A camera, used to obtain images of the counterweight cylinder and the limit slot; A mounting drive device, used to control the position of the counterweight cylinder relative to the counterweight; The counterweight control device for engineering machinery according to claim 8 is used to obtain the image and control the mounting drive device.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the counterweight control method for an engineering machine according to any one of claims 1 to 7.

Images