Counterweight control method, device, equipment and storage medium for engineering machinery

Abstract

The present application discloses a counterweight control method, device, equipment and storage medium for engineering machinery, and relates to the technical field of engineering machinery. The above method includes: extracting the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame based on the image recognition model segmentation; fitting the complete outline of the counterweight cylinder according to the preliminary outline of the counterweight cylinder and the common cylinder shape template; curve fitting the preliminary outline of the limit slot to determine the complete outline of the limit slot; determining the distance between the counterweight cylinder and the limit slot according to the complete outline of the counterweight cylinder and the complete outline of the limit slot; adjusting the relative position of the counterweight cylinder and the limit slot according to the distance to achieve counterweight mounting. The above method can realize automatic mounting of the counterweight by image capture and recognition, and avoids the adverse effects of electromagnetic interference signals in the environment on the determination of the distance between the counterweight cylinder and the counterweight when using a radio device to determine the distance between the counterweight cylinder and the counterweight.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to a counterweight control method, device, equipment and storage medium thereof for engineering machinery. Background Art

[0002] In existing crane counterweight mounting methods, counterweight installation relies on communication between radio devices to measure the distance between the counterweight connector and the counterweight. This distance is then used to connect the counterweight connector to the counterweight. However, radio devices are susceptible to environmental interference, and as the distance between the radio devices increases, the interference signal becomes more pronounced. This makes it difficult to determine the distance between the counterweight connector and the counterweight, hindering successful counterweight mounting. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a counterweight control method, device, equipment and storage medium for engineering machinery, so as to solve the technical problem in the prior art that the radio device is easily affected by environmental interference and hinders the successful mounting of the counterweight.

[0004] In order to achieve the above-mentioned object, the present application provides, in a first aspect, a counterweight control method for an engineering machine, the engineering machine comprising:

[0005] Counterweight, with limit slot;

[0006] The counterweight cylinder is used to connect the counterweight through the limit slot;

[0007] A camera, used to obtain images of the limit slot and the counterweight cylinder;

[0008] Counterweight control methods include:

[0009] Capture video frames of the counterweight cylinder and limit slot through the camera;

[0010] The preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame are extracted based on the image recognition model segmentation;

[0011] Fit the complete outline of the counterweight cylinder based on the preliminary outline of the counterweight cylinder and the common cylinder shape template;

[0012] Perform curve fitting on the preliminary outline of the limit slot to determine the complete outline of the limit slot;

[0013] Determine the distance between the counterweight cylinder and the limit slot according to the complete outline of the counterweight cylinder and the complete outline of the limit slot;

[0014] The relative positions of the counterweight cylinder and the limit slot are adjusted according to the spacing to achieve counterweight mounting.

[0015] In an embodiment of the present application, the counterweight control method also includes: obtaining a post-mounting video frame of the counterweight cylinder and the limit slot; determining the post-mounting spacing between the counterweight cylinder and the limit slot based on the post-mounting video frame; extracting the counterweight top plate contour and the remaining counterweight contours of the counterweight in the post-mounting video frame based on the image recognition model segmentation; performing curve fitting on the counterweight top plate contour and the remaining counterweight contours respectively to determine the lower boundary line of the counterweight top plate and the upper boundary line of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights; determining the first gap value between the counterweight top plate and the remaining counterweights based on the lower boundary line and the upper boundary line; when the post-mounting spacing exceeds the first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds the second distance threshold, a counterweight sinking alarm signal is issued, wherein the value of the spacing between the counterweight top plate and the remaining counterweights is the first gap value.

[0016] In an embodiment of the present application, the counterweight control method also includes: determining a second gap value between the counterweight top plate and the remaining counterweights based on the counterweight top plate contour and the remaining counterweight contours; wherein the value of the gap between the counterweight top plate and the remaining counterweights is the larger of the first gap value and the second gap value.

[0017] In an embodiment of the present application, the preliminary outline of the limiting card slot includes a set of pixel coordinates of the card slot outline in the video frame; curve fitting is performed on the preliminary outline of the limiting card slot to determine the complete outline of the limiting card slot, including: determining the complete outline of the limiting card slot by least squares curve fitting based on the pixel coordinate set of the card slot outline.

[0018] In an embodiment of the present application, the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame are segmented and extracted based on the image recognition model, including: determining the area of ​​interest in the video frame based on the image recognition model, the area of ​​interest including the image of the counterweight cylinder and the image of the limit slot; segmenting and extracting the video frame of the area of ​​interest to obtain the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot.

[0019] In the embodiment of the present application, the camera is a monocular camera.

[0020] In an embodiment of the present application, an image recognition model is obtained by the following steps: obtaining a training image set, wherein the training image set includes a plurality of counterweight area images, and the counterweight area images include images of limit slots, images of counterweight cylinders, images of counterweight top plates, and images of remaining counterweights; annotating the contour ranges of the limit slots, the contour ranges of the counterweight cylinders, the contour ranges of the counterweight top plates, and the contour ranges of the counterweight cylinders in each counterweight area image in the training image set to obtain an annotated image of each counterweight area image; training a preset image segmentation and recognition Transformer model according to the training image set and each annotated image until a preset convergence condition is met to obtain a trained image segmentation and recognition Transformer model as an image recognition model; wherein the preset convergence condition includes: the number of training rounds reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than a preset loss value.

[0021] A second aspect of the present application provides a counterweight control device for engineering machinery, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and, when executing the instructions, to implement a counterweight control method for engineering machinery provided according to any one of the above embodiments.

[0022] The third aspect of the present application provides a counterweight control device for engineering machinery, including: a counterweight with a limit slot; a counterweight cylinder for connecting the counterweight through the limit slot; a camera for acquiring images of the counterweight cylinder and the limit slot; a mounting drive device for controlling the position of the counterweight cylinder relative to the counterweight; according to the second aspect of the present application, a counterweight control device for engineering machinery is provided, which is used to acquire images and control the mounting drive device.

[0023] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the counterweight control method for engineering machinery provided according to any one of the above embodiments.

[0024] Through the above technical solution, the counterweight control method for engineering machinery provided in the embodiment of the present application can realize automatic mounting of the counterweight through image capture and recognition, and avoids the adverse effects of electromagnetic interference signals in the environment on the determination of the distance between the counterweight cylinder and the counterweight when using a radio device to determine the distance between the counterweight cylinder and the counterweight.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0027] Figure 1 The following schematically shows a flow chart of a counterweight control method for engineering machinery according to an embodiment of the present application;

[0028] Figure 2 The following schematically shows a flow chart of another counterweight control method for engineering machinery according to an embodiment of the present application;

[0029] Figure 3 The following schematically shows a structural block diagram of a counterweight control device for engineering machinery according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use such solutions.

[0032] If there are descriptions involving "first", "second", etc. in the embodiments of this application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0033] The radio device used in the existing crane counterweight mounting method may include a radio generating device and a radio receiving device, one of which is set at the counterweight cylinder and the other is set at the counterweight, and is used to measure the distance between the counterweight connector and the counterweight through radio signals. The on-board processor or the remote processor relative to the crane adjusts the relative position of the counterweight connector and the counterweight according to the distance, thereby realizing counterweight mounting. Since the counterweight cylinder and the counterweight are often made of metal, there may be complex electromagnetic interference signals in the construction environment. In the process of determining the relative position of the counterweight cylinder and the counterweight by a radio device, thereby controlling the mounting of the counterweight cylinder and the counterweight, the signal received by the radio receiving device includes not only the signal emitted by the radio generating device to express the relative position of the counterweight cylinder and the counterweight, but also the interference signal. Therefore, when judging the distance between the counterweight cylinder and the counterweight based on the signal received by the radio receiving device, an error occurs. Moreover, as the distance between the counterweight cylinder and the counterweight increases, the intensity of the interference signal becomes stronger relative to the intensity of the signal generated by the radio generating device, and thus the error in the distance between the counterweight cylinder and the counterweight is more likely to occur, thereby causing the crane counterweight mounting to fail or the counterweight cylinder and the counterweight to collide. Based on the analysis of the above-mentioned prior art, the embodiment of the present application provides a counterweight control method for engineering machinery, which determines the relative position between the limit slots of the counterweight cylinder and the counterweight by image capture and recognition, and adjusts the position of the counterweight cylinder and the counterweight, thereby achieving automatic mounting of the counterweight.

[0034] Figure 1 The following schematically shows a flow chart of a counterweight control method for engineering machinery according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a counterweight control method for construction machinery. The construction machinery to which the method is applied includes a counterweight, a counterweight cylinder, and a camera. The counterweight has a limit slot. The counterweight cylinder is connected to the counterweight via the limit slot. The camera is used to capture images of the limit slot and the counterweight cylinder.

[0035] The above-mentioned counterweight control method may include the following steps:

[0036] S102, obtaining video frames of the counterweight cylinder and the limit slot through a camera;

[0037] S104, extracting the preliminary outline of the counterweight cylinder in the video frame based on the image recognition model segmentation and

[0038] Preliminary outline of the limit slot;

[0039] S106, fitting the complete outline of the counterweight cylinder based on the preliminary outline of the counterweight cylinder and the common cylinder shape template;

[0040] S108, performing curve fitting on the preliminary outline of the limit slot to determine the complete outline of the limit slot

[0041] contour;

[0042] S110, determine the balance weight oil according to the complete outline of the balance weight cylinder and the complete outline of the limit slot

[0043] The distance between the cylinder and the limit slot;

[0044] S112. Adjust the relative positions of the counterweight cylinder and the limit slot according to the spacing to achieve counterweight mounting.

[0045] The counterweight control method for engineering machinery provided in the embodiment of the present application uses a camera to capture images of a limit slot and a counterweight cylinder, and obtains a video frame of the counterweight cylinder and the limit slot based on the captured image, and further extracts the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame through image recognition model segmentation. Because the image of the counterweight cylinder and / or the image of the limit slot in the video frame may be blocked and their complete outlines can no longer be displayed in the preliminary outline, further contour processing is performed based on the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot determined above. The shape of the counterweight cylinder is generally regular, so the complete outline of the counterweight cylinder can be obtained by fitting the preliminary outline of the counterweight cylinder with a common cylinder shape template. In addition, for the limit slot whose contour regularity is difficult to represent through a common template, the complete outline of the limit slot is determined by fitting the preliminary outline of the limit slot through curve fitting. Then, the spacing between the counterweight cylinder and the limit slot can be determined based on the complete outline of the counterweight cylinder and the complete outline of the limit slot, and the relative positions of the counterweight cylinder and the limit slot can be adjusted based on the spacing, so that the counterweight cylinder can be successfully extended into and fixed in the limit slot, thereby realizing automatic mounting of the counterweight. Therefore, the counterweight control method for engineering machinery provided in the embodiment of the present application can realize automatic mounting of the counterweight through image capture and recognition, and avoids the adverse effects of electromagnetic interference signals in the environment on the determination of the distance between the counterweight cylinder and the counterweight when using a radio device to determine the distance between the counterweight cylinder and the counterweight.

[0046] It is understandable that the above-mentioned construction machinery may be, for example: various cranes, or other construction machinery such as excavators and loaders that realize counterweight mounting through counterweight cylinders and counterweight limit slots. And the camera included in the construction machinery may be a part of the crane and other machinery for construction, or it may be a combined component of the above-mentioned machinery. The image captured by the camera may be, for example, a plurality of video frames in a time sequence, so it can also be understood as a video image. The image recognition model in step S104 may be, for example, a convolutional neural network model, a yolo model, a Transformer model, or other neural network models that can be used for image recognition. In step S108, the curve fitting method for the preliminary outline of the limit slot may include least squares fitting and spline interpolation. In step S112, the spacing value between the counterweight cylinder and the limit slot may have a functional mapping relationship with the hydraulic control signal strength of the counterweight cylinder, and the relative positions of the counterweight cylinder and the limit slot at different times can be judged through multiple video frames, thereby realizing continuous adjustment of the relative positions of the counterweight cylinder and the limit slot until the counterweight is mounted, or the mounting fails within a preset time period and a failure alarm signal is generated.

[0047] Specifically, step S112 may include:

[0048] Determine the hydraulic control signal received by the counterweight cylinder according to the spacing and the preset spacing value-hydraulic control signal strength mapping relationship;

[0049] Determining a first distance that the counterweight cylinder extends toward the limit slot according to the hydraulic control signal;

[0050] After the counterweight cylinder extends the first distance, the distance between the counterweight cylinder and the limit slot and the corresponding hydraulic control signal are re-determined according to the current video frame of the counterweight cylinder and the limit slot after the first distance is extended.

[0051] In some embodiments of the present application, in order to monitor the sinking of the counterweight after the counterweight is successfully mounted, the counterweight control method for engineering machinery provided in the embodiments of the present application may further include:

[0052] S202, obtaining a video frame of the counterweight cylinder and the limit slot after mounting;

[0053] S204, determining the post-mounting distance between the post-mounting counterweight cylinder and the limit slot according to the post-mounting video frame;

[0054] S206, extracting the contour of the counterweight top plate and the contours of the remaining counterweights in the mounted video frame based on the image recognition model segmentation;

[0055] S208, performing curve fitting on the contour of the counterweight top plate and the contours of the remaining counterweights to determine the lower boundary line of the counterweight top plate and the upper boundary lines of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights;

[0056] S210, determining a first gap value between the counterweight top plate and the remaining counterweights according to the lower boundary line and the upper boundary line;

[0057] S212. When the spacing after mounting exceeds the first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds the second distance threshold, a counterweight sinking alarm signal is issued, wherein the spacing between the counterweight top plate and the remaining counterweights is taken as the first gap value.

[0058] The counterweight consists of a top plate and the remaining counterweights. If the remaining counterweights are tightly connected to the top plate, the counterweights are considered stable. If the remaining counterweights are located farther from the top plate, and the counterweight cylinder is connected to the stopper slots on the remaining counterweights, the counterweights are considered stable. Counterweight sinking is abnormal only if there is a gap between the remaining counterweights and the top counterweight, or if the distance between the two is too great and the stopper slots connecting the counterweight cylinder to the cylinder on the bottom counterweight are disconnected. The counterweight control method for engineering machinery provided in the embodiment of the present application determines the post-mounting spacing between the counterweight cylinder and the limit slot, and extracts the counterweight top plate contour and the remaining counterweight contours based on image recognition model segmentation, and thereby determines the lower boundary line of the counterweight top plate and the upper boundary line of the remaining counterweights through curve fitting, thereby determining a first gap value between the counterweight top plate and the remaining counterweights, and determining whether the remaining counterweights and the counterweight top plate are tightly connected based on the first gap value. At the same time, whether the counterweight cylinder and the limit slots of the remaining counterweights are connected is determined based on the post-mounting spacing determined in step S204, thereby automatically judging the phenomenon of counterweight sinking, and when the post-mounting spacing exceeds a first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds a second distance threshold, judging that the counterweight has sunk and issuing a counterweight sinking alarm signal, so that subsequent devices can take corresponding measures or remind construction personnel to observe whether the counterweight has actually sunk.

[0059] It is understandable that the post-mounting spacing in step S204 can be achieved by, for example, Figure 1The steps S104 to S110 shown are determined. The number of the counterweight blocks of the remaining counterweights can be, for example, single or multiple. In the case where the number of the counterweight blocks of the remaining counterweights is multiple, it is only necessary to consider whether the counterweight cylinder is connected to the limit slot of the bottom counterweight of the remaining counterweights in the judgment of whether the counterweight cylinder is connected. The first distance threshold in step S212 can be determined based on the connection between the limit slot of the counterweight cylinder and the counterweight, so as to achieve the distance of the counterweight cylinder relative to the counterweight after mounting and the thickness of the single counterweight. For example, relative to the video frame after mounting, the first distance threshold can be set to the sum of the distance of the counterweight cylinder relative to the counterweight after mounting and the thickness of the single counterweight. The second distance threshold is used to determine whether there is a gap between the counterweight top plate and the remaining counterweights, so it can be set as small as possible, for example, the minimum non-zero distance between the upper boundary line and the lower boundary line that can be identified by the image recognition model, or the size threshold set by the staff's experience based on the size of the counterweight top plate and the counterweight blocks.

[0060] Furthermore, in order to better determine whether there is a gap between the counterweight top plate and the remaining counterweights, the counterweight control method for engineering machinery provided in the embodiment of the present application may further include:

[0061] The second gap value between the counterweight top plate and the remaining counterweights is determined according to the contour of the counterweight top plate and the contours of the remaining counterweights; wherein the value of the gap between the counterweight top plate and the remaining counterweights is the larger one of the first gap value and the second gap value.

[0062] Because the upper and lower boundaries determined by curve fitting are often straight lines, they often overlook local gaps between the counterweight top plate and the remaining counterweights caused by factors such as bending under load. Therefore, we propose to directly determine the presence of a local gap based on the contours of the counterweight top plate and the remaining counterweights, and represent this gap using a second gap value. The largest gap value between the first and second gap values ​​is selected as the criterion for determining whether a gap exists between the counterweight top plate and the remaining counterweights.

[0063] It can be understood that the second gap value can be, for example, the maximum distance value between the identifiable pixels of the counterweight top plate contour and the remaining counterweight contours on adjacent sides, when there are identifiable pixels along the height or thickness direction of the counterweight top plate and the remaining counterweights. This maximum distance value can be expressed, for example, as the number of pixels between the identifiable pixels of the two in the video frame after mounting.

[0064] In some embodiments of the present application, the preliminary outline of the position-limiting card slot in step S104 includes a card slot outline pixel coordinate set in the video frame;

[0065] Step S108 may include:

[0066] According to the pixel coordinate set of the slot contour, the complete contour of the limit slot is determined by least squares curve fitting.

[0067] It can be understood that the least square curve fitting can obtain a polynomial function for representing the complete outline of the card slot, and the independent variable and dependent variable of the function can be, for example, the horizontal coordinate value and the vertical coordinate value of the complete outline coordinate of the card slot respectively.

[0068] Specifically, the preliminary polynomial for least squares curve fitting can be:

[0069] y=a0+a1*x+a2*x 2 +…+a k *x k

[0070] Substitute the slot outline pixel coordinates into the preliminary polynomial above to obtain the following matrix equation:

[0071]

[0072] where (x1,y1)…(x n ,y n ) is the corresponding contour coordinate point in the slot contour pixel coordinate set.

[0073] Based on the above matrix equation, a0 to a k The values ​​of the coefficients of are used to obtain a polynomial function that represents the complete profile of the card slot.

[0074] In some embodiments of the present application, step S206 may include:

[0075] Determine a region of interest in a video frame based on an image recognition model, where the region of interest includes an image of the counterweight cylinder and an image of the limit slot;

[0076] The video frames of the area of ​​interest are segmented and extracted to obtain the preliminary outlines of the counterweight cylinder and the limit slot.

[0077] Because multiple construction machines may be present at a construction site, the images of the counterweight areas of other machines may be captured in the video frame. Therefore, based on the image recognition model, we first identify the region of interest in the video frame to eliminate interference from the counterweight areas of other machines in subsequent image processing steps.

[0078] It is understandable that there may be multiple areas in the video frame including images of the counterweight cylinder and images of the limit slot, and the area of ​​interest may be, for example, the largest area among the multiple areas.

[0079] In some embodiments of the present application, the camera included in the engineering machinery is a monocular camera.

[0080] Therefore, the images captured by the camera are multiple single video frames captured in a time sequence. Using a monocular camera, both payload detection and counterweight sinking anomaly detection are performed simultaneously, requiring minimal computational power. A single multi-class instance segmentation model can identify and segment multiple targets. This allows for real-time detection on embedded platforms, meeting the computational speed requirements.

[0081] Furthermore, in some embodiments of the present application, the image recognition model is obtained by the following steps:

[0082] Acquire a training image set, wherein the training image set includes a plurality of counterweight area images, the counterweight area images including an image of a limit slot, an image of a counterweight cylinder, an image of a counterweight top plate, and images of other counterweights;

[0083] Annotate the contour range of the limit slot, the contour range of the counterweight cylinder, the contour range of the counterweight top plate, and the contour range of the counterweight cylinder in each counterweight area image in the training image set to obtain an annotated image of each counterweight area image;

[0084] The preset image segmentation and recognition Transformer model is trained according to the training image set and each annotated image until the preset convergence condition is met, so as to obtain the trained image segmentation and recognition Transformer model as the image recognition model;

[0085] Among them, the preset convergence conditions include: the number of training rounds reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than the preset loss value.

[0086] The image recognition model trained using this method can, based on a single video frame, determine the presence of the counterweight slot, counterweight cylinder, counterweight top plate, and remaining counterweights, as well as the location of the outlines of these objects within the video frame. This model is a single, multi-class instance segmentation model, reducing the model's computational and storage requirements.

[0087] In addition, the training image set may also include images of the counterweight area in different weather conditions and at different times, so that the image recognition model can also recognize the contours of the counterweight slot, counterweight cylinder, counterweight top plate and other counterweights during the day, at night and in different weather conditions, thereby improving the stability of mounting detection and counterweight sinking abnormality detection.

[0088] The following uses a crane as an example to provide a complete exemplary description of the counterweight control method for engineering machinery provided in the embodiments of the present application:

[0089] First, a monocular camera was mounted in a fixed position on the crane's counterweight area, ensuring that the counterweight cylinder and the limit slots connecting the cylinder on the bottom counterweight were within the camera's field of view. Using TensorRT, the quantized single model was converted into instance segmentation models for multiple categories and deployed into an embedded system, completing the integration of algorithms for detecting automatic counterweight mounting and sinking anomalies.

[0090] based on Figure 1 The counterweight control method for engineering machinery shown realizes automatic counterweight mounting detection and completes the connection between the counterweight cylinder and the limit slot.

[0091] Then based on Figure 2 The counterweight control method for engineering machinery shown realizes counterweight sinking abnormality detection and issues a counterweight sinking alarm signal when counterweight sinking is detected.

[0092] To sum up, the counterweight control method for engineering machinery provided in the embodiment of the present application adopts a multi-class instance segmentation model algorithm. The main segmentation targets include the counterweight cylinder, the limit slot connecting the cylinder on the bottom counterweight, the counterweight, the top steel plate in the counterweight area and the remaining counterweights for segmentation and extraction. The deep learning instance segmentation model is used to perform rough extraction of each target, which can mainly solve the robustness of target segmentation and extraction in various complex scenes (strong light scenes, rainy scenes, night scenes) and enhance the adaptability of the algorithm to various scenes.

[0093] Using a single model (but being able to identify and segment multiple categories at the same time) can reduce the model's computing resource requirements, enable the model to be more effectively deployed on an embedded system platform, reduce resource consumption, and enable real-time simultaneous detection of counterweight mounting and counterweight sinking anomalies.

[0094] The targets are extracted after adopting the deep learning model, and the interference of segmentation noise is reduced through post-processing fitting and filtering of some traditional algorithms, thereby improving the accuracy of target contour extraction.

[0095] The extraction of the region of interest can limit the interference of targets on the surrounding cranes on the target on the current crane. Extracting the corresponding area on the crane and then performing model segmentation extraction can help increase the stability of the algorithm.

[0096] The present application also provides a counterweight control device for construction machinery, comprising a memory and a processor. The memory is configured to store instructions. The processor is configured to retrieve the instructions from the memory and, when executing the instructions, implement the counterweight control method for construction machinery provided in any of the above-described embodiments.

[0097] like Figure 3As shown, an embodiment of the present application further provides a counterweight control device for engineering machinery, comprising: a counterweight, a counterweight cylinder 310, a camera 320, a mounting drive device 330, and a counterweight control device 340 for engineering machinery. The counterweight has a limit slot. The counterweight cylinder 310 is used to connect the counterweight through the limit slot. The camera 320 is used to obtain images of the counterweight cylinder 310 and the limit slot. The mounting drive device 330 is used to control the position of the counterweight cylinder 310 relative to the counterweight. The counterweight control device for engineering machinery provided in any of the above embodiments is used to obtain images and control the mounting drive device 330.

[0098] An embodiment of the present application further provides a machine-readable storage medium having stored thereon instructions for causing a machine to execute the above-mentioned counterweight control method for engineering machinery.

[0099] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0100] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0103] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0104] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0105] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0106] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0107] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A counterweight control method for engineering machinery, characterized in that: The engineering machinery includes: Counterweight, with limit slot; A counterweight oil cylinder, used for connecting the counterweight through the limit slot; A camera, used to obtain images of the limit slot and the counterweight cylinder; The counterweight control method comprises: Acquire video frames of the counterweight cylinder and the limit slot through the camera; Extracting the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame based on the image recognition model segmentation; Fitting the complete outline of the counterweight cylinder according to the preliminary outline of the counterweight cylinder and a common cylinder shape template; Performing curve fitting on the preliminary contour of the limiting slot to determine the complete contour of the limiting slot; Determine the distance between the counterweight cylinder and the limit slot according to the complete outline of the counterweight cylinder and the complete outline of the limit slot; The relative positions of the counterweight cylinder and the limiting slot are adjusted according to the spacing to achieve counterweight mounting.

2. The counterweight control method according to claim 1, characterized in that: Also includes: Obtaining a video frame of the counterweight cylinder and the limit slot after mounting; Determine the post-mounting distance between the counterweight cylinder and the limit slot after mounting according to the post-mounting video frame; Segmenting and extracting a counterweight top plate contour and other counterweight contours of the counterweight in the mounted video frame based on the image recognition model; Performing curve fitting on the contour of the counterweight top plate and the contours of the remaining counterweights to determine the lower boundary line of the counterweight top plate and the upper boundary lines of the remaining counterweights, wherein the lower boundary line of the counterweight top plate is the boundary line of the counterweight top plate adjacent to the remaining counterweights; Determining a first gap value between the counterweight top plate and the remaining counterweights according to the lower boundary line and the upper boundary line; When the spacing after mounting exceeds a first distance threshold and the spacing between the counterweight top plate and the remaining counterweights exceeds a second distance threshold, a counterweight sinking alarm signal is issued, wherein the spacing between the counterweight top plate and the remaining counterweights is taken as the first gap value.

3. The counterweight control method according to claim 2, characterized in that: Also includes: determining a second gap value between the counterweight top plate and the remaining counterweights according to the counterweight top plate profile and the remaining counterweight profiles; Wherein, the interval between the counterweight top plate and the remaining counterweights is set to the larger one of the first gap value and the second gap value.

4. The counterweight control method according to claim 1, characterized in that: The preliminary outline of the position limiting card slot includes a card slot outline pixel coordinate set in the video frame; Performing curve fitting on the preliminary contour of the limit slot to determine the complete contour of the limit slot includes: According to the slot outline pixel coordinate set, a complete outline of the position-limiting slot is determined by least squares curve fitting.

5. The counterweight control method according to claim 1, characterized in that: The segmentation and extraction of the preliminary outline of the counterweight cylinder and the preliminary outline of the limit slot in the video frame based on the image recognition model includes: Determining a region of interest in the video frame based on the image recognition model, wherein the region of interest includes an image of the counterweight cylinder and an image of the limit slot; The video frame of the region of interest is segmented and extracted to obtain a preliminary outline of the counterweight cylinder and a preliminary outline of the limit slot.

6. The counterweight control method according to claim 2, characterized in that: The camera is a monocular camera.

7. The counterweight control method according to claim 6, characterized in that: The image recognition model is obtained by the following steps: Acquire a training image set, wherein the training image set includes a plurality of counterweight area images, the counterweight area images including an image of the limit slot, an image of the counterweight cylinder, an image of the counterweight top plate, and images of the remaining counterweights; Annotating the contour range of the limit slot, the contour range of the counterweight cylinder, the contour range of the counterweight top plate, and the contour range of the remaining counterweights in each of the counterweight area images in the training image set to obtain an annotated image of each of the counterweight area images; Training a preset image segmentation and recognition Transformer model according to the training image set and each of the labeled images until a preset convergence condition is met, so as to obtain a trained image segmentation and recognition Transformer model as the image recognition model; The preset convergence condition includes: the number of training rounds reaches a preset round value, or the total loss value of the segmentation loss value and the contour loss value of the preset image segmentation and recognition Transformer model is less than a preset loss value.

8. A counterweight control device for engineering machinery, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the counterweight control method for an engineering machine according to any one of claims 1 to 7 when executing the instructions.

9. A counterweight control device for engineering machinery, characterized in that: include: Counterweight, with limit slot; A counterweight oil cylinder, used for connecting the counterweight through the limit slot; A camera, used to obtain images of the counterweight cylinder and the limit slot; A mounting drive device for controlling the position of the counterweight cylinder relative to the counterweight; The counterweight control device for engineering machinery according to claim 8 is used to obtain the image and control the mounting drive device.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions, which are used to enable a machine to execute the counterweight control method for an engineering machine according to any one of claims 1 to 7.

Images