Method, processor, engineering device and storage medium for hoist overshoot monitoring

Abstract

The application discloses a hoist overshoot monitoring method, a processor, engineering equipment and a storage medium, and is applied to engineering equipment of a fuel architecture and a new energy architecture. The engineering equipment comprises a hoist mechanism, the hoist mechanism comprises a boom and a pulley, an image acquisition device is mounted at the head of the boom, an identification pattern is arranged on the pulley, the identification pattern is in the shooting range of the image acquisition device, and the method comprises the following steps: acquiring an image collected by the image acquisition device, wherein the image comprises the identification pattern; determining contour information of the identification pattern in the image according to the image based on a pre-constructed neural network model; determining an actual distance between the pulley and the image acquisition device according to the contour information and predetermined calibration parameters of the image acquisition device; and determining that an overshoot phenomenon occurs in the hoist mechanism in the case that the actual distance is less than a preset safety distance. The application is favorable for reducing a judgment error and improving the accuracy of hoist overshoot monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of engineering machinery, in particular to a hoist overshoot monitoring method, a processor, an engineering device and a storage medium. BACKGROUND

[0002] The hoist monitoring method plays a vital role in the fields of industrial automation and hoisting operations. In the field of automation, especially in hoisting operations, performance monitoring of the hoist is crucial. Traditional hoist monitoring methods mainly rely on manual observation and simple mechanical measurement, such as using a ruler or a pulley block to measure the extension of the steel wire rope or the rotation angle of the hoist to determine whether there is an overshoot risk in the hoisting process. However, manual monitoring is affected by experience and environment, and cannot guarantee accuracy. Therefore, the existing hoist overshoot monitoring method has the problem of large error. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a hoist overshoot monitoring method, a processor, an engineering device and a machine-readable storage medium to solve the problem of large error in the existing hoist overshoot monitoring method.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a hoist overshoot monitoring method applied to an engineering device, the engineering device comprising a hoist mechanism, the hoist mechanism comprising a boom and a pulley, the head of the boom being provided with an image acquisition device, the pulley being provided with a mark pattern, the mark pattern being in the shooting range of the image acquisition device, and the method comprising:

[0005] acquiring an image collected by the image acquisition device, the image comprising the mark pattern;

[0006] determining contour information of the mark pattern in the image based on a pre-constructed neural network model;

[0007] determining an actual distance between the pulley and the image acquisition device according to the contour information and predetermined calibration parameters of the image acquisition device;

[0008] determining that the hoist mechanism has an overshoot phenomenon when the actual distance is less than a predetermined safe distance.

[0009] In the embodiments of the present application, the pre-constructed neural network model comprises a target detection branch, a mark segmentation branch and an accurate segmentation branch. Based on the pre-constructed neural network model, the position information of the mark pattern in the image is determined, which comprises: based on the target detection branch, identifying the region where the mark pattern is located in the image to obtain a local image region containing the mark pattern; based on the mark segmentation branch, segmenting the local image region to obtain a mark region image; and based on the accurate segmentation branch, determining the contour information of the mark pattern according to the mark region image.

[0010] In the embodiment of the present application, based on the accurate segmentation branch, the contour information of the logo pattern is determined according to the logo region image, which comprises: according to the accurate segmentation branch, the first contour, the second contour and the third contour of the logo region image are segmented, the first contour is the contour formed by the logo edge value starting change point in the logo region image, the second contour is the contour formed by the gradient change end point in the logo region image, and the third contour is the contour formed by the midpoint of the first contour and the second contour; the first contour, the second contour and the third contour are fused to obtain the contour information of the logo pattern.

[0011] In the embodiment of the present application, before the contour information of the logo pattern is determined according to the logo region image based on the accurate segmentation branch, it further comprises: the logo pattern in the logo region image is completed according to the geometric feature information of the logo pattern.

[0012] In the embodiment of the present application, before the local image region is segmented to obtain the logo region image based on the logo segmentation branch, it further comprises: the maximum gray value and the minimum gray value in the local image region are determined; the adjustment coefficient is determined according to the maximum gray value and the minimum gray value; the pixel enhancement processing is performed on the local image region according to the adjustment coefficient and the minimum gray value.

[0013] In the embodiment of the present application, the logo pattern comprises a first graph and a second graph, the first graph surrounds the second graph, and the contour information comprises an outer contour corresponding to the first graph and an inner contour corresponding to the second graph; the actual distance between the pulley and the image acquisition device is determined according to the contour information and the predetermined calibration parameter of the image acquisition device, which comprises: the outer contour size is corrected according to the geometric feature information of the first graph, and the inner contour size is corrected according to the geometric feature information of the second graph, so as to obtain the corrected contour information; the actual distance between the pulley and the image acquisition device is determined according to the corrected contour information and the calibration parameter.

[0014] In the embodiment of the present application, the actual distance between the pulley and the image acquisition device is determined according to the corrected contour information and the calibration parameter, which comprises: the logo image area of the logo pattern in the image is determined according to the corrected contour information; the actual distance between the pulley and the image acquisition device is determined according to the actual area of the logo pattern, the logo image area and the calibration parameter.

[0015] The second aspect of the embodiment of the present application provides a processor configured to execute the hoist overshoot monitoring method.

[0016] The second aspect of the embodiment of the application provides an engineering equipment, comprising: a hoisting mechanism, the hoisting mechanism comprising a boom and a pulley, a head of the boom is provided with an image acquisition device, and an identification pattern is arranged on the pulley and is in a shooting range of the image acquisition device; and the processor.

[0017] The fourth aspect of the embodiment of the application provides a machine readable storage medium, a program or instruction is stored on the machine readable storage medium, and the program or instruction is executed by a processor to implement the hoisting overshoot monitoring method.

[0018] The above technical solution is applied to the engineering equipment, the engineering equipment comprises a hoisting mechanism, the hoisting mechanism comprises a boom and a pulley, a head of the boom is provided with an image acquisition device, and an identification pattern is arranged on the pulley and is in a shooting range of the image acquisition device, an image collected by the image acquisition device is acquired, the image comprises the identification pattern, then based on a pre-constructed neural network model, contour information of the identification pattern in the image is determined according to the image, then actual distance between the pulley and the image acquisition device is determined according to the contour information and predetermined calibration parameters of the image acquisition device, and it is determined that the hoisting mechanism has an overshoot phenomenon in the case that the actual distance is less than a preset safety distance. The application can process the image through the pre-constructed neural network to determine the contour information of the identification pattern in the image, and then the actual distance between the pulley and the image acquisition device is determined as a judgment basis for the hoisting overshoot, which is beneficial to reduce the judgment error and improve the accuracy of the hoisting overshoot monitoring.

[0019] Other features and advantages of the embodiment of the application will be described in detail in the following specific embodiment part. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are included to provide a further understanding of the embodiment of the application, and constitute a part of the specification, and are used to explain the embodiment of the application together with the following specific embodiment, but do not constitute a limitation to the embodiment of the application. In the drawings:

[0021] Figure 1 A flowchart of a hoisting overshoot monitoring method provided by the embodiment of the application is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiment of the application more clear, the technical scheme in the embodiment of the application will be clearly and completely described below in combination with the drawings in the embodiment of the application, and it should be understood that the specific embodiment described here is only used to illustrate and explain the embodiment of the application, and is not used to limit the embodiment of the application. Based on the embodiment in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the application.

[0023] It should be noted that if the application embodiments involve directionality indication (such as up, down, left, right, front, back, etc.), the directionality indication is only used to explain the relative position relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directionality indication will also change accordingly.

[0024] In addition, if the application embodiments involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the application.

[0025] The engineering equipment of the embodiment can be a fuel architecture and a new energy architecture. Taking the engineering equipment as an engineering mechanical vehicle as an example, it can include a traditional engineering mechanical vehicle, and can also include a new energy vehicle applied in the engineering mechanical field, such as a new energy mixer, a new energy pump truck, a new energy excavator, etc. In addition, the engineering mechanical vehicle of the embodiment also belongs to an intelligent networked vehicle. The engineering mechanical vehicle includes a sensing / perception system, a communication system, etc. The vehicle operation data and the vehicle peripheral environment information, etc. are collected through the sensing / perception system in the vehicle, and the network connection with other vehicles and the cloud is realized through the communication system. The collected vehicle operation data, vehicle peripheral environment information, etc. are shared to the cloud and other authorized vehicles, so as to realize data sharing, remote analysis, intelligent driving, etc.

[0026] Figure 1 A flowchart of a hoist overshoot monitoring method provided by the embodiment of the application is shown. As shown in Figure 1 The embodiment of the application provides a hoist overshoot monitoring method, which is applied to an engineering equipment, the engineering equipment includes a hoist mechanism, the hoist mechanism includes a boom and a pulley, the head of the boom is provided with an image acquisition device, and the pulley is provided with a mark pattern, the mark pattern is in the shooting range of the image acquisition device. Taking the processor of the engineering equipment as an example, the method can include the following steps.

[0027] Step S101, acquiring an image collected by the image acquisition device, the image including the mark pattern.

[0028] Step S102, determining the contour information of the mark pattern in the image according to the image based on a pre-constructed neural network model.

[0029] In step S103, the actual distance between the pulley and the image acquisition device is determined according to the contour information and predetermined calibration parameters of the image acquisition device.

[0030] In step S104, if the actual distance is less than the preset safe distance, it is determined that the hoisting mechanism has an overshoot phenomenon.

[0031] In the embodiment of the present application, the image acquisition device includes a shooting unit, a positioning base and a counterweight, the shooting unit is connected with the counterweight, the counterweight is movably connected with the positioning base, the positioning base is arranged below the head of the boom, and the counterweight is used to drive the shooting angle of the shooting unit to be vertically downward. In this way, the shooting unit can always shoot vertically downward, and the identification pattern on the pulley is always within the shooting range of the shooting unit.

[0032] Specifically, in the process of the engineering equipment performing hoisting operation, in order to monitor the hoisting overshoot in real time, the processor can acquire images collected by the image acquisition device in real time, and the images contain the identification pattern on the pulley. Further, the processor can analyze the acquired images by using a pre-constructed neural network model, so as to determine the contour information of the identification pattern in the images. The pre-constructed neural network model can be used to accurately identify and extract the contour information of the specific identification pattern in the images.

[0033] Then, the processor can determine the actual distance between the pulley and the image acquisition device according to the contour information of the identification pattern and the calibration parameters of the image acquisition device, such as focal length and sensor size, by geometric calculation or triangulation principle.

[0034] Finally, the processor can compare the calculated actual distance with a preset safe distance. If the actual distance is less than the preset safe distance, the processor determines that the hoisting mechanism has an overshoot phenomenon, and triggers corresponding alarm or safety measures. The preset safe distance is a safety threshold value set according to the actual structure of the engineering equipment and the application scene. If the actual distance is less than the preset safe distance, it means that the pulley of the hoisting mechanism may hit the boom, and there is a safety risk. At this time, alarm information can be output to enable the operator to take targeted risk-avoiding measures. The alarm information can be output in the form of sound, light and text. In this way, the operator can monitor the running state of the hoisting mechanism in real time, especially when there is a potential danger such as overshoot, the operator can quickly take corresponding measures, so as to ensure the safety and efficiency of the engineering operation.

[0035] The technical scheme is applied to an engineering equipment, the engineering equipment comprises a hoisting mechanism, the hoisting mechanism comprises a boom and a pulley, the head of the boom is provided with an image acquisition device, the pulley is provided with an identification pattern, the identification pattern is in the shooting range of the image acquisition device, the image acquired by the image acquisition device is obtained, the image comprises the identification pattern, then based on a pre-constructed neural network model, the contour information of the identification pattern in the image is determined according to the image, then the actual distance between the pulley and the image acquisition device is determined according to the contour information and the predetermined calibration parameter of the image acquisition device, and it is determined that the hoisting mechanism has an overshoot phenomenon in the case that the actual distance is less than a preset safety distance. The image can be processed by the pre-constructed neural network to determine the contour information of the identification pattern in the image, and then the actual distance between the pulley and the image acquisition device is determined according to the contour information, which is used as a judgment basis for hoisting overshoot, so as to reduce the judgment error and improve the accuracy of hoisting overshoot monitoring.

[0036] In the embodiment of the application, the pre-constructed neural network model comprises a target detection branch, an identification segmentation branch and an accurate segmentation branch, and based on the pre-constructed neural network model, the position information of the identification pattern in the image can be determined according to the image, which can comprise: based on the target detection branch, identifying the region where the identification pattern is located in the image to obtain a local image region containing the identification pattern; based on the identification segmentation branch, segmenting the local image region to obtain an identification region image; and based on the accurate segmentation branch, determining the contour information of the identification pattern according to the identification region image.

[0037] It can be understood that, in order to accurately monitor the running state of the hoisting mechanism, especially to prevent the occurrence of overshoot, the embodiment of the application determines the contour information of the identification pattern in the image by using a pre-constructed neural network model, and the neural network model comprises a target detection branch, an identification segmentation branch and an accurate segmentation branch. Specifically, the target detection branch in the neural network model is used to preliminarily analyze the acquired image. The target detection branch can identify the region where the identification pattern is located in the image and extract a local image region containing the identification pattern, which is beneficial to reducing the calculation amount of subsequent processing and improving the accuracy of identification.

[0038] After obtaining the local image region, the identification segmentation branch in the neural network model is used to further segment the region. The identification segmentation branch can accurately identify the boundary between the identification pattern and the background, so as to obtain a clear identification region image. Then, the accurate segmentation branch in the neural network model is used to finely process the identification region image. The accurate segmentation branch can further refine the contour information of the identification pattern to ensure the accuracy and integrity of the contour. This step is crucial for subsequent calculation of the actual distance between the pulley and the image acquisition device.

[0039] Therefore, the pre-constructed neural network model is used to accurately identify and segment the pulley identification pattern, providing accurate data basis for subsequent calculation of the actual distance between the pulley and the image acquisition device, improving the accuracy and efficiency of monitoring, and providing a strong guarantee for the safe operation of engineering equipment.

[0040] In the embodiment of the present application, based on the accurate segmentation branch, the contour information of the identification pattern is determined according to the identification region image, which can include: according to the accurate segmentation branch, the first contour, the second contour and the third contour of the identification region image are segmented, the first contour is the contour formed by the starting change point of the identification edge value in the identification region image, the second contour is the contour formed by the ending change point of the gradient in the identification region image, and the third contour is the contour formed by the midpoint of the first contour and the second contour; the first contour, the second contour and the third contour are fused to obtain the contour information of the identification pattern.

[0041] It can be understood that, in order to more accurately determine the contour information of the identification pattern, the accurate segmentation branch in the pre-constructed neural network model is used in the embodiment of the present application, which can accurately segment the contour of the identification pattern based on the identification region image, and obtain the final contour information by fusing these contours. Specifically, first, the local image region containing the identification pattern, i.e. the identification region image, is extracted from the original image through the target detection branch and the identification segmentation branch in the neural network model.

[0042] Then, the identification region image is further processed by using the accurate segmentation branch. The branch can identify and segment the first contour, the second contour and the third contour, wherein the first contour is the contour formed by the pixel point where the identification edge value starts to change in the identification region image, corresponding to the preliminary boundary between the identification pattern and the background; the second contour is the contour formed by the pixel point where the gradient changes end in the identification region image, because the ending point of gradient change often corresponds to the sharp turning of the edge, so the second contour can more accurately reflect the boundary of the identification pattern; the third contour is the contour formed by the midpoint of the first contour and the second contour, by calculating the midpoint of the two contours, we can obtain a more smooth and stable third contour, which helps to reduce the influence of noise and abnormal points.

[0043] After obtaining the above three contours, the final identification pattern contour information can be obtained by fusion processing. In one example, the fusion processing includes aligning the contours to ensure that the three contours are spatially aligned, i.e. they correspond to different representations of the same identification pattern; then, the points on each contour are weighted averaged to obtain a comprehensive contour, the weight of the weighted average can be determined according to the reliability or importance of the contour, and finally the fused contour is smoothed to eliminate possible noise or jagged edges to obtain the final contour information.

[0044] Therefore, the contour information provided by the accurate segmentation branch can be used to more accurately determine the contour of the identification pattern, thereby improving the accuracy of subsequent hoist overshoot monitoring and providing a reliable guarantee for the safe operation of the engineering equipment.

[0045] In the embodiment of the present application, before the contour information of the identification pattern is determined according to the identification region image based on the accurate segmentation branch, the identification pattern in the identification region image can also be completed according to the geometric feature information of the identification pattern.

[0046] It can be understood that due to the unstable working environment of the engineering equipment caused by changes in weather, day, night, site lighting, etc., and for various reasons such as occlusion and uneven lighting, the identification pattern in the image may be partially missing or unclear during image acquisition. In order to more accurately determine the contour information of the identification pattern, the identification pattern in the identification region image can be completed before using the accurate segmentation branch. Specifically, since the identification pattern has a specific graphical composition, the geometric feature information of the identification pattern is known, and the geometric feature information includes shape, size, direction, and possible symmetry information of the identification pattern.

[0047] Further, the identification pattern in the identification region image can be completed based on the geometric feature information of the identification pattern. In one example, a complete identification pattern that matches the geometric feature of the identification pattern can be searched in a pre-constructed image library or template library. In another example, an image inpainting algorithm (such as sample-based inpainting, structure-based inpainting, etc.) can be used to fill and repair the missing or unclear part.

[0048] Through such a method, the identification pattern can be completed before the contour information of the identification pattern is determined, thereby improving the accuracy and reliability of contour determination. This not only helps to improve the accuracy of monitoring, but also provides a more solid guarantee for the safe operation of the engineering equipment.

[0049] In the embodiment of the present application, before the local image region is segmented based on the identification segmentation branch to obtain the identification region image, the maximum gray value and the minimum gray value in the local image region can also be determined, the adjustment coefficient can be determined according to the maximum gray value and the minimum gray value, and the pixel enhancement processing can be performed on the local image region according to the adjustment coefficient and the minimum gray value.

[0050] It can be understood that in order to improve the quality of the image and make the subsequent segmentation and contour determination process more accurate and reliable, the pixel enhancement processing can also be performed on the local image region before the local image region is segmented by the identification segmentation branch. Specifically, the maximum gray value and the minimum gray value in the local image region are calculated, and these two values define the range of brightness and darkness in the image.

[0051] According to the maximum gray value and the minimum gray value, a correction factor is determined, which is used to control the intensity of pixel enhancement. The correction factor is proportional to the difference between the maximum gray value and the minimum gray value, that is, the greater the difference, the greater the correction factor, so as to enhance the contrast of the image. Further, according to the correction factor and the minimum gray value, the local image region is subjected to pixel enhancement processing. In an example, the gray values of the local image region can be linearly stretched according to the correction factor and the minimum gray value, so as to expand the contrast range of the local image region, and the gray histogram of the local image region is adjusted, so that the gray value distribution of the local image region is more uniform, thereby improving the contrast of the image.

[0052] In this way, before the local image region is segmented, the pixel enhancement processing is performed on the local image region, so as to improve the quality and contrast of the image. This helps to improve the accuracy of segmentation and provides more reliable image data for subsequent contour determination and actual distance calculation.

[0053] In the embodiments of the present application, the identification pattern includes a first pattern and a second pattern, the first pattern surrounds the second pattern, and the contour information includes an outer contour corresponding to the first pattern and an inner contour corresponding to the second pattern. According to the contour information and the predetermined calibration parameters of the image acquisition device, the actual distance between the pulley and the image acquisition device can be determined, which can include: correcting the size of the outer contour according to the geometric feature information of the first pattern, and correcting the size of the inner contour according to the geometric feature information of the second pattern, to obtain corrected contour information; and determining the actual distance between the pulley and the image acquisition device according to the corrected contour information and the calibration parameters.

[0054] It can be understood that the identification pattern can be composed of a first pattern and a second pattern, wherein the first pattern surrounds the second pattern, and the contour information includes an outer contour corresponding to the first pattern and an inner contour corresponding to the second pattern. In order to accurately determine the actual distance between the pulley and the image acquisition device, the size of the outer contour corresponding to the first pattern and the size of the inner contour corresponding to the second pattern can be corrected respectively. Specifically, the size of the outer contour is corrected according to the geometric feature information of the first pattern, including shape, size, and the ratio of each side, etc. The size of the inner contour is corrected according to the geometric feature information of the second pattern. Then, after obtaining the corrected contour information, the processor can determine the actual distance between the pulley and the image acquisition device according to the corrected contour information and the calibration parameters of the image acquisition device.

[0055] In one example, one or more measurement reference points of the identification pattern can be determined according to the corrected contour information, which can be specific points on the contour such as vertexes and center points, or intersection points of certain features such as straight line segments and circular arcs; then, the measurement reference points are converted from the image coordinate system to the actual coordinate system in combination with the calibration parameters of the image acquisition device including focal length, sensor size and pixel size, and the actual distance between the pulley and the image acquisition device is determined through geometric calculation or triangulation principle by using the converted measurement reference point coordinates.

[0056] In another example, the identification pattern is composed of a rectangular outer contour and a circular inner contour, and the circle is located in the rectangle. For outer contour correction, since the length ratio of the sides of the rectangular outer contour is fixed, the length difference on adjacent sides can be calculated and corresponding stretching adjustment can be performed to correct the stretching deformation of the outer contour, and the included angle between the long side and the short side can be calculated and compared with the included angle of the standard rectangle to perform corresponding rotation adjustment to correct the distortion; for inner contour correction, since the distance between the center of the circular inner contour and the vertex of the right angle side of the rectangle in the outer contour is fixed, this fixed distance relationship can be used to correct the deformation inside the identification, and the area ratio of the circle to the entire identification pattern (i.e. the area of the rectangle) is fixed, so the area ratio of the inner contour to the entire identification area can be calculated and compared with the preset fixed ratio to further correct the deformation inside. In this way, the identification pattern in the identification region image is finally corrected in combination with the correction results of the outer contour and the inner contour to ensure that the shape and size meet the preset standard.

[0057] In this way, the actual distance between the pulley and the image acquisition device is more accurately determined by correcting the size of the outer contour and the inner contour of the identification pattern, which is beneficial to improve the accuracy of the hoist overshoot monitoring.

[0058] In the embodiments of the present application, the actual distance between the pulley and the image acquisition device can be determined according to the corrected contour information and the calibration parameters, which can include: determining the identification image area of the identification pattern in the image according to the corrected contour information; determining the actual distance between the pulley and the image acquisition device according to the actual area of the identification pattern, the identification image area and the calibration parameters.

[0059] Specifically, the processor can further calculate the area of the identification pattern in the image, i.e., the identification image area, according to the corrected contour information, using image processing techniques such as pixel counting and shape analysis. The actual area of the identification pattern and the calibration parameters of the image acquisition device are obtained. The actual area of the identification pattern is known at the time of manufacture or design, which represents the true size of the identification pattern in the physical world. The calibration parameters of the image acquisition device include focal length, sensor size, pixel size, etc. These parameters are inherent to the image acquisition system and are used to convert the size in the image to the actual size. Further, by comparing the actual area of the identification pattern and the identification image area, a scale factor can be calculated, which reflects the proportional relationship between the size in the image and the actual size. Then, using the scale factor and the calibration parameters of the image acquisition device, especially the focal length and the sensor size, the actual distance between the pulley and the image acquisition device can be determined through geometric calculation or triangulation principle. This process needs to consider the geometric layout of the image acquisition system, the angle of view of the camera, and the position of the pulley in the image, etc.

[0060] In this way, by combining the corrected contour information, the actual area of the identification pattern, and the calibration parameters of the image acquisition device, the actual distance between the pulley and the image acquisition device can be accurately determined.

[0061] In a specific embodiment of the present application, the hoist overshoot monitoring method in the above-mentioned implementation is applied to the crane operation as an example. The hoist mechanism of the crane includes a boom, an image acquisition device, a movable pulley, a steel wire, and a hook. The hook is connected to the movable pulley and is lifted and lowered together with the movable pulley through the steel wire wound on the movable pulley. The movable pulley has an identification pattern on the side opposite to the side where the hook is located. The identification pattern is used for distance conversion and includes a rectangular pattern and a plurality of circular patterns located within the rectangular pattern. The image acquisition device is installed at the head of the boom and includes a camera, a fixed seat, and a counterweight. Under the action of the counterweight, the camera is always vertically downward. In the state that the camera is turned on, the movable pulley connected to the hook is also always located vertically below the head of the boom. The image acquisition device always automatically tracks and captures the identification pattern on the movable pulley. During the crane operation, the hoist overshoot monitoring method in the above-mentioned implementation can be used to monitor the hoist overshoot of the crane.

[0062] Specifically, during the operation of the crane, an image of the space hook with the identification pattern is collected in real time by the image collection device, and then the image is input into a pre-constructed neural network model to detect the position of the identification pattern in the image. Further, based on a geometric transformation model from image coordinates to world coordinates established in advance according to the calibration parameters of the camera and the known size of the identification pattern, the pre-constructed geometric transformation model is used to convert the position of the identification pattern in the image into the position in the world coordinate system. According to the position of the identification pattern in the world coordinate system and the position of the image collection device, the distance between the identification pattern and the image collection device is determined. This process involves distance calculation in three-dimensional space, which can be realized using vector operations or geometric formulas, which will not be described here.

[0063] Specifically, to improve the data processing efficiency, the image can be input into a neural network for target detection to detect the area where the hook and the identification pattern are located in the image, and obtain a local area image. In this way, the data processing amount can be reduced, thereby improving the data processing efficiency. Then, considering the image distortion problem, the area image can be preliminarily corrected according to the geometric features of the rectangular pattern in the identification pattern to obtain a preliminarily corrected area image.

[0064] Further, considering that the operation of the engineering equipment is affected by factors such as weather, day, night, on-site lighting, and the like, the operation environment is extremely unstable. To reduce the influence of environmental factors, the preliminarily corrected area image can be subjected to image enhancement processing to enhance the contour edge information of the identification pattern.

[0065] In one example, the contour edge information can be enhanced by scaling the image features, making the black parts in the image darker and the bright parts brighter. First, the adjustment coefficient is determined according to the maximum gray value and the minimum gray value in the current area image, and then the gray value of each pixel point in the area image is scaled according to the adjustment coefficient and the minimum gray value, satisfying the following formula:

[0066] Gt = Gs x (255 / (Gmax- Gmin))- Gmin x (255 / (Gmax- Gmin));

[0067] Wherein, Gt is the transformed gray value, Gs is the gray value of the pixel point in the original area image, Gmax is the maximum gray value in the original area image, and Gmin is the minimum gray value in the original image.

[0068] In another example, a specific edge can be enhanced according to the shape of the identification pattern. In the case of the identification pattern including a rectangular pattern in the specific embodiment, the right-angle edge and the straight line gradient that can be detected in the region image are enhanced. The adjustment value of a pixel point on the right-angle edge is determined according to the square of the gray value of the pixel point itself divided by the gray value of the adjacent pixel point, so as to realize the strengthening of the right-angle edge.

[0069] Further, the region image after enhancement is identified and segmented by using the trained neural network, so as to segment the background of the identification pattern and obtain an identification region image containing only the identification pattern. Then, considering that the camera is installed at the head of the hoisting arm, the hook is connected by a steel wire rope, and the identification pattern can be shielded when the image is collected, which can cause a large error in subsequent distance conversion. Therefore, the shielded part of the identification pattern can be first completed. Since the geometric shape of the outer contour of the identification pattern is regular and fixed rectangular, the identification region image can be completed through geometric relationship. Specifically, the straight lines in the identification region image are first detected, and then the included angles between the detected straight lines are calculated and judged whether they are close to 90 degrees, so as to screen out the straight line pairs that can form right angles. According to the geometric characteristics of the rectangle, when three right angles are detected in the identification region image, the fourth right angle can be completed according to the geometric relationship. After the four right angles of the rectangle are determined, the shielded identification edge is completed according to the position and direction of the right angles. In this way, the identification region image after preliminary completion is obtained.

[0070] Further, in order to accurately determine the area of the identification pattern in the image, the trained neural network can be used to accurately segment the identification contour of the above completed identification region image, so as to obtain the contour information of the identification pattern, including the outer contour of the rectangle and the inner contour of the circle. The input data of the neural network model is the identification region image to be segmented. During training, the first contour formed by the starting point of the identification edge value, the second contour formed by the ending point of the gradient change, and the third contour formed by the midpoint of the first contour and the second contour are respectively used as three different labels to train the neural network model. Then, the trained neural network model is used to predict the above three different contours. The three contour labels have a strong supervisory effect on the model, and the contour segmentation accuracy is improved by 30% compared with single MASK or single contour output accuracy. Therefore, according to the neural network model, the first contour, the second contour and the third contour of the identification region image can be obtained, and then the three contours are fused to obtain the contour information of the identification pattern.

[0071] Further, to ensure the accuracy of the detection result, the identification pattern in the image can be secondarily corrected based on the contour information, including identification outer contour size correction and identification inner contour size correction. Since the outer contour is a rectangle, the length of the side of the rectangle is fixed and unchangeable, thus the distortion of the outer contour can be corrected by the stretching and deforming of the outer contour on the adjacent sides and the angle change of the long and short sides. Since the inner contour is a circle, the distance between the point on the inner contour circle and the corresponding point on the outer contour is fixed, and the area ratio of the circle to the entire identification pattern is fixed, thus the inner deformation of the outer contour can be corrected accordingly, and thus the corrected contour information of the identification pattern is obtained.

[0072] Finally, the area of the identification pattern in the image is determined based on the contour information, and then the actual distance between the pulley and the image acquisition device is determined in combination with the calibration parameters of the shooting unit, including the object distance and the focal length.

[0073] In this way, by using the accurate segmentation method based on the neural network and the correction method based on the geometric relationship, the area of the identification is accurately obtained, thereby improving the accuracy of the measurement, providing a reliable basis for distance conversion and overrun monitoring, and improving the accuracy of the overrun monitoring.

[0074] The embodiment of the present application also provides a processor configured to execute the hoist overrun monitoring method in the embodiment.

[0075] The embodiment of the present application also provides an engineering device, comprising: a hoist mechanism, the hoist mechanism comprising a boom and a pulley, the head of the boom being provided with an image acquisition device, the pulley being provided with an identification pattern, the identification pattern being in the shooting range of the image acquisition device; and the processor in the above embodiment.

[0076] The embodiment of the present application also provides a machine readable storage medium, the machine readable storage medium storing programs or instructions, the programs or instructions being executed by the processor to implement the hoist overrun monitoring method in the above embodiment.

[0077] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0078] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0079] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0080] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0081] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0082] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, for storing instructions and data used and / or generated by the computing device. The memory can also include non-volatile memory, such as read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other non-volatile memory.

[0083] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0084] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0085] The above only is an embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for monitoring winch overshoot, characterized in that: Applied to engineering equipment, the engineering equipment includes a hoisting mechanism, the hoisting mechanism includes a boom and a pulley, an image acquisition device is installed on the head of the boom, and an identification pattern is provided on the pulley, and the identification pattern is within the shooting range of the image acquisition device. The method includes: Acquiring an image captured by the image acquisition device, wherein the image includes the identification pattern; Determining, based on the image and based on the pre-built neural network model, contour information of the identification pattern in the image; determining an actual distance between the pulley and the image acquisition device according to the profile information and predetermined calibration parameters of the image acquisition device; When the actual distance is less than the preset safety distance, it is determined that an overshoot phenomenon occurs in the hoisting mechanism.

2. The method according to claim 1, characterized in that The pre-built neural network model includes a target detection branch, a marker segmentation branch, and a precise segmentation branch. The method of determining the position information of the marker pattern in the image based on the image based on the pre-built neural network model includes: Based on the target detection branch, identifying the area where the identification pattern is located in the image to obtain a local image area containing the identification pattern; Segmenting the local image region based on the identification segmentation branch to obtain an identification region image; Based on the precise segmentation branch, contour information of the identification pattern is determined according to the identification area image.

3. The method according to claim 2, characterized in that The determining, based on the precise segmentation branch and according to the identification area image, the contour information of the identification pattern includes: Segmenting the marked area image according to the precise segmentation branch to obtain a first contour, a second contour, and a third contour, wherein the first contour is formed by a point where a marked edge value in the marked area image starts to change, the second contour is formed by a point where a gradient change in the marked area image ends, and the third contour is formed by a midpoint between the first contour and the second contour; The first outline, the second outline, and the third outline are fused to obtain outline information of the identification pattern.

4. The method according to claim 2, characterized in that Before determining the contour information of the identification pattern according to the identification area image based on the precise segmentation branch, the method further includes: The identification pattern in the identification area image is completed according to the geometric feature information of the identification pattern.

5. The method according to claim 2, characterized in that Before segmenting the local image region based on the marker segmentation branch to obtain the marker region image, the method further includes: Determine the maximum grayscale value and the minimum grayscale value in the local image area; determining an adjustment coefficient according to the maximum grayscale value and the minimum grayscale value; Perform pixel enhancement processing on the local image area according to the adjustment coefficient and the minimum grayscale value.

6. The method according to claim 1, characterized in that The identification pattern includes a first graphic and a second graphic, the first graphic surrounds the second graphic, and the outline information includes an outer outline corresponding to the first graphic and an inner outline corresponding to the second graphic; Determining the actual distance between the pulley and the image acquisition device according to the contour information and predetermined calibration parameters of the image acquisition device includes: Correcting the outer contour size of the outer contour according to the geometric feature information of the first figure, and correcting the inner contour size of the inner contour according to the geometric feature information of the second figure, so as to obtain corrected contour information; The actual distance between the pulley and the image acquisition device is determined according to the corrected contour information and the calibration parameters.

7. The method according to claim 6, characterized in that The determining the actual distance between the pulley and the image acquisition device according to the corrected contour information and the calibration parameters includes: determining a marking image area of ​​the marking pattern in the image according to the corrected contour information; The actual distance between the pulley and the image acquisition device is determined according to the actual area of ​​the identification pattern, the area of ​​the identification image and the calibration parameters.

8. A processor, characterized in that: The device is configured to perform the method for monitoring winch overshoot according to any one of claims 1 to 7.

9. An engineering equipment, characterized in that: include: A hoisting mechanism, the hoisting mechanism comprising a boom and a pulley, an image acquisition device being mounted on the head of the boom, and an identification pattern being provided on the pulley, the identification pattern being within the shooting range of the image acquisition device; The processor according to claim 8.

10. A machine-readable storage medium storing a program or instruction, characterized in that: When the program or the instruction is executed by a processor, the method for monitoring winch overshoot according to any one of claims 1 to 7 is implemented.

Images