Method, device, storage medium and processor for determining hook position

Abstract

Embodiments of the present application provide a method, device, storage medium and processor for determining the position of a hook. The method comprises: obtaining a real-time initial image; inputting the initial image into a pre-trained neural network to output a first region of interest by the pre-trained neural network, wherein the pre-trained neural network is used to extract features of the initial image to obtain a plurality of feature maps of different scales, fuse the plurality of feature maps of different scales to extract a positioning feature of the hook, and determine the first region of interest in the initial image according to the positioning feature; enlarge the first region of interest in the initial image to obtain a second region of interest; and in the case that the second region of interest includes a winch rope feature, determine the position of the hook according to the second region of interest. The above technical solution can more accurately determine the position of the hook, improve the intelligent level of the device, reduce the operation risk, and reduce the labor cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, apparatus, storage medium, and processor for determining the position of a hook. Background Technology

[0002] With the development of modular construction and intelligent technology, users' demands for intelligent and automated lifting operations of crane products are gradually upgrading, especially in automated lifting operations with hazardous lifting scenarios or repetitive lifting requirements, where precise lifting of crane products is of paramount importance. Therefore, to improve the intelligence, operational safety, and key performance of crane products, it is essential to achieve precise positioning during lifting.

[0003] For example, in the automated lifting process of crawler cranes, obtaining the spatial position of the hook is a key part of achieving precise lifting positioning. Traditionally, the spatial position of the hook is calculated by calibrating the winch rope. This method has significant individual variability, meaning that different cranes require recalibration. Furthermore, when the crane has large deflections, the error increases sharply, making its adaptability very limited. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, storage medium, and processor for determining the position of a hook.

[0005] To achieve the above objectives, a first aspect of this application provides a method for determining the position of a hook, comprising:

[0006] Acquire the initial image in real time;

[0007] An initial image is input into a pre-trained neural network to output a first region of interest. The pre-trained neural network is used to extract features from the initial image to obtain multiple feature maps of different scales. The feature maps of multiple different scales are fused to extract the positioning features of the hook, and the first region of interest is determined in the initial image based on the positioning features.

[0008] Expand the region of first interest to obtain the region of second interest;

[0009] Expand the first region of interest in the initial image to obtain the second region of interest;

[0010] If the second region of interest includes the characteristics of the winch rope, the position of the hook is determined based on the second region of interest.

[0011] In this embodiment of the application, after obtaining the second region of interest, the method further includes: performing grayscale processing on the second region of interest to obtain a grayscale image; performing gradient enhancement processing on the grayscale image to obtain a gradient-enhanced image; performing binarization processing on the gradient-enhanced image to obtain a binarized image; and determining whether the second region of interest includes hoisting rope features based on the binarized image.

[0012] In this embodiment of the application, after performing grayscale processing on the second region of interest to obtain a grayscale image, the method further includes: traversing the grayscale image to obtain a grayscale histogram of the grayscale image, the grayscale histogram including the number of pixels for each grayscale point; accumulating the number of pixels corresponding to each grayscale point in ascending order of grayscale value, and determining the grayscale value of the corresponding grayscale point as a first threshold when the accumulated number of pixels is greater than or equal to N for the first time; accumulating the number of pixels corresponding to each grayscale point in descending order of grayscale value, and determining the grayscale value of the corresponding grayscale point as a second threshold when the accumulated number of pixels is greater than or equal to M for the first time; wherein the first threshold and the second threshold are used to perform gradient enhancement processing on the grayscale image, and / or to perform binarization processing on the gradient-enhanced image.

[0013] In this embodiment of the application, a gradient enhancement process is performed on a grayscale image to obtain a gradient-enhanced image, including: performing gradient enhancement processing on the grayscale image according to the following formula to obtain the gradient-enhanced image:

[0014]

[0015] Among them, v in It is the gray value of any pixel in a grayscale image, v out t1 is the grayscale value of the pixel after gradient enhancement processing, 'a' is a preset coefficient, 'r1', 'r2', and 'r3' are concavity and convexity coefficients, and 'r1' > 'r2' > ​​'r3'. t1 is the first threshold, and t2 is the second threshold.

[0016] In this embodiment of the application, binarizing the gradient enhancement image to obtain a binarized image includes: determining the average of a first threshold and a second threshold as a binarization threshold; and binarizing the gradient enhancement image according to the binarization threshold to obtain a binarized image.

[0017] In this embodiment, N is determined based on the total number of pixels in the grayscale image and a first preset ratio value, and M is determined based on the total number of pixels and a second preset ratio value. The first preset ratio value and the second preset ratio value are determined based on the magnification factor of the first region of interest.

[0018] In this embodiment of the application, determining whether a second region of interest includes hoisting rope features based on a binarized image includes: adjusting the grayscale value of the image region corresponding to the first region of interest in the binarized image to the maximum value to obtain a blank background image; detecting straight lines in the blank background image; generating a first arc and a second arc in the blank background image using the center point of the image region corresponding to the first region of interest as a circle and employing a first preset radius and a second preset radius respectively; and determining that the second region of interest includes hoisting rope features if the number of intersections between the first arc and the straight line is the same as the number of intersections between the second arc and the straight line.

[0019] In this embodiment of the application, when the second region of interest includes the features of the winch rope, determining the position of the hook based on the second region of interest includes: when there are multiple target regions, determining the distance between the center point of each target region and the center point of the initial image, wherein the target region is the second region of interest including the features of the winch rope; and determining the position of the hook based on the target region with the minimum distance.

[0020] In this embodiment, the method further includes a neural network training step, which includes: acquiring a sample image of the hook; inputting the sample image into the neural network to output a predicted region in the sample image containing the hook, wherein the neural network is used to extract features from the sample image to obtain multiple sample feature maps of different scales, fusing the multiple sample feature maps of different scales to extract the sample location features of the hook, and determining the predicted region in the sample image where the hook is located based on the sample location features; determining the loss value between the predicted region and the target marker region corresponding to the sample image through the SIou loss function; and obtaining the trained neural network when the loss value reaches a preset threshold.

[0021] A second aspect of this application provides a processor configured to perform the above-described method for determining the position of a hook.

[0022] A third aspect of this application provides a crane, comprising:

[0023] Hook;

[0024] The boom, equipped with image acquisition equipment, is used to capture initial images of the hook in real time during the lifting process; and

[0025] The aforementioned processor.

[0026] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned method for determining the position of a hook.

[0027] The above technical solution acquires a real-time initial image; a first region of interest (ROI) is output through a pre-trained neural network, where the pre-trained neural network is used to extract features from the initial image, obtaining multiple feature maps of different scales. These feature maps are then fused to extract the hook's positioning features, and the first ROI is determined in the initial image based on these features. The first ROI is then expanded in the initial image to obtain a second ROI. If the second ROI includes features of the winch rope, the hook position is determined based on the second ROI. This technical solution enables more accurate hook positioning, improves equipment intelligence, reduces operational risks, and lowers labor costs.

[0028] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0030] Figure 1 This illustration schematically shows a flowchart of a method for determining the position of a hook according to an embodiment of this application;

[0031] Figure 2 This illustration schematically shows another process diagram of a method for determining the position of a hook according to an embodiment of this application;

[0032] Figure 3 A schematic diagram of a crane according to an embodiment of this application is shown.

[0033] Figure 4 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] Figure 1 This illustration schematically shows a flowchart of a method for determining the position of a hook according to an embodiment of this application. Figure 1 As shown, in one embodiment of this application, a method for determining the position of a hook is provided, comprising the following steps:

[0036] Step 101: Obtain the initial image in real time.

[0037] Step 102: Input the initial image into a pre-trained neural network to output a first region of interest through the pre-trained neural network. The pre-trained neural network is used to extract features from the initial image to obtain multiple feature maps of different scales. The multiple feature maps of different scales are fused to extract the positioning features of the hook, and the first region of interest is determined in the initial image based on the positioning features.

[0038] Step 103: Expand the first region of interest in the initial image to obtain the second region of interest.

[0039] Step 104: If the second region of interest includes the features of the winch rope, determine the position of the hook based on the second region of interest.

[0040] A hook is one of the most common lifting devices in hoisting machinery, usually suspended from the wire rope of a hoisting mechanism by means of pulley blocks and other components. A neural network is an algorithmic model that mimics the behavioral characteristics of animal neural networks, performing distributed parallel information processing. The processor can acquire a real-time initial image and input it into a pre-trained neural network. The pre-trained neural network can process the input initial image to output a first region of interest (ROI). Specifically, the pre-trained neural network can extract features from the input initial image to obtain multiple feature maps of different scales. These feature maps can be fused to extract the hook's positioning features, and the first ROI can be determined in the initial image based on these positioning features. Feature extraction refers to the methods and processes for extracting characteristic information from an image. Positioning features refer to features that can determine the possible location of the hook in the initial image. The processor can expand the first ROI in the initial image to obtain a second ROI. In other words, the pre-trained neural network can obtain the first ROI in the initial image. The processor can expand the first ROI in the initial image to obtain a second ROI. If the second ROI includes features of the winch rope, the processor can determine the position of the hook based on the second ROI.

[0041] For example, the processor can acquire an initial image A in real time. The initial image A is input into a pre-trained neural network to extract features from it, resulting in multiple feature maps at different scales. Assume four feature maps of different scales are obtained: A1, A2, A3, and A4. These four feature maps are then fused to extract the hook's positioning features. Based on these positioning features, a first region of interest (ROI) is determined in the initial image A. The processor then expands the first ROI in the initial image A to obtain a second ROI. If the second ROI includes features of the winch rope, the hook's position can be determined based on the second ROI.

[0042] In one embodiment, after obtaining the second region of interest (ROI), the processor can perform grayscale processing on the ROI to obtain a grayscale image; perform gradient enhancement processing on the grayscale image to obtain a gradient-enhanced image; perform binarization processing on the gradient-enhanced image to obtain a binarized image; and determine whether the ROI includes hoisting rope features based on the binarized image. Here, grayscale processing refers to the process of converting a color image into a grayscale image. Specifically, after obtaining the second ROI, the ROI can be extracted from the initial image to obtain a corresponding candidate image, and then the candidate image can be grayscale processed to obtain a grayscale image. Gradient enhancement processing refers to processing the image to make the processed image more suitable for the specific application than the original image. Binarization processing refers to setting the grayscale value of the pixels in the image to 0 or 255, that is, making the entire image present a visual effect of only black and white.

[0043] For example, after obtaining the second region of interest, the processor extracts the second region of interest from the initial image A to obtain a candidate image a1. The candidate image a1 is then processed into grayscale to obtain a grayscale image a2. Gradient enhancement processing is then applied to the grayscale image a2 to obtain a gradient-enhanced image a3. The gradient-enhanced image a3 is then binarized to obtain a binarized image a4. Based on the binarized image a5, it is determined whether the second region of interest includes hoisting rope features.

[0044] In one embodiment, after performing grayscale processing on the second region of interest to obtain a grayscale image, the processor can further traverse the grayscale image to obtain a grayscale histogram, which includes the number of pixels for each grayscale point. The number of pixels corresponding to each grayscale point is accumulated in ascending order of grayscale values. When the accumulated number of pixels is first greater than or equal to N, the grayscale value of the corresponding pixel is determined as a first threshold. Similarly, the number of pixels corresponding to each grayscale point is accumulated in descending order of grayscale values. When the accumulated number of pixels is first greater than or equal to M, the grayscale value of the corresponding pixel is determined as a second threshold. The first and second thresholds are used for gradient enhancement processing of the grayscale image, and / or for binarization processing of the gradient-enhanced image. The grayscale histogram is a function of the grayscale distribution and represents a statistical representation of the grayscale distribution in the image. A grayscale histogram is a statistical representation of the frequency of occurrence of all pixels in a digital image according to their grayscale values. It can represent the number of pixels with a certain grayscale level in the image and reflect the frequency of occurrence of that grayscale level in the image.

[0045] For example, after performing grayscale processing on the second region of interest to obtain a grayscale image a1, the processor can further iterate through the grayscale image a1 to obtain its grayscale histogram. The number of pixels at grayscale points t1, t2, t3, t4, and t5 in the grayscale histogram are 2, 7, 9, 25, and 29, respectively. The grayscale values ​​of pixels t1, t2, t3, t4, and t5 are 15, 17, 54, 64, and 77, respectively. The number of pixels corresponding to each grayscale point t1, t2, t3, t4, and t5 is accumulated in ascending order of grayscale value (t1, t2, t3, t4, t5). Assuming that the number of pixels is greater than or equal to N for the first time when accumulating to pixel grayscale point t3, the grayscale value 54 corresponding to pixel grayscale point t3 is determined as the first threshold. Following the order of grayscale values ​​from largest to smallest (t5, t4, t3, t2, t1), the number of pixels corresponding to grayscale points t5, t4, t3, t2, and t1 is accumulated. Assuming that the number of pixels is greater than or equal to M for the first time when accumulating to grayscale point t4, the grayscale value 64 of pixel t4 is determined as the second threshold. The first threshold 54 and the second threshold 64 are used for gradient enhancement processing of the grayscale image a1, and / or for binarization processing of the gradient-enhanced image.

[0046] In one embodiment, N is determined based on the total number of pixels in the grayscale image and a first preset ratio value, and M is determined based on the total number of pixels and a second preset ratio value. The first and second preset ratio values ​​are determined based on the magnification factor of the first region of interest. Assuming the magnification factor of the first region of interest is 7 times, the first preset ratio value can be 1 / 8, the second preset ratio value is 6 / 8, and the remaining 1 / 8 can be the grayscale transition region. The number of pixels at grayscale points t1, t2, t3, t4, and t5 in the grayscale histogram are 2, 7, 9, 25, and 29, respectively. The total number of pixels in the grayscale histogram is 72. That is, N can be the product of the total number of pixels 72 and the first preset ratio value 1 / 8, which is 9; M can be the product of the total number of pixels 72 and the second preset ratio value 6 / 8, which is 54. The grayscale values ​​of pixels t1, t2, t3, t4, and t5 are 15, 17, 54, 64, and 77, respectively. Following the order of grayscale values ​​from smallest to largest (t1, t2, t3, t4, t5), the number of pixels corresponding to grayscale points t1, t2, t3, t4, and t5 is accumulated. After accumulating the number of pixels for t2, the accumulated number of pixels is 9, which equals the value of N, 9. The grayscale value of t2, 17, is determined as the first threshold. Following the order of grayscale values ​​from largest to smallest (t5, t4, t3, t2, t1), the number of pixels corresponding to grayscale points t5, t4, t3, t2, and t1 is accumulated. After accumulating the number of pixels for t4, the accumulated number of pixels is 54, which equals the value of M, 54. The grayscale value of t4, 64, is determined as the second threshold.

[0047] In one embodiment, performing gradient enhancement processing on a grayscale image to obtain a gradient-enhanced image includes: performing gradient enhancement processing on the grayscale image according to the following formula to obtain the gradient-enhanced image:

[0048]

[0049] Among them, v in It is the gray value of any pixel in a grayscale image, v out t1 is the grayscale value of the pixel after gradient enhancement processing, 'a' is a preset coefficient, 'r1', 'r2', and 'r3' are concavity and convexity coefficients, and 'r1' > 'r2' > ​​'r3'. t1 is the first threshold, and t2 is the second threshold.

[0050] In one embodiment, the processor can determine the binarization threshold as the average of a first threshold and a second threshold; the gradient enhancement image is then binarized according to the binarization threshold to obtain a binarized image. For example, the first threshold is 17 and the second threshold is 54. The processor can determine the binarization threshold as the average of the first threshold 17 and the second threshold 54, which is 35.5. The gradient enhancement image is then binarized according to the binarization threshold 35.5 to obtain a binarized image.

[0051] In one embodiment, the processor can adjust the grayscale value of the image region corresponding to the first region of interest in the binarized image to its maximum value to obtain a blank background image; detect straight lines in the blank background image; generate a first arc and a second arc in the blank background image using the center point of the image region corresponding to the first region of interest as a circle and employing a first preset radius and a second preset radius respectively; if the number of intersections between the first arc and the straight line is the same as the number of intersections between the second arc and the straight line, determine that the second region of interest includes hoisting rope features. A template covering method can be used to cover the image region corresponding to the first region of interest in the binarized image to obtain the blank background image.

[0052] For example, the processor can adjust the grayscale values ​​of the image region corresponding to the first region of interest in the binarized image to the maximum value to obtain a blank background image. Straight lines are detected in the blank background image. Using the center point of the image region corresponding to the first region of interest as a circle, and 1.5 times the side length h of the first region of interest as a first preset radius, i.e., 1.5h, a first arc is generated in the blank background image with radius 1.5h. Using 2.5 times the side length h of the first region of interest as a second preset boundary, i.e., 2.5h, a second arc is generated in the blank background image with radius 2.5h. If the number of intersections between the first arc and the straight line is the same as the number of intersections between the second arc and the straight line, it can be determined that the second region of interest includes hoisting rope features.

[0053] In one embodiment, when multiple target regions exist, the processor can determine the distance between the center point of each target region and the center point of the initial image. The target region is a second region of interest that includes the features of the winch rope. The position of the hook is determined based on the target region with the minimum distance. The distance between the center point of each target region and the center point of the initial image can be Euclidean distance, which refers to the true distance between two points in m-dimensional space, or the natural length of a vector (i.e., the distance from the point to the origin).

[0054] For example, when there are two target regions, the processor calculates the Euclidean distance x1 between the center point of target region 1 and the center point of the initial image, and the Euclidean distance x2 between the center point of target region 2 and the center point of the initial image, respectively. If x1 > x2, the position of the hook is determined based on target region 2 corresponding to x2; if x2 > x1, the position of the hook is determined based on target region 1 corresponding to x1; if x2 = x1, the position of the hook can be determined based on either target region 1 corresponding to x1 or target region 2 corresponding to x2.

[0055] In one embodiment, the processor can train the neural network. The specific training steps are as follows: acquiring a sample image of the hook; inputting the sample image into the neural network to output a predicted region containing the hook in the sample image; wherein the neural network is used to extract features from the sample image, obtaining multiple sample feature maps of different scales; fusing the multiple sample feature maps of different scales to extract the sample location features of the hook; and determining the predicted region where the hook is located in the sample image based on the sample location features; determining the loss value between the predicted region and the corresponding target label region in the sample image using the SIou loss function; and obtaining the trained neural network when the loss value reaches a preset threshold. The SIou function can improve the accuracy of orientation matching in the initial detection image and improve the accuracy of determining the hook's position.

[0056] For example, the SIou function mainly includes four cost functions: Angle loss, Distance loss, Shape loss, and IOU loss. The neural network can be a convolutional neural network (CNN), a type of feedforward neural network that includes convolutional computations and has a deep structure; it is one of the representative algorithms of deep learning. It consists of an input layer, hidden layers, and an output layer. CNNs have representation learning capabilities and can perform translation-invariant classification of input information according to their hierarchical structure. The detection anchor box size of a CNN can be determined by iteratively applying the ISODATA dynamic clustering algorithm to the objective function and obtaining its optimal value. The processor can use the ISODATA dynamic clustering algorithm to iteratively apply the objective function and obtain its optimal value, thereby determining the detection anchor box size of the CNN. Based on the size of the detection anchor box, the CNN can extract features from the sample image, obtaining multiple sample feature maps at different scales. Suppose we obtain four sample feature maps at different scales, which could be 64*64, 32*32, 16*16, and 8*8 respectively. We fuse these four feature maps to extract the hook's localization features and determine the predicted region of the hook in the sample image based on these features. The SIou loss function is used to determine the loss value between the predicted region and the corresponding target label region in the sample image. When the loss value reaches a preset threshold, the trained neural network is considered complete.

[0057] Figure 2 This schematically illustrates another flow diagram of a method for determining the position of a hook according to an embodiment of this application. For example... Figure 2As shown, a camera is mounted on the top of the crane's boom. During hoisting, the camera's height above the ground is acquired, and the camera's focal length is dynamically controlled based on this height, ensuring that the camera can acquire real-time initial images even during high-altitude operations. The acquired initial image is loaded and preprocessed to reduce its size to 512*512. An offline-trained hook detection model (i.e., a neural network) is invoked to process the 512*512 initial image, obtaining candidate hook regions (i.e., the first region of interest). A coordinate inverse preprocessing operation is performed to obtain the original image coordinates of the candidate hook regions. This coordinate inverse preprocessing operation includes using the SIOU LOSS function as a loss function to improve the accuracy of direction matching in the initial image detection, using the ISODATA dynamic clustering algorithm to iterate the objective function, and determining the detection frame size of the detection model based on the optimal value of the objective function. The scale of the candidate hook region image is determined based on the frame size. The candidate hook region image is enlarged, and the enlarged candidate hook region image (i.e., the second region of interest) is then processed into a grayscale image. A stepped image enhancement algorithm is used to process the enlarged candidate hook region image to improve the contrast between the hook region and the background region. Gradient enhancement processing is applied to the grayscale image according to the following formula to obtain the gradient-enhanced image:

[0058]

[0059] Among them, v in It is the gray value of any pixel in a grayscale image, v out t1 is the grayscale value of the pixel after gradient enhancement processing, 'a' is a preset coefficient, 'r1', 'r2', and 'r3' are concavity and convexity coefficients, and 'r1' > 'r2' > ​​'r3'. t1 is the first threshold, and t2 is the second threshold.

[0060] The grayscale image is traversed to obtain its grayscale histogram, which includes the number of pixels at each grayscale point. The number of pixels at each grayscale point is counted to determine the corresponding grayscale point. The binarization threshold is then determined based on the grayscale value of the corresponding grayscale point. The candidate initial image is enlarged by a factor of 6. The number of pixels corresponding to each grayscale point is accumulated in ascending order of grayscale value. When the number of pixels at the determined grayscale point is 1 / 7 of the total number of pixels, the grayscale value corresponding to the current pixel is determined as grayscale threshold t1 (i.e., the first threshold). The number of pixels corresponding to each grayscale point is then accumulated in descending order of grayscale value. When the number of pixels at the determined grayscale point is 5 / 7 of the total number of pixels, the grayscale value corresponding to the current pixel is determined as grayscale threshold t2 (i.e., the second threshold). The average of grayscale thresholds t1 and t2 is used as the binarization threshold. The gradient enhancement image is binarized according to the binarization threshold to obtain the binarized image of the ROI region.

[0061] A progressive circular arc intersection method is used to determine whether the processed ROI (i.e., the second region of interest) includes a hoisting rope. First, the grayscale value of the original detection region (i.e., the initial image) of the enlarged candidate hook region is set to zero to determine if the processed ROI contains straight lines. Then, using the center point of the circumscribed square of the original detection region in the processed ROI as the center, a first arc (i.e., the first arc line) is drawn with a first radius of 1.5 times the side length of the circumscribed square (i.e., the first preset radius), and a second arc (i.e., the second arc line) is drawn with a second radius of 2.5 times the side length of the circumscribed square (i.e., the second preset radius). If the number of intersections between the first arc and the first radius is the same as the number of intersections between the second arc and the second radius, the processed ROI is determined to contain a hoisting rope, and the processed ROI is identified as the hook region.

[0062] When there are multiple Regions of Interest (ROIs) containing the winch rope, determine the Euclidean distance between the center point of each ROI and the center point of the hook region image. The ROI with the smallest Euclidean distance is determined as the final hook region. The hook position is then determined based on the final hook region.

[0063] The above technical solution acquires a real-time initial image; a first region of interest (ROI) is output through a pre-trained neural network, where the pre-trained neural network is used to extract features from the initial image, obtaining multiple feature maps of different scales. These feature maps are then fused to extract the hook's positioning features, and the first ROI is determined in the initial image based on these features. The first ROI is then expanded in the initial image to obtain a second ROI. If the second ROI includes features of the winch rope, the hook position is determined based on the second ROI. This technical solution enables more accurate hook positioning, improves equipment intelligence, reduces operational risks, and lowers labor costs.

[0064] Figure 1 , 2 This is a flowchart illustrating a method for determining the position of a hook in one embodiment. It should be understood that, although... Figure 1 , 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 , 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0065] In one embodiment, such as Figure 3 As shown, a crane 300 is provided, comprising:

[0066] Hook 301;

[0067] The boom 302 is equipped with an image acquisition device for real-time acquisition of initial images of the hook 301 during the hoisting process; and

[0068] Processor 303.

[0069] This application provides a processor for running a program, wherein the program executes the above-described method for determining the position of a hook.

[0070] This application provides a storage medium storing a program that, when executed by a processor, implements the method described above for determining the position of a hook.

[0071] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores initial image and area image data. The network interface A02 communicates with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a method for determining the position of a hook.

[0072] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0073] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring a real-time initial image; inputting the initial image into a pre-trained neural network to output a first region of interest (ROI), wherein the pre-trained neural network is used to extract features from the initial image to obtain multiple feature maps of different scales, fusing the multiple feature maps of different scales to extract the positioning features of the hook, and determining the first ROI in the initial image based on the positioning features; expanding the first ROI in the initial image to obtain a second ROI; and determining the hook position based on the second ROI if the second ROI includes features of the winch rope.

[0074] In one embodiment, after obtaining the second region of interest, the method further includes: performing grayscale processing on the second region of interest to obtain a grayscale image; performing gradient enhancement processing on the grayscale image to obtain a gradient-enhanced image; performing binarization processing on the gradient-enhanced image to obtain a binarized image; and determining, based on the binarized image, whether the second region of interest includes hoisting rope features.

[0075] In one embodiment, after performing grayscale processing on the second region of interest to obtain a grayscale image, the method further includes: traversing the grayscale image to obtain a grayscale histogram of the grayscale image, the grayscale histogram including the number of pixels at each grayscale point; accumulating the number of pixels corresponding to each grayscale point in ascending order of grayscale value, and determining the grayscale value of the corresponding grayscale point as a first threshold when the accumulated number of pixels is greater than or equal to N for the first time; accumulating the number of pixels corresponding to each grayscale point in descending order of grayscale value, and determining the grayscale value of the corresponding grayscale point as a second threshold when the accumulated number of pixels is greater than or equal to M for the first time; wherein the first threshold and the second threshold are used to perform gradient enhancement processing on the grayscale image, and / or to perform binarization processing on the gradient-enhanced image.

[0076] In one embodiment, performing gradient enhancement processing on a grayscale image to obtain a gradient-enhanced image includes: performing gradient enhancement processing on the grayscale image according to the following formula to obtain the gradient-enhanced image:

[0077]

[0078] Among them, v in It is the gray value of any pixel in a grayscale image, v out t1 is the grayscale value of the pixel after gradient enhancement processing, 'a' is a preset coefficient, 'r1', 'r2', and 'r3' are concavity and convexity coefficients, and 'r1' > 'r2' > ​​'r3'. t1 is the first threshold, and t2 is the second threshold.

[0079] In one embodiment, binarizing a gradient enhancement image to obtain a binarized image includes: determining the average of a first threshold and a second threshold as a binarization threshold; and binarizing the gradient enhancement image according to the binarization threshold to obtain a binarized image.

[0080] In one embodiment, N is determined based on the total number of pixels in the grayscale image and a first preset ratio value, and M is determined based on the total number of pixels and a second preset ratio value. The first preset ratio value and the second preset ratio value are determined based on the magnification factor of the first region of interest.

[0081] In one embodiment, determining whether a second region of interest includes hoisting rope features based on a binarized image includes: adjusting the grayscale value of the image region corresponding to the first region of interest in the binarized image to the maximum value to obtain a blank background image; detecting straight lines in the blank background image; generating a first arc and a second arc in the blank background image using the center point of the image region corresponding to the first region of interest as a circle and employing a first preset radius and a second preset radius respectively; determining that the second region of interest includes hoisting rope features if the number of intersections between the first arc and the straight line is the same as the number of intersections between the second arc and the straight line.

[0082] In one embodiment, when the second region of interest includes the hoisting rope feature, determining the hook position based on the second region of interest includes: if there are multiple target regions, determining the distance between the center point of each target region and the center point of the initial image, wherein the target region is the second region of interest including the hoisting rope feature; and determining the hook position based on the target region corresponding to the minimum distance.

[0083] In one embodiment, the method further includes a neural network training step, which includes: acquiring a sample image of the hook; inputting the sample image into the neural network to output a predicted region in the sample image containing the hook, wherein the neural network is used to extract features from the sample image to obtain multiple sample feature maps of different scales, fusing the multiple sample feature maps of different scales to extract the sample location features of the hook, and determining the predicted region where the hook is located in the initial image based on the sample location features; determining the loss value between the predicted region and the target label region corresponding to the sample image through the SIou loss function; and obtaining the trained neural network when the loss value reaches a preset threshold.

[0084] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes a method for determining the position of a hook.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0090] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0091] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0093] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining the position of a lifting hook, characterized in that, The method includes: Acquire the initial image in real time; The initial image is input into a pre-trained neural network to output a first region of interest. The pre-trained neural network is used to extract features from the initial image to obtain multiple feature maps of different scales. The multiple feature maps of different scales are fused to extract the positioning features of the hook, and the first region of interest is determined in the initial image based on the positioning features. The first region of interest is expanded in the initial image to obtain a second region of interest; The second region of interest is processed to obtain a grayscale image; The grayscale image is subjected to gradient enhancement processing to obtain a gradient-enhanced image; The gradient-enhanced image is binarized to obtain a binarized image; Based on the binarized image, determine whether the second region of interest includes hoisting rope features; If the second region of interest includes the features of the winch rope, the hook position is determined based on the second region of interest; The method further includes, after performing grayscale processing on the second region of interest to obtain a grayscale image: Traverse the grayscale image to obtain a grayscale histogram of the grayscale image, wherein the grayscale histogram includes the number of pixels for each grayscale point; The number of pixels corresponding to each pixel grayscale point is accumulated in ascending order of grayscale value. When the accumulated number of pixels is greater than or equal to N for the first time, the grayscale value of the corresponding pixel grayscale point is determined as the first threshold. The number of pixels corresponding to each pixel grayscale point is accumulated in descending order of grayscale value. When the accumulated number of pixels is greater than or equal to M for the first time, the grayscale value of the corresponding pixel grayscale point is determined as the second threshold. Wherein, the first threshold and the second threshold are used to perform gradient enhancement processing on the grayscale image, and / or to perform binarization processing on the gradient-enhanced image.

2. The method for determining the position of a hook according to claim 1, characterized in that, The step of performing gradient enhancement processing on the grayscale image to obtain a gradient-enhanced image includes: The grayscale image is then subjected to gradient enhancement processing according to the following formula to obtain a gradient-enhanced image: in, v in It is the gray value of any pixel in a grayscale image. v out It is the grayscale value of the pixel after gradient enhancement processing. a These are preset coefficients. r 1. r 2. r 3 is the concavity / convexity coefficient, and r 1> r 2> r 3, t 1 is the first threshold. t 2 is the second threshold.

3. The method for determining the position of a hook according to claim 1, characterized in that, The step of binarizing the gradient-enhanced image to obtain a binarized image includes: The average of the first threshold and the second threshold is determined as the binarization threshold; The gradient enhancement image is binarized according to the binarization threshold to obtain a binarized image.

4. The method for determining the position of a hook according to claim 1, characterized in that, N is determined based on the total number of pixels in the grayscale image and a first preset ratio value, and M is determined based on the total number of pixels and a second preset ratio value. The first preset ratio value and the second preset ratio value are determined based on the magnification factor of the first region of interest.

5. The method for determining the position of a hook according to claim 1, characterized in that, The step of determining whether the second region of interest includes hoisting rope features based on the binarized image includes: The grayscale value of the image region corresponding to the first region of interest in the binarized image is adjusted to the maximum value to obtain a blank background image; Detect straight lines in the blank background image; Using the center point of the image region corresponding to the first region of interest as a circle, and employing a first preset radius and a second preset radius respectively, a first arc and a second arc are generated in the blank background image; If the number of intersections between the first arc and the straight line is the same as the number of intersections between the second arc and the straight line, then the second region of interest is determined to include the winch rope feature.

6. The method for determining the position of a hook according to claim 1, characterized in that, When the second region of interest includes features of the winch rope, determining the hook position based on the second region of interest includes: In the case of multiple target regions, the distance between the center point of each target region and the center point of the initial image is determined, wherein the target region is a second region of interest including the features of the winch rope; The position of the hook is determined based on the target area corresponding to the minimum distance.

7. The method for determining the position of a hook according to any one of claims 1 to 6, characterized in that, The method further includes a neural network training step, which includes: Obtain a sample image of the hook; The sample image is input into a neural network to output a predicted region in the sample image that includes a hook. The neural network is used to extract features from the sample image to obtain multiple sample feature maps of different scales. The multiple sample feature maps of different scales are fused to extract the sample positioning features of the hook, and the predicted region in the sample image where the hook is located is determined based on the sample positioning features. The loss value between the predicted region and the target labeled region corresponding to the sample image is determined by the SIou loss function. When the loss value reaches a preset threshold, the trained neural network is obtained.

8. A processor, characterized in that, It is configured to perform the method for determining the position of a hook as described in any one of claims 1 to 7.

9. A crane, characterized in that, include: Hook; The boom is equipped with an image acquisition device for acquiring initial images of the hook in real time during the hoisting process. as well as The processor according to claim 8.

10. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the method for determining the position of a hook according to any one of claims 1 to 7.

Images