Method and device for positioning object hoisted by tower crane, computing equipment and storage medium

Abstract

Embodiments of the invention provide a tower crane hoisting object positioning method and apparatus, a computing device and a storage medium, a first sliding window is slid in a tower crane operation site image to obtain a first candidate frame for VLM model search, and the first candidate frame with a confidence greater than or equal to a first confidence value and a score greater than or equal to a first score is reserved to a candidate pool; sliding a second sliding window in the image block in each first candidate frame with the score of the first N in the candidate pool to obtain a plurality of second candidate frames, and retaining the second candidate frame with the first score greater than or equal to the second score; and on the basis of the pixel level, inward stepping edge shrinkage is conducted on each reserved second candidate frame edge by edge, a plurality of third candidate frames are obtained, and the third candidate frame with the optimal confidence degree after edge shrinkage of each second candidate frame is obtained from the third candidate frames and used for positioning the hoisted object. According to the method, the contradiction between generalization and positioning precision is broken through by utilizing the VLM model, and training-free high-precision positioning is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a tower crane hoisting object positioning method and device, a computing device, and a storage medium. BACKGROUND

[0002] In the field of industrial automation, especially in scenarios involving heavy object handling such as tower crane scheduling and port loading and unloading, accurate identification and positioning of target objects are the core prerequisites for achieving automated operations. With the development of artificial intelligence technology, vision-based target detection solutions have become a research hotspot in this field. However, existing technologies still have significant limitations in dealing with the complexity of tower crane operation scenarios.

[0003] Current mainstream visual target detection technologies mainly fall into two categories: image understanding technology based on visual language models (VLMs) and target positioning technology based on traditional deep learning. Both have obvious shortcomings in terms of technical principles and application effects, making it difficult to meet the needs of tower crane scheduling alone.

[0004] VLM (Visual Language Model) is a multi-modal artificial intelligence model that combines visual (image / video) and language (text) processing capabilities. By combining a large language model (LLM) with a visual encoder, it achieves comprehensive processing capabilities for images, text, and videos. Its core architecture includes a visual encoder, a projector, and a language model. It uses techniques such as parameter-efficient fine-tuning to align cross-modal information and enable embodied intelligent interaction, generating natural language related to vision. It is used for cross-modal interaction and reasoning, completing complex tasks that computer image processing models or natural language models alone cannot accomplish.

[0005] Image understanding technology based on visual language models (VLMs) has strong generalization ability and does not require training for specific objects. It can identify various objects in an image (including object categories, colors, shapes, etc.) through natural language interaction. However, the core shortcoming of this type of model is insufficient positioning accuracy. Its design goal is "image-level" or "region-level" semantic understanding, rather than "pixel-level" precision positioning. Even though some VLMs support outputting the approximate area where an object is located (such as a rectangular range), they cannot provide high-precision location information (such as object center point coordinates or precise pixel range of the bounding box) that meets the needs of tower crane operations. In tower crane hoisting operations, VLMs can only provide vague feedback such as "the object is on the left side of the image," but cannot provide specific (x, y) coordinates or bounding boxes with an error of less than a certain number of pixels, resulting in deviation in the crane's grasp and affecting operational efficiency and safety.

[0006] The target positioning technology based on traditional deep learning, such as the YOLO (You Only Look Once) series, can output the accurate bounding box (including the upper left corner and lower right corner coordinates), center point coordinates and confidence of the target object by training on a large-scale labeled dataset (such as COCO, VOC), and the positioning error can be controlled within 10 pixels, which fully meets the requirements of tower crane operation for positioning accuracy. However, its fatal defect is poor generalization and serious dependence on targeted training: the model performance is strongly bound to the training data: it is difficult to adapt to the diversity of objects in tower crane operation: the hoisting objects of the tower crane have strong diversity, including standardized containers, steel, and irregularly shaped construction waste, customized mechanical equipment, etc., and the size and shape of the same type of object vary greatly (such as steel pipes of different specifications and cement bags of different weights).

[0007] The particularity of tower crane scheduling operation further magnifies the limitations of the above two types of technology, forming a core contradiction that generalization and positioning accuracy cannot be considered.

[0008] Therefore, the existing technology always faces the dilemma of "strong generalization but insufficient positioning accuracy, high positioning accuracy but poor generalization" when dealing with the "diverse object positioning" in the tower crane scheduling scene, and a technical solution that does not require targeted training and considers generalization and high-precision positioning is needed to break through the current technical bottleneck. SUMMARY

[0009] Therefore, the embodiments of the present application provide a tower crane hoisting object positioning method and device, computing equipment and storage medium, which are based on the generalization ability of the VLM model, and perform phased positioning of the hoisting object through full-image coarse search, in-box fine search and boundary optimization, break through the positioning accuracy short board through multi-dimensional algorithm improvement, upgrade the positioning effect from regional fuzzy recognition to pixel-level accurate positioning, and consider the requirements of industrial scene for efficiency and stability. The technical solution of the present application breaks through the contradiction between generalization and positioning accuracy, realizes high-precision positioning without training, and on the premise of not relying on specific object labeled data training, optimizes the detection logic and positioning strategy of the VLM model, so that the model has the abilities of "cross-class generalization recognition" and "pixel-level high-precision positioning". The generalization of the VLM model is particularly suitable for various hoisting objects (such as reinforcing steel bars, containers and cement pipes) in tower crane operation, can cover the full size range from "small size scattered goods" (such as packed bricks) to "large size heavy equipment" (such as containers and prefabricated boards), and is compatible with the positioning requirements of regular shapes (such as steel pipes and cement columns) and irregular shapes (such as construction waste and bulk steel), without the need to adjust model parameters due to changes in object type or shape, adapt to the flexibility of on-site operation, and reduce the cost of human and algorithm.

[0010] In a first aspect, the embodiments of the present application provide a hoisting object positioning method, comprising: in a tower crane operation site image, sliding a first sliding window to obtain a plurality of first candidate boxes, searching for a hoisting object in an image block in each first candidate box by using a VLM model, obtaining a confidence and a score of each first candidate box, and retaining a first candidate box with a confidence greater than or equal to a first confidence value and a score greater than or equal to a first score value to a candidate pool; wherein the score of each candidate box changes positively with the confidence of the candidate box and changes inversely with the area of the candidate box, and the confidence of each candidate box is the probability of the hoisting object existing in the image block in the candidate box output by the VLM model; in the image block in each first candidate box with a score in the top N in the candidate pool, sliding a second sliding window to obtain a plurality of second candidate boxes, searching for the hoisting object in an image block in each second candidate box by using the VLM model, calculating the score of each second candidate box, until a second candidate box with a first score greater than or equal to a second score value is obtained, and the second score value is greater than or equal to the first score value; performing inwards stepwise edge shrinking on each retained second candidate box based on a pixel level to obtain a plurality of third candidate boxes, searching for the hoisting object in an image block in each third candidate box by using the VLM model, obtaining the confidence of each third candidate box, and retaining the third candidate box with the optimal confidence after the edge shrinking of each second candidate box, for positioning the hoisting object.

[0011] Based on the generalization ability of the VLM model, the hoisting object is positioned in stages through full-image coarse search, in-box fine search and boundary optimization, the positioning accuracy is improved through multi-dimensional algorithm, the positioning effect is upgraded from regional fuzzy recognition to pixel-level accurate positioning, and the requirements of industrial scenes for efficiency and stability are met; the generalization ability of the VLM model is particularly suitable for various hoisting objects (such as reinforcing bars, containers and cement pipes) in tower crane operations, can cover the full size range from "small size scattered goods" (such as packed bricks) to "large size heavy equipment" (such as containers and prefabricated boards), is compatible with the positioning requirements of regular shapes (such as steel pipes and cement columns) and irregular shapes (such as construction waste and bulk steel), does not need to adjust model parameters due to changes in object type or shape, is flexible in adapting to on-site operations, reduces the cost of manpower and computing power in technology landing, and retains the first candidate box and the second candidate box in the search process by using the candidate box score, which takes into account the confidence and compactness of the candidate box, and facilitates accurate identification of the position of the hoisting object.

[0012] In a possible implementation of the first aspect, the method further comprises: when a first candidate box is retained, if the intersection over union of the first candidate box and any first candidate box in the candidate pool is greater than or equal to an intersection over union threshold, retaining the candidate box with a higher score of the two candidate boxes in the candidate pool.

[0013] In summary, the candidate box with a large score is retained from the two candidate boxes with a large intersection-over-union in the candidate pool to avoid retaining candidate boxes with similar positions in the candidate pool and reduce the workload of subsequent in-box fine search.

[0014] In a possible implementation of the first aspect, the first sliding window includes a plurality of sliding windows with step changes in scale. In the process of obtaining the plurality of first candidate boxes by sliding the first sliding window in the live image, the scale change range of the first sliding window is also adjusted, specifically including: obtaining a first range according to the area proportion of the hoisting object in the live image, the first range being an initial range of the scale change of the first sliding window; step changing the first sliding window in the first range, and when the confidence of the first candidate box corresponding to any sliding position of any scale of the first sliding window in the first range is greater than or equal to a first confidence value, expanding the change range of the first sliding window to a second range, otherwise expanding the change range of the first sliding window to a third range, where the third range is greater than or equal to the second range; and the sliding step length of each first sliding window is a scale corresponding to a first proportion of the first sliding window, and the step change length of the scale change of the first sliding window is a scale corresponding to a set proportion of the live image.

[0015] In summary, when the confidence of the first candidate box corresponding to any sliding position of any scale of the first sliding window in the first range is greater than or equal to the first confidence value, the change range of the first sliding window is expanded to the relatively small second range to moderately expand the first sliding window, thereby avoiding an excessively rapid increase in the computational load, otherwise the change range of the first sliding window is expanded to the larger third range to expand the first sliding window to obtain a positioning box of the hoisting object with a large area proportion to cover the entire hoisting object.

[0016] In a possible implementation of the first aspect, the second sliding window includes a plurality of sliding windows with step changes in scale. In the process of obtaining the plurality of second candidate boxes by sliding the second sliding window in the image block in each first candidate box ranked in the top N in the candidate pool, the second sliding window is changed in scale according to the step change length of the second sliding window, for each second sliding window, the second sliding window is slid in the image block at the sliding step length of the second sliding window to obtain a plurality of second candidate boxes corresponding to the second sliding window; and the sliding step length of each second sliding window is a scale corresponding to a second proportion of the second sliding window, and the second proportion is smaller than the first proportion; and the step change length of the scale change of the second sliding window is a length corresponding to the set proportion of the retained first candidate box.

[0017] From the above, the second sliding window includes a plurality of sliding windows with step changes in scale, and the second candidate box is positioned by a plurality of step changes, which is suitable for hoisting objects of various sizes. In addition, the second scale is smaller than the first scale, so that the step length of the fine search in the second candidate box is more precise than the coarse search step length of the first candidate box. In addition, the first sliding window changes in scale by a set proportion of the scale of the tower crane operation site image, and the second sliding window changes in scale by a set proportion of the scale of the first candidate box, so that the second sliding window changes in scale more precisely.

[0018] In a possible implementation of the first aspect, the second candidate box with a score greater than or equal to the second score includes: in the image blocks in the first candidate box with a score of the top N in the candidate pool, when a second candidate box obtained by sliding the second sliding window of any scale has a score greater than the second score, stopping searching the image blocks in the first candidate box, and retaining the second candidate box.

[0019] From the above, in the image blocks in the first candidate box, when a second candidate box obtained by sliding the second sliding window of any scale has a score greater than the second score, the search of the image blocks in the first candidate box is stopped, and the second candidate box is retained. The second candidate box is the second candidate box close to the optimal, so that the calculation amount is reduced when the second candidate box close to the optimal is obtained.

[0020] In a possible implementation of the first aspect, further comprising: in the image blocks in the first candidate box with a score of the top N in the candidate pool, when any second candidate box obtained by sliding the second sliding window of any scale has a score less than the second score, retaining the second candidate box with the highest score in the image blocks in the first candidate box.

[0021] From the above, in the image blocks in the first candidate box, when a second candidate box obtained by sliding the second sliding window of any scale has a score less than the second score, the second candidate box with the highest score is retained, so that the second candidate box close to the optimal is obtained.

[0022] In a possible implementation of the first aspect, the process of searching for the hoisted object in the image block in each second candidate frame by using the VLM model further includes: when the score of the second candidate frame is greater than or equal to a third score value, which is between the first score value and the second score value, simultaneously performing a pixel-level edge shortening adjustment and a pixel-level edge lengthening adjustment on the four edges of the second candidate frame; when the confidence of the original retained second candidate frame and the confidence of the second candidate frame after the scaling adjustment are both greater than or equal to the second confidence value and the standard deviation is less than a set difference value, the score of the retained second candidate frame remains unchanged; otherwise, the four edges of the second candidate frame after the edge shortening adjustment are again simultaneously subjected to a second pixel-level edge shortening adjustment, and the four edges of the second candidate frame after the edge lengthening adjustment are again simultaneously subjected to a second pixel-level edge lengthening adjustment; and the average confidence and the average score of the original retained second candidate frame, the two second candidate frames after the first scaling edge adjustment, and the two second candidate frames after the second scaling edge adjustment are taken as the new confidence and the new score of the original retained second candidate frame.

[0023] According to the above, the new confidence and the new score of the second candidate frame are obtained according to the standard deviation of the confidence of the second candidate frame after the scaling adjustment, cross validation of the pixel-level adjustment is achieved, and the problem of the influence of the fluctuation of the confidence of single detection of the VLM model on the score of the candidate frame is reduced.

[0024] In a possible implementation of the first aspect, the process of searching for the hoisted object in the image block in each second candidate frame by using the VLM model includes: determining, by using the VLM model, whether the image block in each second candidate frame includes the hoisted object; and when the image block in the second candidate frame includes the hoisted object, determining, by using the VLM model, whether the image block in the second candidate frame includes an interference object of the hoisted object in terms of category and / or form.

[0025] According to the above, the secondary detection in terms of category and / or form by using the VLM model avoids false detection caused by noise interference and improves positioning accuracy.

[0026] In a possible implementation of the first aspect, before the second sliding window is slid in the image block in each of the first candidate boxes ranked in the top N in the candidate pool to obtain the second candidate box, the method further includes: when there are M first candidate boxes with scores less than a fourth score among the first candidate boxes ranked in the top N in the candidate pool, sliding a third sliding window in the image to obtain a plurality of fourth candidate boxes, searching for the hoisting object in the image block in each of the fourth candidate boxes by using the VLM model to obtain fourth candidate boxes ranked in the top M, and replacing the M first candidate boxes, the fourth score being less than the second score; wherein a sliding step of the third sliding window is a third scale corresponding to a third proportion of the third sliding window, the third proportion being between the first proportion and the second proportion; wherein the third sliding window includes a plurality of sliding windows with stepped scales, and a variation range of the third sliding window is less than or equal to the second range.

[0027] According to the above, when there are M first candidate boxes with scores less than a fourth score among the first candidate boxes ranked in the top N in the candidate pool, the M candidate boxes are medium-quality candidate boxes, and the third sliding window between the first sliding window and the second sliding window is used to search for candidate boxes with higher scores to replace the original retained first candidate boxes by using an intermediate sliding step.

[0028] In a possible implementation of the first aspect, when the confidence of the third candidate box obtained after any one-time stepwise edge shrinking of any edge is reduced or less than a third confidence value set in advance relative to the confidence obtained before the current edge shrinking, the edge shrinking of the edge is stopped, and the edge is restored to a state before the current stepwise edge shrinking.

[0029] According to the above, when the confidence of the third candidate box obtained after any one-time stepwise edge shrinking of any edge is reduced or less than a third confidence value set in advance relative to the confidence obtained before the current edge shrinking, the edge shrinking of the edge is stopped, and the edge is restored to a state before the current stepwise edge shrinking, which not only reduces unnecessary calculation, but also retains the best adjusted edge shrinking result.

[0030] In a possible implementation of the first aspect, the process of searching for the hoisting object in any image block by using the VLM model includes: inputting, to the VLM model, the image block, a description of the hoisting object, a task description of the search, and an output format corresponding to the task; the task at least includes one of the following: judging whether the hoisting object exists in the image block, judging whether an interference object of the hoisting object exists in the image block in terms of category and / or form; the VLM model performs semantic search in the image block according to the task, and outputs a search result in the output format; when the task is to judge whether the hoisting object exists in the image block, the output format includes a probability that the hoisting object exists in the image block.

[0031] According to the above, by adding the description of the hoisting object and the task description in the prompt word of the VLM model, the VLM model can perform more semantic analysis and obtain more accurate candidate box confidence.

[0032] In a possible implementation of the first aspect, the process of calculating the score of each candidate box comprises: obtaining the confidence of the candidate box output by the VLM model; and obtaining the score of the candidate box according to the area of the candidate box and the confidence.

[0033] In a possible implementation of the first aspect, the process further comprises: for each third candidate box obtained by shrinking the edge of each second candidate box, selecting a final candidate box for positioning the hoisting object based on the ranking of the candidate box area and the confidence according to the requirements of the positioning scene.

[0034] According to the above, the third candidate boxes with multiple sets of high-reliability positioning are retained, the deviation risk of a single solution is avoided, the multiple sets of positioning results are prioritized, and the dispatching system can select an optimal solution according to the actual work scene (such as the target size and the complexity of the surrounding environment).

[0035] In the second aspect, an embodiment of the present application provides a hoisting object positioning device, comprising: a full-image coarse search module, configured to slide a first sliding window in a tower crane work site image to obtain a plurality of first candidate boxes, search for a hoisting object in an image block in each first candidate box by using a VLM model, obtain the confidence of each first candidate box, calculate the score of the first candidate box, and retain a first candidate box with a confidence greater than or equal to a first confidence value and a score greater than or equal to a first score value in a candidate pool; wherein the score of each candidate box changes positively with the confidence of the candidate box and changes inversely with the area of the candidate box, and the confidence of each candidate box is the probability of the existence of the hoisting object in the image block in the candidate box output by the VLM model; a block-in coarse search module, configured to slide a second sliding window in the image block in each first candidate box with a score in the top N in the candidate pool, obtain a plurality of second candidate boxes, search for the hoisting object in the image block in each second candidate box by using the VLM model, and calculate the score of the second candidate box until a second candidate box with a score greater than or equal to a second score value is obtained, the second score value being greater than or equal to the first score value; and a boundary optimization module, configured to perform inwards stepwise edge shrinking on each retained second candidate box based on a pixel level to obtain a plurality of third candidate boxes, search for the hoisting object in the image block in each third candidate box by using the VLM model, obtain the confidence of each third candidate box, and retain a third candidate box with an optimal confidence after the edge shrinking of each second candidate box, for positioning the hoisting object.

[0036] Based on the generalization ability of the VLM model, the hoisting object is positioned in stages through full-image coarse search, in-frame fine search, and boundary optimization. The positioning accuracy is improved through multi-dimensional algorithms, and the positioning effect is upgraded from regional fuzzy recognition to pixel-level accurate positioning. The generalization ability of the VLM model is particularly suitable for various hoisting objects in tower crane operations (such as reinforcing steel, containers, and cement pipes), covering the full size range from "small size and scattered goods" (such as packed bricks) to "large size and heavy equipment" (such as containers and prefabricated boards). It is compatible with the positioning needs of regular shapes (such as steel pipes and cement columns) and irregular shapes (such as construction waste and bulk steel). It does not need to adjust model parameters due to changes in object type or shape, and it is flexible in adapting to on-site operations, reducing the cost of human and computing power. In addition, the search process also retains the first and second candidate boxes based on the candidate box score, taking into account the confidence and compactness of the candidate box, which facilitates accurate identification of the position of the hoisting object.

[0037] In a possible implementation of the second aspect, the full-image coarse search module is further configured to, when retaining a first candidate box, retain, in the candidate pool, a candidate box with a higher score of the two candidate boxes when an intersection over union of the first candidate box and any first candidate box in the candidate pool is greater than or equal to an intersection over union threshold.

[0038] As described above, the candidate box with a higher score of the two candidate boxes with a larger intersection over union is retained in the candidate pool to avoid retaining candidate boxes with similar positions in the candidate pool and reduce the workload of subsequent in-frame fine search.

[0039] In a possible implementation of the second aspect, the first sliding window includes a plurality of sliding windows with step changes in scale. In the process of obtaining a plurality of first candidate boxes by sliding the first sliding window in the on-site image, the full-image coarse search module is further configured to adjust the scale change range of the first sliding window, specifically comprising: obtaining a first range according to the area proportion of the hoisting object in the on-site image, the first range being the initial range of the scale change of the first sliding window; step changing the first sliding window within the first range, and when the confidence of the first candidate box corresponding to any sliding position of any scale of the first sliding window within the first range is greater than or equal to a first confidence value, expanding the change range of the first sliding window to a second range, otherwise expanding the change range of the first sliding window to a third range, wherein the third range is greater than or equal to the second range; wherein the sliding step length of each first sliding window is the scale corresponding to the first proportion of the first sliding window, and the step length of the scale change of the first sliding window is the scale corresponding to the set proportion of the on-site image.

[0040] When the confidence of the first candidate box corresponding to any sliding position of the first sliding window of any scale in the first range is greater than or equal to the first confidence value, the range of the first sliding window is expanded to a relatively small second range, so as to moderately enlarge the first sliding window and avoid rapid increase of the calculation amount, otherwise the range of the first sliding window is expanded to a larger third range, so as to increase the first sliding window and obtain a hoisting object with a large area ratio to cover the entire hoisting object.

[0041] In a possible implementation of the second aspect, the second sliding window includes a plurality of sliding windows with step changes in scale; in the process of sliding the second sliding window in the image block in each first candidate box ranked in the top N in the candidate pool to obtain a plurality of second candidate boxes, the second sliding window changes in step length according to the change in the scale of the second sliding window; for each second sliding window, the second sliding window is slid in the image block at a sliding step length corresponding to the scale of the second ratio of the second sliding window to obtain a plurality of second candidate boxes corresponding to the second sliding window; wherein the sliding step length of each second sliding window is the scale corresponding to the second ratio; the second ratio is smaller than the first ratio; and the step length of the change in the scale of the second sliding window is the length corresponding to the set ratio of the reserved first candidate box.

[0042] According to the above, the second sliding window includes a plurality of sliding windows with step changes in scale, which realizes positioning of the hoisting object through a plurality of second candidate boxes with step changes, and is suitable for hoisting objects of various sizes. In addition, the second ratio is smaller than the first ratio, which realizes finer searching step length in the second candidate box than the coarse searching step length of the first candidate box. Furthermore, the first sliding window changes in scale with the set ratio of the image scale of the tower crane operation site, and the second sliding window changes in scale with the set ratio of the scale of the first candidate box, which realizes finer change in the scale of the second sliding window.

[0043] In a possible implementation of the second aspect, when the box-in-searching module retains the second candidate box with a score greater than or equal to the second score, the box-in-searching module is specifically configured to: in the image block in one first candidate box ranked in the top N in the candidate pool, when a second candidate box obtained by sliding the second sliding window of any scale has a score greater than the second score, stop searching the image block in the first candidate box, and retain the second candidate box.

[0044] According to the above, when a second candidate box obtained by sliding the second sliding window of any scale in the image block in the first candidate box has a score greater than the second score, the searching of the image block in the first candidate box is stopped, and the second candidate box is retained, so that the second candidate box is close to the optimal second candidate box, thereby reducing the calculation amount when the close-to-optimal second candidate box is obtained.

[0045] In a possible implementation of the second aspect, the in-frame searching module is further configured to: when the scores of any second candidate box obtained by sliding the second sliding window of any scale are all less than the second score, in the image block in the first candidate box with the top N scores in the candidate pool, retain the second candidate box with the highest score in the image block in the first candidate box.

[0046] According to the above, when the scores of any second candidate box obtained by sliding the second sliding window of any scale are all less than the second score, in the image block in the first candidate box, the second candidate box with the highest score is retained, so that the second candidate box close to the optimal is obtained.

[0047] In a possible implementation of the second aspect, the in-frame searching module is further configured to: in the process of searching the hoisting object in the image block in each second candidate box by using the VLM model, for the second candidate box with a score greater than or equal to a third score, the third score being between the first score and the second score, simultaneously perform a pixel-level edge shortening adjustment and a pixel-level edge lengthening adjustment on four edges of the second candidate box; when the confidence of the originally retained second candidate box and the confidence of the second candidate box after the scaling adjustment are both greater than or equal to a second confidence value and the standard deviation is less than a set difference value, the score of the retained second candidate box remains unchanged; otherwise, the four edges of the second candidate box after the edge shortening adjustment are again subjected to a second pixel-level edge shortening adjustment, and the four edges of the second candidate box after the edge lengthening adjustment are again subjected to a second pixel-level edge lengthening adjustment; the average of the confidence and the average of the score of the originally retained second candidate box, the two second candidate boxes after the first scaling edge adjustment, and the two second candidate boxes after the second scaling edge adjustment are taken as the new confidence and the new score of the originally retained second candidate box.

[0048] According to the above, by obtaining the new confidence and the new score of the second candidate box according to the standard deviation of the confidence of the second candidate box after the scaling adjustment, cross-validation of the pixel-level adjustment is achieved, and the problem of the influence of the fluctuation of the confidence of single detection on the score of the candidate box is reduced.

[0049] In a possible implementation of the second aspect, the process of searching the hoisting object in the image block in each second candidate box by using the VLM model is specifically configured to: determine whether the image block in each second candidate box includes the hoisting object by using the VLM model; when the image block in the second candidate box includes the hoisting object, determine whether the image block in the second candidate box includes an interference object of the hoisting object in terms of category and / or shape by using the VLM model.

[0050] According to the above, the secondary detection in terms of category and / or shape by using the VLM model avoids false detection caused by noise interference and improves positioning accuracy.

[0051] In a possible implementation of the second aspect, before the second sliding window is slid in the image block in each first candidate box ranked in the top N in the candidate pool to obtain a second candidate box, the in-box fine search module is further configured to: when there are M first candidate boxes with scores less than a fourth score in the first candidate boxes ranked in the top N in the candidate pool, slide a third sliding window in the image to obtain a plurality of fourth candidate boxes, search for the hoisting object in the image block in each fourth candidate box by using the VLM model, obtain fourth candidate boxes ranked in the top M, and replace the M first candidate boxes, the fourth score being less than the second score; the sliding step of the third sliding window is a third proportion corresponding to a scale of the third sliding window, and the third proportion is between the first proportion and the second proportion; the third sliding window includes sliding windows of a plurality of step scales, and the change range of the third sliding window is less than or equal to the second range.

[0052] According to the above, when there are M first candidate boxes with scores less than a fourth score in the first candidate boxes ranked in the top N in the candidate pool, the M candidate boxes are medium-quality candidate boxes, and the third sliding window between the first sliding window and the second sliding window is used to search for candidate boxes with high scores to replace the original retained first candidate boxes with a step length of the second intermediate sliding step.

[0053] In a possible implementation of the second aspect, when the confidence of the third candidate box obtained after any one step of edge shrinking of any one edge is less than or less than a third confidence value set in advance relative to the confidence obtained before the current edge shrinking, the edge shrinking of the edge is stopped, and the edge is restored to a state before the current step of edge shrinking.

[0054] According to the above, when the confidence of the third candidate box obtained after any one step of edge shrinking of any one edge is less than or less than a third confidence value set in advance relative to the confidence obtained before the current edge shrinking, the edge shrinking of the edge is stopped, and the edge is restored to a state before the current step of edge shrinking. This not only reduces unnecessary calculation, but also retains the best result of the adjusted edge shrinking.

[0055] In a possible implementation of the second aspect, when the VLM model is used to search for the hoisting object in any image block, the VLM model is input with the image block, a description of the hoisting object, a task description of the search, and an output format corresponding to the task; the task includes at least one of the following: judging whether the hoisting object exists in the image block, judging whether an interference object of the hoisting object exists in the image block in terms of category and / or shape; the VLM model performs semantic search in the image block according to the task, and outputs a search result in the output format; when the task is to judge whether the hoisting object exists in the image block, the output format includes a probability that the hoisting object exists in the image block.

[0056] From the above, by adding the description of the hoisting object and the task description in the prompt word of the VLM model, the VLM model can perform more semantic analysis and obtain more accurate candidate box confidence.

[0057] In a possible implementation of the second aspect, when calculating the score of each candidate box, the confidence of the candidate box output by the VLM model is obtained; and the score of the candidate box is obtained according to the area of the candidate box and the confidence.

[0058] In a possible implementation of the second aspect, the method further includes: a positioning ranking module configured to, for each third candidate box obtained by shrinking the edge of each second candidate box, based on the ranking of the candidate box area and the confidence, select a final candidate box for positioning the hoisting object according to the requirement of the positioning scene based on the ranking.

[0059] From the above, the third candidate box with multiple sets of high-reliability positioning is retained, the deviation risk of a single scheme is avoided, the priority of multiple positioning results is ranked, and the scheduling system can select an optimal scheme according to an actual operation scene (such as a target size and a surrounding environment complexity).

[0060] In a third aspect, an embodiment of the present application provides a computing device, including, a bus; a communication interface connected with the bus; at least one processor connected with the bus; and at least one memory connected with the bus and storing program instructions, the program instructions, when executed by the at least one processor, causing the at least one processor to execute the method in any embodiment of the first aspect of the present application.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores program instructions, the program instructions, when executed by a computer, causing the computer to execute the method in any embodiment of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 FIG. 1 is a flowchart of a hoisting object positioning method embodiment one of the present application; Figure 2 FIG. 2 is a flowchart of a hoisting object positioning method embodiment two of the present application; Figure 3A FIG. 3 is a flowchart of a full-image coarse search method of the hoisting object positioning method embodiment two of the present application; Figure 3B FIG. 4 is a flowchart of an in-frame fine search method of the hoisting object positioning method embodiment two of the present application. Figure 3C Figure 2 is a flowchart illustrating a boundary optimization method according to a second embodiment of the hoisting object positioning method of the present application; Figure 4 Figure 1 is a structural diagram of a hoisting object positioning device according to a first embodiment of the present application; Figure 5 Figure 2 is a structural diagram of a hoisting object positioning device according to a second embodiment of the present application; Figure 6 Figure 3 is a structural diagram of a computing device according to the present application. DETAILED DESCRIPTION

[0063] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0064] In the following description, the terms "first\second\third, etc." or modules A, B, C, etc. are not only used to distinguish similar objects, or to distinguish different embodiments, but also do not represent a specific order of the objects. It is understood that the specific order or sequence can be interchanged as permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0065] In the following description, the labels indicating steps, such as S110, S120, etc., do not necessarily mean that the steps are executed in this order. The order of the steps can be interchanged or executed simultaneously as permitted.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0067] Because of the particularity of tower crane scheduling operations, hoisting objects are diverse and uncertain: in the operation scene of the tower crane (such as a construction site, a port, and a logistics park), hoisting objects include construction materials (rebar, cement, and bricks), transportation equipment (containers and forklifts), and scattered goods (packaged construction waste), and the size (such as a 2-meter long steel pipe and a 10-meter long steel pipe) and form (such as bundled rebar and bulk rebar) of the same type of object differ greatly. If a YOLO model is used, it is necessary to continuously build a labeled data set and train the model for new types of objects, which cannot quickly respond to the needs of field operations; if a VLM is used, although it can identify various types of objects, the positioning accuracy is insufficient to enable the hoist to accurately grasp.

[0068] To solve the above-mentioned contradictions, the prior art has made the following partial improvement attempts, but the effect is limited, and the root problem has not been solved.

[0069] Scheme one: "VLM + YOLO" is used in series. First, the object class in the image is recognized by VLM, and then the YOLO model trained for the object is called for positioning. This scheme can balance generalization and accuracy, but it still does not solve the core problem of the YOLO model "which needs to be trained specifically"; Scheme two: VLM fine-tuning optimization. Add "boundary box annotation data" in the pre-training process of VLM to make it have the ability to output high-precision positioning results. This scheme can improve the positioning accuracy of VLM, but fine-tuning requires a large amount of annotation data (usually tens of thousands of samples containing boundary boxes), still needs to be trained, and the generalization of the fine-tuned model decreases. The positioning accuracy of the object type not involved in fine-tuning is still not ideal, and the complexity of model deployment and maintenance is also increased.

[0070] Embodiments of the present application provide a hoisting object positioning method and device, a computing device and a storage medium. The method comprises: sliding a first sliding window in a tower crane operation site image to obtain a plurality of first candidate boxes; searching for a hoisting object in an image block in each first candidate box using a VLM model to obtain a confidence of each first candidate box and calculate a score of the first candidate box; retaining a first candidate box with a confidence greater than or equal to a first confidence value and a score greater than or equal to a first score value in a candidate pool; wherein the score of each candidate box changes positively with the confidence of the candidate box and changes inversely with the area of the candidate box, and the confidence of each candidate box is the probability of the existence of a hoisting object in the image block in the candidate box output by the VLM model; sliding a second sliding window in the image block of each first candidate box with the top N scores in the candidate pool to obtain a plurality of second candidate boxes; searching for the hoisting object in the image block in each second candidate box using the VLM model to obtain a confidence of each second candidate box and calculate a score of the second candidate box; retaining each second candidate box with the highest score and greater than or equal to a second score value, the second score value being greater than or equal to the first score value; performing inwards step-by-step edge shrinking on each edge of each retained second candidate box based on pixel level to obtain a plurality of third candidate boxes; searching for the hoisting object in the image block in each third candidate box using the VLM model to obtain a confidence of each third candidate box; retaining the third candidate box with the optimal confidence after edge shrinking of each second candidate box for positioning of the hoisting object.

[0071] The technical solution of the present application is based on a hoisting object positioning and instruction analysis scheme enhanced by a visual language model (VLM), which breaks through the contradiction between "generalization and positioning accuracy", realizes high-precision positioning without training, and under the premise of not relying on specific object annotation data training, optimizes the detection logic and positioning strategy of the VLM, so that the model has the ability of "cross-class generalization recognition" and "pixel-level high-precision positioning"; It is especially suitable for various hoisting objects (such as reinforcing steel bars, containers, and cement pipes) in tower crane operations, and can cover the full size range from "small size scattered goods" (such as packed bricks) to "large size heavy equipment" (such as containers and prefabricated boards), and simultaneously compatible with the positioning needs of regular shapes (such as steel pipes and cement columns) and irregular shapes (such as construction waste and bulk steel). Without adjusting model parameters due to changes in object type or shape, the flexibility of adapting to on-site operations reduces the human and computing power costs of technology landing; and by adapting to connected large model services, there is no need for local deployment and maintenance of large model weight files, which facilitates quick connection with existing tower crane control systems and reduces the technical landing threshold of industrial scenarios.

[0072] The following will be combined Figure 1 An embodiment of a hoisting object positioning method of the present application is introduced.

[0073] Figure 1 A flow of an embodiment of a hoisting object positioning method is shown, including steps S110 to S130.

[0074] S110: Slide a first sliding window in the tower crane operation site image to obtain a plurality of first candidate boxes; search for hoisting objects in the image block in each first candidate box using a VLM model to obtain the confidence of each first candidate box and calculate the score of the first candidate box; retain the first candidate box with confidence greater than or equal to the first confidence value and score greater than or equal to the first score value to the candidate pool.

[0075] Wherein, the score of each candidate box changes positively with the confidence of the candidate box and changes inversely with the area of the candidate box, and the confidence of each candidate box is the probability of the existence of a hoisting object in the image block in the candidate box. The first candidate box with a score greater than or equal to the first score value is a candidate box of medium quality. Each candidate box is scored based on confidence and compactness, so it is suitable for hoisting objects of various shapes.

[0076] Wherein, this step is a full-image coarse search, which searches for hoisting objects in the entire site image through medium-granularity candidate boxes, quickly locks the area where the source hoisting object may exist, generates a candidate box pool with high confidence and high compactness, narrows the range for the subsequent fine search stage, and relies on the generalization of the VLM model to adapt to diversified hoisting objects without the need for training for specific hoisting objects.

[0077] In some embodiments, the first sliding window includes several sliding windows with step changes starting from a medium scale, and the medium scale is 50% matched with the tower crane operation site image.

[0078] In some embodiments, in the process of obtaining the plurality of first candidate boxes by sliding the first sliding window in the tower crane site image, the range of the first sliding window scale change is also adjusted, specifically including: obtaining a first range according to the area proportion of the hoisting object in the site image, the first range being the initial range of the first sliding window scale change; after searching in the tower crane operation site image using the first sliding window of each scale, the scale of the first sliding window is step-changed. The sliding step length of the first sliding window is the length of the first proportion of the first sliding window. For example, the first proportion is 20%.

[0079] In some embodiments, the first range is obtained according to the scale proportion of the hoisting object in the site image, the first range being the initial range of the first sliding window scale change, and the first range being in a medium scale range. For example, for a hoisting object with an image proportion of 0.2-0.6, the first range is 0.3-0.5 of the scale of the tower crane operation site image. When the confidence of any sliding position corresponding to the first candidate box of any scale of the first sliding window in the first range is greater than or equal to a first confidence value, the change range of the first sliding window is expanded to a second range, so as to moderately enlarge the first sliding window and avoid rapid increase of the calculation amount. The second range is greater than or equal to the first range. Otherwise, the change range of the first sliding window is expanded to a third range, so as to increase the first sliding window and obtain the first candidate box meeting the requirements. The third range is greater than or equal to the second range. For example, the second range is 0.3-0.6 of the scale of the tower crane operation site image, and the second range is 0.1-0.8 of the scale of the tower crane operation site image.

[0080] In some embodiments, after obtaining the first candidate box with a confidence greater than or equal to the first confidence value and a score greater than or equal to the first score, when the intersection over union of the obtained first candidate box and any first candidate box in the candidate pool is greater than or equal to an intersection over union threshold, the candidate box with a greater score is retained in the candidate pool among the two candidate boxes, so as to avoid retaining candidate boxes with similar positions in the candidate pool and reduce the workload of subsequent fine search.

[0081] In some embodiments, the process of obtaining the confidence of each first candidate box by searching the hoisting object in the image block within each first candidate box using the VLM model comprises: inputting a prompt word to the VLM model, the prompt word including: the image block, a description of the hoisting object (for example, a red fire hydrant), a description of the search task (hoisting object in the image block), and a corresponding output format; the VLM model performs semantic search for the hoisting object in the image block and outputs a result, the result including a probability that the hoisting object exists in the image block, i.e., the confidence of the corresponding first candidate box. Inputting the description of the hoisting object to the VLM model increases the description in terms of shape or color, thereby improving the accuracy of semantic search of the VLM model.

[0082] S120: In the image block within each first candidate box ranked in the top N in the candidate pool, slide a second sliding window to obtain a plurality of second candidate boxes, search for the hoisting object in the image block within each second candidate box using the VLM model, obtain the confidence of each second candidate box, and calculate the score of the second candidate box; for each first candidate box ranked in the top N, retain the second candidate box whose first score is greater than or equal to a second score.

[0083] The second score is greater than or equal to the first score. The second candidate box with a score greater than or equal to the second score is close to the optimal candidate box.

[0084] The search of the present step is a fine search within the first candidate box. The first candidate box determined in step S110 is used as an ROI (region of interest), and the target range is further narrowed by fine scanning, thereby further compressing the positioning error. The fine search is equivalent to a fine search in the tower crane operation site image, and the calculation amount is greatly reduced. The first candidate box ranked in the top N is directly called from the coarse positioning output, and the ROI region corresponding to the first candidate box is used as the fine search range, thereby avoiding the waste of efficiency caused by indiscriminate fine scanning of the entire image.

[0085] In some embodiments, the second sliding window includes a plurality of sliding windows with step sizes. In the process of sliding the second sliding window in the image block within each first candidate box ranked in the top N in the candidate pool to obtain a plurality of second candidate boxes, the second sliding window is changed in step length according to the change of the size of the second sliding window, so that the hoisting object is positioned using a suitable second candidate box. For each second sliding window, the second sliding window is slid in the image block with its sliding step length to obtain a plurality of second candidate boxes corresponding to the second sliding window, so as to completely cover the ROI region and prevent missing search.

[0086] In some embodiments, the sliding step length of the first sliding window is a length of a first proportion of the first sliding window in the sliding direction dimension, the sliding step length of the second sliding window is a length of a second proportion of the second sliding window in the sliding direction dimension, the first proportion is greater than or equal to the second proportion, thus the sliding granularity of the second sliding window is greater than the sliding granularity of the first sliding window, and the sliding search based on the second sliding window is more accurate than the sliding search based on the first sliding window. For example, the first proportion is 20%, and the second proportion is 10%.

[0087] In some embodiments, the step length of the first sliding window dimension change is a length of a set proportion of the tower crane construction site image, and the step length of the second sliding window dimension change is a length of a set proportion of the corresponding reserved first candidate box. Thus, the first sliding window dimension changes in proportion to the dimension of the tower crane construction site image, the second sliding window dimension changes in proportion to the dimension of the first candidate box, and the second sliding window dimension changes are more accurate. For example, the set proportion is 10%.

[0088] In some embodiments, when the score of a second candidate box obtained by sliding the second sliding window of any dimension is greater than the second score, the second candidate box is close to the optimal, and to reduce the amount of calculation, the search in the image block in the first candidate box is stopped, and the search in the image block in the next first candidate box is started.

[0089] In some embodiments, in the image block in a first candidate box with a score in the top N in the candidate pool, when the score of any second candidate box obtained by sliding the second sliding window of any dimension is less than the second score, to find a second candidate box close to the optimal in the first candidate box, the second candidate box with the highest score in the image block in the first candidate box is reserved as the second candidate box close to the optimal.

[0090] In some embodiments, the process of searching the hoisting object in the image block in each second candidate box by using the VLM model includes: determining whether the image block in each second candidate box includes the hoisting object by using the VLM model; and determining whether the image block in the second candidate box has an interference object of the hoisting object in terms of category and / or form by using the VLM model. The secondary detection in terms of category and / or form by using the VLM model avoids false detection caused by noise interference and improves positioning accuracy.

[0091] In some embodiments, when the image block in a second candidate box has an interference object of the hoisting object, the second candidate box does not have the hoisting object, and the second candidate box is deleted when the second candidate box is reserved and the image block in the second candidate box is determined to have an interference object of the hoisting object in terms of category and / or form by using the VLM model from the perspective of semantic search. The interference object of the hoisting object is an interference object of the hoisting object.

[0092] In some embodiments, the process of searching the image block in the second candidate box for the hoisting object by using the VLM model includes that: the prompt words input to the VLM model include: the image block, the description of the hoisting object (the hoisting object, its color and / or shape, etc., adding the color and / or shape to improve the accuracy of the VLM model search), the task description of the search, and the output format corresponding to the task; the task includes two sub-tasks: judging whether the hoisting object exists in the image block, and judging whether the interference of the hoisting object exists in the image block in terms of the category and / or the shape; the VLM model performs semantic search in the image block according to the task, and outputs the search result in the output format; the search result includes not only the probability of the existence of the hoisting object in the image block, i.e., the confidence of the second candidate box, but also whether the interference of the hoisting object exists in the image block.

[0093] In some embodiments, in the process of searching the hoisting object in the image block in each reserved second candidate box by using the VLM model, for the second candidate box with a score greater than or equal to a third score, the third score is between the first score and the second score, the four edges of the second candidate box are adjusted at the pixel level, the new score of the second candidate box is obtained according to the standard deviation of the confidence of the adjusted second candidate box, and the cross-validation of the pixel-level adjustment reduces the influence of the fluctuation of the confidence of the single detection of the VLM on the score of the candidate box.

[0094] Specifically, for the second candidate box with a score greater than or equal to the third score, the four edges of the second candidate box are simultaneously adjusted once at the pixel level by scaling and once at the pixel level by expanding; when the confidence of the original reserved second candidate box and the confidence of the second candidate box after the scaling adjustment are both greater than or equal to the second confidence value and the standard deviation is less than the set difference value, the score of the reserved second candidate box remains unchanged; otherwise, the four edges of the second candidate box after the edge scaling adjustment are again simultaneously adjusted at the pixel level for a second time, and the four edges of the second candidate box after the edge expansion adjustment are again simultaneously adjusted at the pixel level for a second time; the mean of the confidences and the mean of the scores of the original reserved second candidate box, the two second candidate boxes after the first scaling edge adjustment, and the two second candidate boxes after the second scaling edge adjustment are taken as the new confidence and the new score of the original reserved second candidate box.

[0095] In some embodiments, before the second sliding window is slid in the image blocks in each first candidate box ranked in the top N in the candidate pool to obtain the second candidate box, when there are M first candidate boxes ranked in the top N in the candidate pool and scored less than the fourth score, the M first candidate boxes are medium-quality candidate boxes, the first candidate boxes need to be pre-processed before fine search, the third sliding window is slid in the tower crane operation site image to obtain fourth candidate boxes, and the hoisting object is searched in the image blocks in each fourth candidate box based on the name of the hoisting object using the VLM model to obtain fourth candidate boxes ranked in the top M, and the M first candidate boxes are replaced, the fourth score is less than the second score and greater than or equal to the first score. In this scenario, the search granularity in this pre-processing is between the medium granularity in step S110 and the fine search granularity in step S120, and the search result is used to find a replacement for the original first candidate box. The sliding step of the third sliding window is the length of the third proportion of the third sliding window, and the third proportion is between the first proportion and the second proportion, so that the sliding step of the third sliding window is between the sliding steps of the first sliding window and the second sliding window. In some embodiments, the third sliding window also includes sliding windows of multiple step sizes, the upper limit of the change range of the third sliding window is the same as the upper limit of the second range, and the lower limit of the change range of the third sliding window is greater than the lower limit of the first range.

[0096] In some embodiments, the prompt word input to the VIM model in the search process of this pre-processing is the prompt word in step S110, which is not described in detail here.

[0097] S130: Based on the pixel level, each edge of each retained second candidate box is stepwise shrunk edge by edge to obtain a plurality of third candidate boxes; the hoisting object is searched in the image blocks in each third candidate box using the VLM model to obtain the confidence of each third candidate box; the third candidate box with the optimal confidence after the edge of each second candidate box is shrunk is retained for positioning of the hoisting object.

[0098] In this step, the optimal second candidate box obtained by fine search is adjusted at the pixel level to accurately calibrate the boundary coordinates based on the stable and effective second candidate box output by fine search, so that the output coordinates completely match the source target contour one by one to meet the accuracy requirements of positioning.

[0099] In some embodiments, when performing the inward step-by-step edge shrinking on each retained second candidate frame, in order to reduce the amount of calculation, when the confidence of the third candidate frame obtained after any edge is shrunk for one time step decreases or is less than a set third confidence value relative to the confidence obtained before the current edge shrinking, the edge shrinking of the edge is stopped, the boundary corresponding to the edge has been optimized, and the edge is restored to the state before the current step-by-step edge shrinking to retain the adjustment result of the last step-by-step edge shrinking, and then the next edge starts the inward step-by-step edge shrinking.

[0100] In some embodiments, the VLM model is input with the prompt word input in step S110 in the process of searching for the hoisting object in the image block in each third candidate frame to obtain the confidence of each third candidate frame by using the VLM model, which is not described in detail here.

[0101] In some embodiments, the process of calculating the score of each candidate frame includes: obtaining the confidence of the candidate frame output by the VLM model; and obtaining the score of the candidate frame according to the area of the candidate frame and the confidence, so that the score of each candidate frame changes positively with the confidence of the candidate frame and changes inversely with the area of the candidate frame.

[0102] In some embodiments, the third candidate frame corresponding to the confidence of each retained second candidate frame is ranked based on the area and the confidence of the candidate frame, and the final candidate frame for positioning the hoisting object is selected based on the ranking according to the specific scene requirements of the tower crane operation, so as to select a more suitable positioning frame for different scenes.

[0103] In summary, one embodiment of the hoisting object positioning method adopts a three-stage progressive positioning strategy: full-image coarse search, in-frame fine search, and boundary optimization. This strategy is based on the generalization ability of the VLM model, and improves its positioning accuracy through multi-dimensional algorithms to upgrade the positioning effect from regional fuzzy recognition to pixel-level accurate positioning, while taking into account the requirements of industrial scenes for efficiency and stability.

[0104] The following will be described in combination with Figures 2-3C An embodiment two of the hoisting object positioning method of the present application is introduced.

[0105] The embodiment two of the hoisting object positioning method integrates the method described in the embodiment one of the hoisting object positioning method, and has all the advantages thereof.

[0106] The hoisting object positioning method embodiment two is used for positioning a hoisting object of a tower crane, solves the problem that generalization and positioning accuracy are difficult to be considered in a tower crane scheduling scene, does not need to separately construct a labeling data set or train a special model for various hoisting objects (such as reinforcing steel bars, containers, cement pipes, etc.) in a tower crane operation, and only through optimizing an algorithm to improve the sensitivity of a VLM to a target position, an accurate bounding box and a center point coordinate meeting operation requirements can be output, the problems such as "new type objects cannot be quickly positioned" and "model quantity redundancy" are solved, and the human and algorithm costs of technology landing are reduced. Moreover, the hoisting object positioning method embodiment two is suitable for the scene of the diversity and form of tower crane hoisting objects, including the scene of "many types, various forms, and large size differences" of hoisting objects, through optimizing the multi-scale detection logic of a VLM model, the VLM model can cover the full size range from "small size scattered goods" (such as packed bricks) to "large size heavy equipment" (such as containers and prefabricated boards), and is compatible with the positioning requirements of regular shapes (such as steel pipes and cement columns) and irregular shapes (such as construction waste and bulk steel), and does not need to adjust model parameters due to the change of object types or forms, and is suitable for the flexibility of field operations.

[0107] The hoisting object positioning method embodiment two is a lightweight and easy-to-deploy integrated solution, a positioning system adapts a VLM service (such as Ali Cloud DashScope) of a third-party model, does not need to localize deployment and maintain a large model weight file, outputs structured data and visual results, is convenient for quick docking with an existing tower crane control system, and reduces the technical landing threshold of an industrial scene.

[0108] Figure 2 A flow of the hoisting object positioning method embodiment two is shown, including steps S210 to S260.

[0109] S210: According to a user language instruction, a name of a hoisting object is obtained by using a LLM model.

[0110] In this step, the following is included: After receiving a user natural language scheduling instruction, a positioning system automatically generates a prompt (prompt word) containing strict constraints, the prompt includes: please analyze the following scheduling instruction, only extract a hoisting object (source_object) and a target landing point (destination), and output a JSON containing only the two fields, without adding any explanation. After running the generated prompt and calling a LLM model interface to obtain a return result, the system first tries to extract the content wrapped by the json mark (such as {"source_object": "red container", "destination": "warehouse east platform"}), and ensures that the target can be accurately matched and searched in the subsequent positioning link.

[0111] Note that this embodiment only solves the positioning problem of the hoisting object. If the target landing point is a specific object, the method of this embodiment can also be used.

[0112] S220: Obtain a tower crane work site image including the hoisting object, and adapt it to a pre-processed image supported by the VLM model.

[0113] Among them, the positioning system collects real-time images in the tower crane work area through industrial cameras deployed in the tower crane work area (covering the hoisting object, the surrounding environment and the landing point area), first reads the raw image data output by the camera using OpenCV, and then calls the image encoding interface to convert it to PNG format; At the same time, the image is pre-processed (the scaling function of OpenCV is called to adjust the original image to the resolution range supported by the VLM model, such as 1024x768 pixels, to ensure that the image size meets the model input requirements and avoid model analysis failure due to improper resolution), Ensure that the resolution of the output PNG image is adapted to the VLM input specification, and finally generate standardized image data that can be directly used for model inference.

[0114] S230: Slide a first sliding window in the pre-processed image to obtain a plurality of first candidate boxes; search for the hoisting object in the image block within each first candidate box using the VLM model to obtain the confidence of each first candidate box and calculate the score of the first candidate box; retain the first candidate box whose confidence is greater than or equal to the first confidence value and whose score is greater than or equal to the first score value to the candidate pool.

[0115] Among them, the PNG image after preprocessing is directly called, and the hoisting object name obtained in step S210 is used as the search keyword to drive the VLM model to perform full-image region detection, avoiding detection failure due to incompatible image formats or blurred target information.

[0116] Among them, the scale of the first sliding window changes in three ranges, and the first sliding window in each range includes a plurality of sliding windows with step changes in scale. The scale of the first sliding window in each range can change step by step in increments or step by step in decrements. Here, step refers to the same step length each time.

[0117] Among them, the first range is the initial range of the scale change of the first sliding window, which is determined according to the estimated value of the hoisting object in the pre-processed image. When the confidence of the first candidate box corresponding to any sliding position of any scale of the first sliding window in the first range is greater than or equal to the set first confidence value, the change range of the first sliding window is expanded to the second range, otherwise the change range of the first sliding window is expanded to the third range. The second range is greater than or equal to the first range, and the third range is greater than or equal to the second range.

[0118] This step includes the following procedures: (1) Break through the inefficiency of traditional fixed sequence scanning, combine the size distribution characteristics of the hoisting object in the tower crane operation, adopt the medium scale priority strategy, initially focus on the first sliding window of medium scale, for example, the tower crane scene is mostly medium and small objects with an image proportion of 0.2-0.6, the scale change range of the first sliding window is initially focused on the range of 0.3-0.5 proportion of the preprocessed image, to adapt to common targets such as steel and cement pipes; If a first candidate box with a confidence value ≥ a first confidence value (for example, 0.7) is detected, the change range of the subsequent first sliding window is immediately expanded to a second range (for example, in the proportion range of 0.2-0.6 of the preprocessed image), reducing the invalid calculation of "too small / too large scale" (such as small parts <0.1, large equipment >0.7). If no first candidate box with a confidence value ≥ a first confidence value is detected, the change range of the subsequent first sliding window is expanded to a third range (for example, in the proportion range of 0.1-0.8 of the preprocessed image), to ensure that no special target is missed.

[0119] (2) Score and screen the first candidate box based on confidence and compactness, for example, to solve the problem that the VLM model easily outputs large size redundant boxes, the embodiment designs a comprehensive scoring formula as shown in formula (1), which preferentially ensures the existence of reliable hoisting objects (confidence accounts for 60%), while penalizing redundant boxes (the smaller the area accounts for, the higher the compactness score); only the first candidate box with a score ≥0.5 is retained, sorted in descending order of score and limited to a maximum of 10 in the pool, to avoid calculation overload in the subsequent detailed search stage.

[0120] (1) Wherein, 0.6 and 0.4 can be set according to the actual scene. The candidate box calculation of formula (1) is suitable not only for the first candidate box, but also for the score calculation of each candidate box in the embodiments of the present application.

[0121] (3) De-duplicate the retained first candidate box based on real-time IOU, adopt the strategy of scanning and de-duplicating at the same time, every time a new first candidate box is generated, immediately calculate the IOU (intersection over union) with the existing first candidate box in the pool, if the IOU is higher than the set intersection threshold (for example, ≥0.7), it is determined as a duplicate box, then the first candidate box with a higher score is retained and the low-score duplicate box is discarded, to reduce the subsequent candidate box pool cleaning cost and ensure that no duplicate candidate box pool is output.

[0122] Wherein, the specific implementation method of this step is referred toThe full image rough search method of the embodiment shown in Figure 3A

[0123] S240: In the image blocks in each first candidate box ranked in the top N in the candidate pool, a plurality of second candidate boxes are obtained by sliding a second sliding window; in the image blocks in each second candidate box, a hoisting object is searched based on the name of the hoisting object by using a VLM model, a confidence of each second candidate box is obtained, and a score of the second candidate box is calculated; for each first candidate box ranked in the top N, a second candidate box corresponding to a first score greater than or equal to a second score is retained, and if there is no second candidate box with a score greater than or equal to the second score, a second candidate box with the highest score is retained.

[0124] In the score of the first candidate box ranked in the top N is directly called the score of the coarse positioning output, the corresponding image block is taken as the ROI region, and the ROI region is taken as the fine search range to avoid the waste of efficiency caused by indiscriminate fine scanning of the whole image, and the name of the hoisting object is taken as the keyword for the VLM model search to ensure consistency of the search. This step includes the following processes: (1) Sampling adaptive scale progressive fine scanning, dynamically adjusting the fine scanning strategy according to the score of the first candidate box to balance the accuracy and efficiency. For example: for a high-quality first candidate box with a score greater than or equal to a fourth score (for example, the fourth score is 0.8), a small-scale second sliding window progressive search method is used, for example, the scale of the second sliding window changes from 0.7 to 0.3 (decreases by 0.1 each time) relative to the ROI size, and the sliding step is set to 10% of the window size (minimum is greater than or equal to 6 pixels), to achieve high-density fine scanning. For an ordinary first candidate box with a score between the first score and the fourth score (0.5-0.8), a third sliding window with an intermediate scale is first used to search in the preprocessed image, the upper limit of the scale range of the third sliding window is the same as the upper limit of the second range, and the lower limit of the third sliding window is between the lower limit of the second range and the lower limit of the first range. For example, the scale of the third sliding window is 0.4-0.6 of the scale of the preprocessed image, the sliding step is set to 15%, the selected new candidate box replaces the ordinary candidate box, and then a small-scale progressive search method is used. For each first candidate box ranked in the top N, if a second candidate box with a score greater than or equal to a second score (for example, the second score is 0.9) is detected by a second sliding window of a certain scale, it is determined that the second candidate box is close to the optimal, and the ROI fine scanning is immediately stopped, the search of the next first candidate box ranked in the top N is started, and the overall time consumption is shortened.

[0125] For each first candidate box ranked in the top N, if a second candidate box with a score greater than or equal to a second score is not detected by a second sliding window of a certain scale, the second candidate box with the highest score is retained as the second candidate box close to the optimal.

[0126] (2) Multi-round cross-validation to eliminate random errors in VLM: To address the issue of confidence fluctuation (±0.1) in a single detection of the VLM model, cross-validation is introduced. For example, for the second candidate box with a score ≥ the third score (for example, the third score is 0.85) in the fine scan, three adjacent regions within ±3 pixels of its periphery are selected and detected using the VLM model respectively; if the mean confidence of the three detections is ≥0.8 and the standard deviation is ≤0.05, it is determined to be a stable and valid box; if the fluctuation is large (standard deviation >0.05), the range is expanded to ±5 pixels and the mean of five detections is taken to ensure the reliability of the results.

[0127] (3) Use semantic association to filter interference. Utilize the image text association capability of the VLM model to add semantic verification. Extract the image region corresponding to the second candidate box of the fine scan and input Prompt to VLM: Is the object in this region consistent with [lifting object] in terms of category and shape? Only output "yes / no"; if it returns "no", then exclude the box (judged as interference) to avoid misjudging non-target objects, especially suitable for complex operation scenarios with multiple overlapping objects.

[0128] For the specific implementation method of this step, please refer to [link / reference]. Figure 3B The in-frame fine search method of this embodiment is shown.

[0129] S250: For each retained second candidate box, perform pixel-level inward step shrinking on each side. After each step shrinking of any side, a third candidate box is obtained. In the image block within each third candidate box, the hanging object is searched using the VLM model to obtain the confidence of each third candidate box, until the third candidate box with the best confidence after shrinking the second candidate box is obtained.

[0130] Based on the stable and effective second candidate box output by the fine-search method, and combined with the pixel resolution of the preprocessed image (e.g., 1024×768), a greedy edge trimming method combined with a two-way verification optimization method is used to accurately calibrate the boundary coordinates. This overcomes the defect of the traditional method of simultaneously and unidirectionally reducing the boundary on all four sides, which easily leads to the loss of the target. By adopting an edge-by-edge trimming method, it ensures that the output third candidate box can be used for path planning of the tower crane control system. This step includes the following process: (1) Shrink the four sides of the frame, left, top, right and bottom, in steps of 1 pixel / time for small frame and 2 pixels / time for large frame. Each shrinking of any side will yield a third candidate frame. (2) Each time the edge is trimmed, a third candidate box is obtained and the verification is performed simultaneously: the VLM model detects whether the box after trimming still contains the source lifting object, that is, the confidence of the third candidate box is ≥ the third confidence value (for example, the third confidence value is 0.8). (3) If the verification passes, retain the adjusted third candidate box; if any fails, restore the edge to the state before the last adjustment.

[0131] The specific implementation method of this step is referred to Figure 3C The boundary optimization method of the embodiment is shown.

[0132] S260: Rank each retained third candidate frame based on the candidate frame area and the confidence, and select the final candidate frame for lifting object positioning based on the ranking according to the needs of the positioning scene, so as to select a more suitable positioning frame for different scenes.

[0133] Among them, based on all effective third candidate frames obtained in step S250, multiple groups of high-reliability positioning results are retained to avoid the risk of deviation of a single scheme; according to unified standard specification rectangular frame coordinates (with the lower left corner of the image as the origin), confidence and other key information, JSON format data that can be directly analyzed by machines is generated to ensure compatibility with the tower crane control system; the multiple positioning results are prioritized to facilitate the dispatching system to select the optimal solution according to the actual work scene (such as target size, surrounding environment complexity). This step includes the following processes: (1) The data of each optimal third candidate frame obtained in step S250 includes: the original coordinate information of each optimal third candidate frame (left lower corner, right upper corner pixel coordinates, default coordinate system based on the lower left corner of the image as the origin); and the corresponding comprehensive confidence of each optimal third candidate frame.

[0134] (2) The optimal third candidate list is sorted according to the rule of "comprehensive confidence descending + rectangular frame compactness ascending" (preferably selecting a scheme with high confidence and a frame body more suitable for the target), specifically: the first sorting dimension is the comprehensive confidence (weight 100%, from high to low); the second sorting dimension is the area of the rectangular frame (when the confidence is the same, the frame with smaller area is more compact and has higher priority); limit the number of output candidate frames: at most 5 groups.

[0135] (3) The output JSON data contains two modules: "basic information" and "positioning candidate list", and the field definitions are as follows:

[0136] (4) Select a more suitable positioning frame according to the tower crane scene. For example, a vertical bucket-shaped or block-shaped lifting object selects the final positioning frame based on the confidence, and for an inclined structural frame, the final positioning frame is selected based on the combination of confidence and score.

[0137] In steps S230 to S250 of the embodiment, the target existence confidence output by the VLM model is the core basis for judgment, and the consistency and reliability of the confidence in different steps and different scenarios are ensured depending on the standardized Prompt template design and model output analysis rules. The confidence (value 0~1) is output by the VLM model according to the judgment probability of “whether the input image block contains the hoisting object”, and the core logic is as follows: (A) For the current image block to be detected (the image block in the first sliding window when full-image coarse search, the image block in the second sliding window when ROI sub-region fine search in the frame, or the image block in the third sliding window when boundary optimization), the image block is converted into Base64 encoding format conforming to the input requirements of the VLM model; (B) Input the Prompt containing “image block + target name description + detection task description + output format constraint” to the VLM model; (C) The VLM model outputs structured results containing “whether the target is contained” and “corresponding confidence” according to the Prompt instruction; (D) Extract the confidence value in the result, and after range verification (ensure that 0≤confidence≤1), use it for candidate box scoring, screening and optimization judgment.

[0138] Among them, the Prompt instruction used in each step is described in detail in the full-image coarse search method, the frame-in fine search method and the boundary optimization method of the embodiment.

[0139] In the above process, the confidence analysis and abnormal handling of the candidate box are processed according to the following rules.

[0140] (i) During format analysis, no matter which stage, the JSON output by the VLM model is extracted through a unified Python string extraction function (compatible with “pure text JSON” and “JSON in ```json code block package”), which can still be effectively parsed when the VLM output format fluctuates slightly; (ii) During range verification, the extracted confidence value needs to be verified in the range of “0≤confidence≤1”. If it is out of range (such as VLM model output 1.05 or -0.02), it is automatically corrected to 1.0 or 0.0; (iii) When an exception occurs, if the VLM model output cannot be parsed into JSON (such as network fluctuation leading to output garbled code), the default output is `confidence=0.0`, and the image block is marked as “no target detected” to avoid system crash.

[0141] The full-image coarse search method of the second embodiment of the hoisting object positioning method will be introduced below. Figure 3A

[0142] Figure 3A ​A flowchart of a full image coarse search method of a hoisting object positioning method embodiment two is shown, including steps S2310 to S2390.

[0143] S2310: Take the first range as the current range of the first sliding window scale change, and select the current scale of the first sliding window from the current range.

[0144] Wherein, the first range is obtained according to the estimated value of the scale proportion of the hoisting object in the preprocessed image, and the first range is the initial range of the first sliding window scale change. In this embodiment, the range of the sliding window scale change is the change range of the scale of the searched image, and in this embodiment, each sliding window is a rectangle, and the range of the sliding window scale change is the change range of the length and width of the rectangle relative to the length and width of the searched image, and the length and width of the sliding window change synchronously according to the same proportion.

[0145] For example, the first range is 0.3 to 0.5, i.e. the change range of the length of the first sliding window is 0.3 to 0.5 of the length of the preprocessed image, and the change range of the width of the first sliding window is 0.3 to 0.5 of the width of the preprocessed image.

[0146] Wherein, in this embodiment, the step length of the sliding window scale change is a set proportion length of the scale of the searched image, for example, the set proportion is 0.1.

[0147] Wherein, the current scale of the first sliding window can start from the upper limit of the current range, or from the lower limit, or from the middle, for example, when the first range is 0.3 to 0.5. The preferred method starts from 0.5 and decreases first.

[0148] S2320: Within the current range of the first sliding window scale change, slide the first sliding window of the current scale in the preprocessed image by one sliding step length, obtain the current first candidate box, and search for the hoisting object in the image block in each first candidate box using the VLM model.

[0149] Wherein, in this embodiment, the sliding step length of the first sliding window is the length of the first sliding window in the sliding direction scale of the first proportion, which is at least 12 pixels, for example, the first proportion is 20%. The first sliding window of the current scale slides one position each time, and the first sliding window of the current scale at the current sliding position is a first candidate window in the preprocessed image.

[0150] Wherein, the VLM model is used to search for the hoisting object, the confidence of the current first candidate box is obtained, and the area of the current first candidate box is combined to calculate the score of the current first candidate box.

[0151] Where, in the core requirements of this step: input PROMPT (i.e. prompt word), quickly determine whether the sliding window contains the lifting object "{source_object}." The efficiency and recall rate are considered, and a certain degree of confidence fluctuation is allowed. The VLM model only outputs strictly formatted JSON, without adding any extra words. The JSON contains "contains" (Boolean, true / false) and "confidence" (i.e. confidence, 0.0-1.0 floating point number, step change 0.1) two fields.

[0152] Where, the task of PROMPT is to determine whether the current image block contains "{source_object}" (the lifting object name obtained by instruction analysis, such as "red container" "steel bundle"). Requirements: if the image block clearly contains the object, output high confidence (≥0.7); if it is suspected to contain (outline is blurred but can be recognized), output medium confidence (0.5-0.7); if it does not contain or cannot be recognized at all, output low confidence (<0.5).

[0153] For example, the following is a steel bundle covered with blue rainproof cloth as the lifting object, and the input of this step of PROMPT is introduced.

[0154] The input of PROMPT includes: The image block in the current first candidate frame; The description of the lifting object is "steel bundle covered with blue rainproof cloth"; Task: determine whether the current image block contains "steel bundle covered with blue rainproof cloth"; The output format of JSON: if the image block clearly contains the object, output high confidence (≥0.7); if it is suspected to contain (outline is blurred but can be recognized), output medium confidence (0.5-0.7); if it does not contain or cannot be recognized at all, output low confidence (≤0.4).

[0155] VLM mode output example: ```json {"contains": true, "confidence": 0.82} ``` Where, "contains" true means that the current image block contains the lifting object, and "confidence" 0.82 means that the probability of the existence of the lifting object in the current image block is 0.82, i.e. the confidence of the current first candidate frame is 0.82.

[0156] S2330: When the confidence of the current first candidate frame is greater than or equal to the first confidence value and the current range of the scale change of the first sliding window is the first range, the first sliding window is changed to the second range of the current range of the scale change.

[0157] For example, the first confidence value is 0.7.

[0158] For example, when the first range is 0.3 to 0.5, the second range is 0.2 to 0.6. S2340: When the confidence of the current first candidate frame is greater than or equal to the first confidence value and the score is greater than or equal to the first score value, the first candidate frame is calculated with the candidate frame in the candidate pool to determine whether to keep the first candidate frame in the candidate pool.

[0159] For example, the first score value is 0.5, which takes into account the confidence and compactness of the first candidate frame.

[0160] For example, the set intersection-over-union threshold value is 0.7.

[0161] For example, the set intersection-over-union threshold value is 0.7.

[0162] For example, the set intersection-over-union threshold value is 0.7.

[0163] S2350: Determine whether the sliding of the first sliding window of the current scale is completed.

[0164] If the first sliding window of the current scale is not completed, step S2360 is executed, otherwise, step S2320 is returned.

[0165] S2360: Determine whether the sliding of the first sliding window of all scales in the current range is completed.

[0166] If the sliding of the first sliding window of all scales in the current range is completed, step S2380 or S2390 is executed, otherwise, step S2370 is executed to select the next first sliding window with a step change in the current range as the first sliding window of the current scale.

[0167] S2370: Select the first sliding window of the next step change in the current range as the first sliding window of the current scale.

[0168] Wherein, no matter the current range is the first range, the second range or the third range at this time, the same steps are executed, and after the execution of the current step is completed, the execution of S2320 is returned.

[0169] Wherein, if the current range of the scale change of the first sliding window is the first range, it means that the second range has not been expanded at this time, and it is not suitable to expand to the second range, the step S2380 is executed, otherwise, it means that the overall search of the medium scale has been completed at this time, the S2390 is executed. S2380: When the current range is the first range and the confidence of all the first candidate boxes in the first range is less than the first confidence value, the first sliding window is changed to the third range as the current range of the scale change, and one of the first sliding windows in the third range is selected as the first sliding window of the current scale.

[0170] Wherein, when the current range is the first range and the confidence of all the first candidate boxes in the first range is less than the first confidence value, it means that the second range has not been expanded at this time, and it is not suitable to expand to the second range, and it needs to be expanded to the third range.

[0171] Wherein, the third range is greater than the second range, and exemplarily, when the first range is 0.3 to 0.5, the second range is 0.2 to 0.6, and the third range is 0.1 to 0.8.

[0172] S2390: When the current range is the second range or the third range, output the first candidate box and its score in the candidate pool.

[0173] The following will be combined Figure 3B to introduce a frame-in fine search method of a lifting object positioning method embodiment two.

[0174] Figure 3B A flow of a frame-in fine search method of a lifting object positioning method embodiment two is shown, including steps S2410 to S2490.

[0175] S2410: Select the first candidate box and its score of the top N scores from the candidate pool.

[0176] Wherein, the N of the top N is based on the scene setting, and exemplarily, N is equal to 10.

[0177] S2420: For each first candidate box of the top N scores, judge whether its score is greater than or equal to the fourth score value.

[0178] If the score is greater than or equal to a fourth score value, step S2440 is performed, otherwise step S2430 is performed. The fourth score value is greater than the first score value, and for example, the fourth score value is equal to 0.8.

[0179] S2430: Selecting each third sliding window of the scale step in the fourth range, sliding in the pre-processed image respectively, obtaining each fourth candidate box, searching the hoisting object in the image block in each fourth candidate box by using the VLM model, retaining the fourth candidate box with a high score, and replacing the corresponding first candidate box in the top N score.

[0180] In this step, the fourth candidate box with a score of the top M is obtained, and the M first candidate boxes are replaced respectively. The "top" means the top M.

[0181] The sliding step length of the third sliding window is a third proportion of the length of the third sliding window in the sliding direction scale, and the third proportion is less than the first proportion corresponding to the first sliding window. For example, the third proportion is 0.15.

[0182] The third sliding window includes a plurality of sliding windows of stepped scales, the upper limit of the fourth range is the same as the upper limit of the second range, and the lower limit of the fourth range is greater than the lower limit of the first range. For example, the fourth range is 0.4 to 0.6.

[0183] The JSON output by the PROMPT and the VLM model in this step can refer to the description of the JSON output by the PROMPT and the VLM model in step S2320 of the flow of the full-image coarse search method of one hoisting object positioning method embodiment two, which will not be described in detail here.

[0184] S2440: Obtaining an image block in one first candidate box in the top N score as a current interest image, and selecting a second sliding window of one scale in a fifth range as a second sliding window of a current scale.

[0185] Each first candidate box in the top N score needs to be selected and searched separately. A second sliding window of one scale in the fifth range is selected as a second sliding window of a current scale. The second sliding window of the current scale can start from the largest second sliding window in the fifth range and step down, or start from the smallest second sliding window and step up.

[0186] S2450: Sliding the second sliding window in the current interest image to obtain a second candidate box, and searching the hoisting object in the image block in each second candidate box by using the VLM model.

[0187] The size of the second sliding window varies in a fifth range, and the fifth range is 0.3 to 0.7, for example.

[0188] In the embodiment, the sliding step of the second sliding window is a length of a second proportion of the size of the second sliding window in the sliding direction, and the second proportion is 10%, for example. The second sliding window of the current size is a second candidate window in the current interest image when the second sliding window of the current size slides one position.

[0189] In the current interest image searching process, if the image block in the second candidate frame exists (i.e., the confidence is greater than the second confidence value, and the second confidence value is 0.8, for example), the VLM model is also used to exclude the second candidate frame in which the image block is a disturbance from the category and / or form, so as to improve the accuracy of the confidence judgment.

[0190] In the core requirement of the step, the input PROMPT is used to accurately identify the lifting object in the ROI region (interest image) and exclude similar disturbances. Only the JSON in strict format is output, and no extra text is added. The JSON includes three fields of "contains" (Boolean value, true / false, indicating whether it is included), "confidence" (floating point number of 0.0-1.0, indicating the confidence), and "is_interference" (Boolean value, true / false, indicating whether it is a disturbance).

[0191] In the task of the PROMPT in the step, task 1 is to determine whether the lifting object "{source_object}" exists in the current image block (ROI region), and task 2 is to determine whether the object is the target (non-similar disturbance) with the same category and form as the lifting object "{source_object}". The requirements are as follows: clearly included and non-disturbance, confidence≥second confidence value; suspected to include or difficult to distinguish disturbance, confidence=0.5-0.7; not included or disturbance, confidence≤0.4 and is_interference=true.

[0192] For example, the following introduces the PROMPT input and JSON output of the step by taking the red container as an example.

[0193] The input PROMPT includes: The image block, i.e., the image block in each second candidate frame The description of the hoisting object, for example, "red container"; The task description includes: task 1. Determine whether there is a "red container" in the current image block (ROI region); task 2. If there is, determine whether the object is the target (non-similar interference objects such as "red truck" and "blue container") that is completely consistent with the "red container" category and shape. Requirements: clearly contains and non-interference, confidence >= 0.8; suspected to contain or difficult to distinguish interference, confidence = 0.5-0.7; does not contain or is an interference, confidence <= 0.4 and is_interference = true.

[0194] Output JSON in strict format, without adding any extra words. The JSON includes "contains" (Boolean, true / false), "confidence" (floating point number between 0.0 and 1.0), and "is_interference" (Boolean, true / false, to determine whether it is an interference).

[0195] For example, the JSON output by the VLM is: ```json {"contains": true, "confidence": 0.91, "is_interference": false} ``` Where "contains" true indicates that the current image block includes the hoisting object, "confidence" 0.91 indicates that the probability of the hoisting object existing in the current image block is 0.91, i.e. the confidence of the current second candidate box is 0.91. "is_interference" false indicates that there is no interference or similar object in the current image block.

[0196] S2453: When the score of the current second candidate box is greater than or equal to a third score value, the score of the current second candidate box is de-bounced.

[0197] Wherein the third score value is greater than the fourth score value, for example, the third score value is 0.85.

[0198] Wherein the de-bouncing step includes: (1) Expand each side of the current second candidate box by 3 pixels to obtain a new second candidate box and reduce it by 3 pixels to obtain a new second candidate box, and then search for the hoisting object in the image block within the new second candidate box using the VLM model.

[0199] (2) If the average of the confidence values of the original current second candidate box and the two new second candidate boxes is greater than or equal to the second confidence value and the standard deviation is less than or equal to the set standard deviation (for example, the standard deviation is 0.5), the score of the original current second candidate box is kept unchanged.

[0200] (3) Otherwise, each edge of the current second candidate box is respectively enlarged by 2 pixels to obtain a new second candidate box and reduced by 2 pixels to obtain a new second candidate box, then the hoisting object in the image block in the new second candidate box is searched by using the VLM model, and the average of the confidence values and the average of the scores of the original current second candidate box and the four new second candidate boxes are taken as the confidence value and the score of the original current second candidate box.

[0201] S2456: When the score of the current second candidate box is greater than or equal to the second score value, the search of the current interest image is stopped, it is indicated that the search of the current interest image is completed, and the current second candidate box is kept.

[0202] The second score value is greater than the third score value, and for example, the second score value is equal to 0.9.

[0203] S2460: When the score of the current second candidate box is less than the second score value, it is judged whether the second sliding window of each scale in the fifth range is completed sliding and the corresponding search.

[0204] When the second sliding window of each scale in the fifth range is completed sliding and the corresponding search, step S2470 is executed; otherwise, when the sliding of the second sliding window of the current scale is not completed, step S2450 is returned, the sliding of the second sliding window of the current scale is continued to search, the sliding of the second sliding window of the current scale has been completed, and the search of the second sliding window of other scales has not been completed, step S2450 is returned, and a second sliding window is reselected.

[0205] S2470: For each second candidate box kept in the search process of the current interest image, the second candidate box with the highest score is obtained as the final second candidate box kept for the current interest image.

[0206] S2480: It is judged whether the interest image block in each first candidate box in the top N scores is completed.

[0207] If the interest image block in each first candidate box in the top N scores is completed, step S2490 is executed, otherwise step S2440 is returned.

[0208] S2490: The final kept second candidate box corresponding to each first candidate box in the top N scores is output.

[0209] The following will be described in combination with Figure 3CIntroduce a boundary optimization method of the second embodiment of the hoisting object positioning method.

[0210] Figure 3C The flow of the boundary optimization method of the second embodiment of the hoisting object positioning method is shown, including steps S2510 to S2550.

[0211] S2510: Obtain each reserved second candidate box.

[0212] Each first candidate box in the top N scores corresponds to a second candidate box.

[0213] S2520: Select one of the second candidate boxes as the currently optimized second candidate box.

[0214] S2530: Stepwise edge adjustment is performed on each edge of the currently optimized second candidate box based on the pixel level, a third candidate box is obtained, and the hoisting object is searched in the image block in the third candidate box using the VLM model. The new candidate box with the highest confidence is reserved as the third candidate box.

[0215] This step includes the following flow: (a) Select an edge of the currently optimized second candidate box as the current edge, and obtain the confidence of the current candidate box.

[0216] (b) Adjust the edge inward by a certain number of pixels to obtain a third candidate box, and calculate the confidence of the new candidate box using the VLM model.

[0217] The certain number is set according to the resolution of the camera. The higher the resolution, the larger the certain number. For example, the resolution of 1080P corresponds to a certain number of 2, and the resolution of 720P corresponds to a certain number of 1.

[0218] (c) Determine whether the confidence of the third candidate box obtained this time is not lower than the confidence obtained last time and greater than a third confidence value, the third confidence value being greater than or equal to the first confidence value. For example, the third confidence value is 0.8.

[0219] If the requirement is not met, continue to adjust the current edge inward and perform the search and judgment as above.

[0220] If the requirement is met, stop adjusting the current edge, and save the position and confidence of the current edge before this adjustment.

[0221] (d) Select another edge of the currently changed second candidate box, and continue steps (a), (b) and (c) to obtain the third candidate box with the optimal confidence corresponding to each edge of the currently changed second candidate box after edge adjustment.

[0222] Wherein, in the core requirements of the present step: input PROMPT, verify whether the adjusted third candidate frame contains the hoisting object completely, focus on the "target integrity" judgment, and the confidence needs to reflect the completeness of the target in the frame. Only output the JSON in strict format, without adding any extra words. The JSON contains "contains_complete" (Boolean value, true / false, whether the target is completely contained) and "confidence" (0.0-1.0 floating point number) two fields. The task is to judge whether the current image block (adjusted bounding box region) completely contains "{source_object}", that is, the main part of the target (≥90%) is in the image block, without obvious cutting or missing. Requirements: complete target and clear main body, confidence≥0.85; partial inclusion (such as main body missing <30%) or outline blurred (such as confidence=0.5-0.8); not contained or main body missing ≥30%, confidence≤0.45 and contains_complete=false.

[0223] For example, the following takes a cement pipe as the hoisting object to introduce the PROMPT input and Jason output of the present step.

[0224] The input PROMPT includes: The image block in the third candidate frame; The description of the hoisting object is "long strip cement pipe"; The task description is whether the current image block (the image block in the third candidate frame) completely contains the "cement pipe", that is, the main part of the target (≥90%) is in the image block, without obvious cutting or missing. Requirements: complete target and clear main body, confidence≥0.85; partial inclusion (main body missing <30%) or outline blurred, confidence=0.5-0.8; not contained or main body missing ≥30%, confidence≤0.45 and contains_complete=false; The output format is JSON, without adding any extra words. The JSON contains two fields: "contains_complete" (Boolean, true / false, whether the target is completely contained) and "confidence" (float number between 0.0 and 1.0). Task: Determine whether the current image block (adjusted bounding box area) completely contains the "concrete pipe", i.e. the main part of the target (≥90%) is within the image block, without obvious cropping or missing. Requirements: The target is completely contained and the main part is clear, confidence ≥ 0.85; partially contained (main part missing <30%) or outline blurred, confidence = 0.5-0.8; not contained or main part missing ≥30%, confidence ≤ 0.45 and contains_complete=false.

[0225] The JSON output by the VLM model is: ```json {"contains_complete": true, "confidence": 0.89} ```。

[0226] Where "contains_complete" true means that the current image block includes the hoisting object, and "confidence" 0.89 means that the probability of the hoisting object existing in the current image block is 0.89, i.e. the confidence of the current third candidate box is 0.89.

[0227] S2540: Determine whether each retained second candidate box is complete.

[0228] If each retained second candidate box is complete, execute step S2550, otherwise return to step S2520 and select a second candidate box that is not complete to continue execution.

[0229] S2550: Output each third candidate box and its confidence and area.

[0230] In summary, the second embodiment of the hoisting object positioning method receives a user natural language scheduling instruction, and analyzes a hoisting object description through an LLM model; collects tower crane operation site images based on an industrial camera, and converts the images into a format meeting input requirements of a VLM model; in the third phase, a "full image coarse search, in-frame fine search, and boundary optimization" strategy is adopted for the source target, high-precision positioning is achieved through VLM multi-scale scanning; and the positioning results of the source target are integrated to generate structured data containing coordinates and confidence. The second embodiment of the hoisting object positioning method is suitable for various hoisting objects (such as reinforcing steel bars, containers, and cement pipes) in tower crane operations, can cover the full size range from "small size and scattered goods" (such as packed bricks) to "large size and heavy equipment" (such as containers and prefabricated boards), and is compatible with the positioning requirements of regular-shaped objects (such as steel pipes and cement columns) and irregular-shaped objects (such as construction waste and bulk steel), without the need to adjust model parameters due to changes in object types or shapes, with high flexibility in adapting to on-site operations, and low labor and computing power costs in technology landing.

[0231] The following will be described in combination with Figure 4 A first embodiment of a hoisting object positioning device is introduced.

[0232] The first embodiment of the hoisting object positioning device implements the first embodiment of the hoisting object positioning method, and has all the advantages thereof.

[0233] Figure 4 A structure of the first embodiment of the hoisting object positioning device is shown, which includes a full image coarse search module 410, an in-frame fine search module 420, and a boundary optimization module 430.

[0234] The full image coarse search module 410 is configured to slide a first sliding window in the on-site image of the tower crane operation to obtain a plurality of first candidate boxes; search for the hoisting object in the image block in each first candidate box by using the VLM model to obtain the confidence of each first candidate box and calculate the score of the first candidate box; and retain the first candidate box with the confidence greater than or equal to a first confidence value and the score greater than or equal to a first score value in the candidate pool. The working principle and advantages thereof can be referred to the step S110 of the first embodiment of the hoisting object positioning method.

[0235] The in-frame fine search module 420 is configured to slide a second sliding window in the image block in each first candidate box with the top N scores in the candidate pool to obtain a plurality of second candidate boxes; search for the hoisting object in the image block in each second candidate box by using the VLM model to obtain the confidence of each second candidate box and calculate the score of the second candidate box; and for each first candidate box with the top N scores, retain the second candidate box with the first score corresponding thereto greater than or equal to a second score value. The working principle and advantages thereof can be referred to the step S120 of the first embodiment of the hoisting object positioning method.

[0236] The boundary optimization module 430 is used to perform inward step-by-step shrinking of each edge of each retained second candidate box at the pixel level to obtain multiple third candidate boxes; in the image patch within each third candidate box, the VLM model is used to search for the hoisting object to obtain the confidence score of each third candidate box; the third candidate box with the best confidence score after shrinking the edges of each second candidate box is retained for the localization of the hoisting object. For its working principle and advantages, please refer to step S130 of Embodiment 1 of a hoisting object localization method.

[0237] The following is combined Figure 5 This paper introduces a second embodiment of a lifting object positioning device.

[0238] Embodiment 2 of a lifting object positioning device implements Embodiment 2 of a lifting object positioning method, and has all its advantages.

[0239] Figure 5 The structure of a lifting object positioning device according to an embodiment is shown, including: a target parsing module 510, an image conversion module 520, a full-image coarse search module 530, a box-based fine search module 540, a boundary optimization module 550, and a positioning and sorting module 560.

[0240] The target parsing module 510 is used to obtain the name of the object to be lifted using an LLM model based on user language instructions. For its working principle and advantages, please refer to step S210 of Embodiment 2 of a method for locating a lifted object.

[0241] The image conversion module 520 is used to acquire on-site images including the hoisted object and adapt them to preprocessed images supported by the VLM model. For its working principle and advantages, please refer to step S220 of Embodiment 2 of a hoisted object positioning method.

[0242] The full-image coarse search module 530 is used to slide a first sliding window in the preprocessed image to obtain multiple first candidate boxes; within each first candidate box, the VLM model is used to search for the hoisting object, obtain the confidence score of each first candidate box, and calculate the score of the first candidate box; first candidate boxes with a confidence score greater than or equal to the first confidence value and a score greater than or equal to the first score are retained in the candidate pool. For its working principle and advantages, please refer to step S230 of Embodiment 2 of a hoisting object localization method.

[0243] The in-frame fine search module 540 is configured to slide a second sliding window in each image block in a first candidate frame that is scored as a top N candidate in the candidate pool to obtain a second candidate frame, search for the hoisting object in each image block in the second candidate frame based on the name of the hoisting object by using the VLM model, retain, for each first candidate frame, a second candidate frame corresponding to the first candidate frame, which has a first score greater than or equal to a second score, and if there is no second candidate frame with a score greater than or equal to the second score, retain the second candidate frame with the highest score. For details of the working principle and advantages, refer to step S240 of the second hoisting object positioning method embodiment.

[0244] The boundary optimization module 550 is configured to perform pixel-level inward stepwise edge shrinking on each retained second candidate frame, obtain a third candidate frame after each stepwise edge shrinking of any edge, search for the hoisting object in each image block in the third candidate frame, and obtain a third candidate frame with an optimal confidence after the edge shrinking of each second candidate frame. For details of the working principle and advantages, refer to step S250 of the second hoisting object positioning method embodiment.

[0245] The positioning ranking module 560 is configured to rank the third candidate frame with the optimal confidence corresponding to each retained second candidate frame based on the area and the confidence of the candidate frame, and select a final candidate frame for hoisting object positioning based on the ranking according to the requirements of a positioning scene, so as to select a more suitable positioning frame for different scenes. For details of the working principle and advantages, refer to step S260 of the second hoisting object positioning method embodiment.

[0246] The embodiments of the present application also provide a computing device, which will be described below in detail. Figure 6

[0247] The computing device 600 includes a processor 610, a memory 620, a communication interface 630, and a bus 640.

[0248] It should be understood that the communication interface 630 in the computing device 600 shown in the figure can be used for communication between other devices.

[0249] The processor 610 can be connected with the memory 620. The memory 620 can be used for storing program codes and data. Therefore, the memory 620 can be an internal storage unit of the processor 610, can be an external storage unit independent of the processor 610, or can be a component including the internal storage unit of the processor 610 and the external storage unit independent of the processor 610.

[0250] ​Optionally, the computing device 600 can also include a bus 640. The bus 640 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 640 can be implemented by a number of buses and can be a combination of buses. The bus 640 can be a number of buses, such as address buses, data buses, control buses, etc. For the sake of simplicity, only one bus is shown in the figure, but it is understood that there can be more than one bus, and there can be different types of buses.

[0251] It should be understood that the processor 610 can be a central processing unit (CPU) in the embodiments of the present application. The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. Alternatively, the processor 610 can be one or more integrated circuits executing programs to implement the techniques described in the embodiments of the present application.

[0252] The memory 620 can include read-only memory and random access memory, and provide instructions and data to the processor 610. A portion of the memory 620 can also include non-volatile random access memory. For example, the processor 610 can also store device type information.

[0253] When the computing device 600 is running, the processor 610 executes computer-executable instructions in the memory 620 to perform the operational steps of the methods.

[0254] It should be understood that the computing device 600 according to the embodiments of the present application can correspond to a subject performing the methods according to the embodiments of the present application, and the above and other operations and / or functions of the various modules in the computing device 600 are respectively for implementing the corresponding processes of the methods of the embodiments, and for brevity, will not be repeated here.

[0255] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0257] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0259] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0260] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0261] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to perform the operation steps of the methods.

[0262] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0263] The computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a transmitted data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, transmit or transmit a program for use by or in conjunction with an instruction execution system, device or component.

[0264] The program code embodied on the computer readable media can be transmitted using any appropriate medium, including, but not limited to, wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0265] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0266] Note that the above merely describes the preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, reconfigurations and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and all of them belong to the protection scope of the present application.

Claims

1. A method for positioning an object lifted by a tower crane, characterized in that, include: In the image of the tower crane operation site, a first sliding window is slid to obtain multiple first candidate boxes. Within each first candidate box, a VLM model is used to search for the object to be lifted. The confidence score and score of each first candidate box are obtained, and the first candidate boxes with a confidence score greater than or equal to the first confidence value and a score greater than or equal to the first score are retained in the candidate pool. The score of each candidate box changes positively with the confidence score of the candidate box and negatively with the area of ​​the candidate box. The confidence score of each candidate box is the probability that there is a lifted object in the image block within the candidate box, as output by the VLM model. In each first candidate box within the image patch that ranks in the top N in the candidate pool, a second sliding window is slid to obtain multiple second candidate boxes. In each second candidate box within the image patch, the lifting object is searched using a VLM model, and the score of each second candidate box is calculated until the first second candidate box with a score greater than or equal to the second score is obtained, and the second score is greater than or equal to the first score. At the pixel level, each retained second candidate box is progressively shrunken inwards to obtain multiple third candidate boxes. Within each third candidate box, the image block is searched using a VLM model to find the hoisting object and obtain the confidence score of each third candidate box. The third candidate box with the best confidence score after shrunkening is retained and used to locate the hoisting object.

2. The method according to claim 1, characterized in that, Also includes: When retaining a first candidate box, if the intersection-union ratio (IU) of the first candidate box with any first candidate box in the candidate pool is greater than or equal to the IU threshold, the candidate box with the higher score among the two candidate boxes is retained in the candidate pool.

3. The method according to claim 1, characterized in that, The first sliding window includes multiple sliding windows with scale-step changes. During the process of obtaining multiple first candidate boxes by sliding the first sliding window in the scene image, the scale change range of the first sliding window is also adjusted, specifically including: A first range is obtained based on the area ratio of the hoisted object in the on-site image, and the first range is the initial range of the scale change of the first sliding window; The first sliding window is stepped within a first range. When the confidence of the first candidate box corresponding to any sliding position of the first sliding window at any scale within the first range is greater than or equal to the first confidence value, the range of change of the first sliding window is expanded to the second range. Otherwise, the range of change of the first sliding window is expanded to the third range, wherein the third range is greater than or equal to the second range. Wherein, the sliding step size of each first sliding window is the scale corresponding to the first proportion of the first sliding window, and the step size of the scale change of the first sliding window is the scale corresponding to the set proportion of the scene image.

4. The method according to claim 3, characterized in that, The second sliding window includes sliding windows with multiple scale steps; In the process of sliding a second sliding window in the image block within each first candidate box that scores in the top N in the candidate pool to obtain multiple second candidate boxes, the second sliding window is changed according to the step size of the second sliding window scale. For each second sliding window, the second sliding window is slid within the image block with its sliding step size to obtain multiple second candidate boxes corresponding to the second sliding window. Wherein, the sliding step size of each second sliding window is the scale corresponding to the second ratio of the second sliding window, and the second ratio is smaller than the first ratio; the step size of the second sliding window scale change is the length corresponding to the set ratio of the retained first candidate box.

5. The method according to claim 4, characterized in that, The process of obtaining a second candidate box until the first score is greater than or equal to the second score includes: In the candidate pool, if the score of a second candidate box obtained by sliding a second sliding window of any scale is greater than the second score, the search for the image patch in the first candidate box is stopped and the second candidate box is retained.

6. The method according to claim 5, characterized in that, Also includes: In the image patch within a first candidate box that ranks among the top N in the candidate pool, if the score of any second candidate box obtained by sliding the second sliding window at any scale is less than the second score, the second candidate box with the highest score in the image patch within that first candidate box is retained.

7. The method according to claim 6, characterized in that, Also includes: During the process of searching for the hoisting object using the VLM model in the image block within each second candidate box, for the second candidate box with a score greater than or equal to the third score, the four sides of the second candidate box are simultaneously adjusted by pixel-level shrinking and pixel-level expanding, and the third score is located between the first score and the second score. When the confidence scores of the original retained second candidate boxes and the two second candidate boxes after scaling adjustment are both greater than or equal to the second confidence value and the standard deviation is less than the set difference, the scores of the retained second candidate boxes remain unchanged. Otherwise, the four sides of the second candidate box after the edge shrinking adjustment are simultaneously adjusted again at the pixel level, and the four sides of the second candidate box after the edge expansion adjustment are simultaneously adjusted again at the pixel level; the average confidence score and average score of the original second candidate box, the two second candidate boxes after the first edge scaling adjustment, and the two second candidate boxes after the second edge scaling adjustment are used as the new confidence score and new score of the original second candidate box.

8. The method according to claim 1, characterized in that, The process of searching for the hoisting object using a VLM model in the image patch within each second candidate box includes: The VLM model is used to determine whether the image block within each second candidate box includes the hoisting object. When the image block within the second candidate box includes the hoisting object, the VLM model is used to determine whether there are any interfering objects related to the hoisting object in the image block within the second candidate box based on category and / or morphology.

9. The method according to claim 4, characterized in that, Before obtaining the second candidate box by sliding the second sliding window through the image patch within each of the top N first candidate boxes in the candidate pool, the process also includes: When there are M first candidate boxes with scores less than the fourth score in the first candidate box with scores in the top N in the candidate pool, slide the third sliding window in the image to obtain multiple fourth candidate boxes. In the image block within each fourth candidate box, use the VLM model to search for the hoisting object, calculate the score of each fourth candidate box, obtain the top M fourth candidate boxes with scores, and replace the M first candidate boxes. The fourth score is less than the second score. The sliding step size of the third sliding window is the scale corresponding to the third scale of the third sliding window, and the third scale is located between the first scale and the second scale. The third sliding window includes multiple sliding windows with step scales, and the range of variation of the third sliding window is less than or equal to that of the second range.

10. The method according to claim 1, characterized in that, When performing inward step-by-step shrinking of each retained second candidate box at the pixel level, if the confidence of the third candidate box obtained after any step-by-step shrinking of any edge decreases or is less than the set third confidence value compared to the confidence obtained before the current shrinking, the shrinking of that edge is stopped and the edge is restored to the state before the current step-by-step shrinking.

11. The method according to claim 8, characterized in that, The process of searching for a hoisting object in any image patch using a VLM model includes: The VLM model is input with the image patch, the description of the hoisting object, the description of the search task, and the output format corresponding to the task; the task includes at least one of the following: determining whether the hoisting object exists in the image patch, or determining whether there are interfering objects of the hoisting object in the image patch based on category and / or morphology; The VLM model performs a semantic search in the image patch according to the task and outputs the search results in the output format. When the task is to determine whether the hoisting object exists in the image patch, the output format includes the probability that the hoisting object exists in the image patch.

12. The method according to claim 1, characterized in that, The process of calculating the score for each candidate box includes: Obtain the confidence level of the candidate boxes output by the VLM model; The score of the candidate box is obtained based on the area of ​​the candidate box and the confidence level.

13. The method according to claim 1, characterized in that, Also includes: For each second candidate box, the third candidate box with the best confidence obtained after shrinking its edges is selected based on the comprehensive ranking of the candidate box area and confidence, and the final candidate box for locating the hoisting object is selected based on the ranking according to the specific scenario of the tower crane operation.

14. A device for positioning a lifting object, characterized in that, include: The full-image coarse search module is used to obtain multiple first candidate boxes by sliding a first sliding window in the tower crane operation site image. Within each first candidate box, the VLM model is used to search for the hoisting object in the image patch. The confidence score of each first candidate box is obtained, and the score of the first candidate box is calculated. First candidate boxes with a confidence score greater than or equal to the first confidence value and a score greater than or equal to the first score are retained in the candidate pool. The score of each candidate box changes positively with the confidence score and negatively with the area of ​​the candidate box. The confidence score of each candidate box is the probability that there is a hoisting object in the image patch within the candidate box, as output by the VLM model. The in-frame fine search module is used to slide a second sliding window in the image block of each first candidate box with a score of the top N in the candidate pool to obtain multiple second candidate boxes. In the image block of each second candidate box, the VLM model is used to search for the hoisting object and calculate the score of the second candidate box. The first second candidate box with a score greater than or equal to the second score is retained, and the second score is greater than or equal to the first score. The boundary optimization module is used to perform inward step-by-step shrinking of each retained second candidate box at the pixel level to obtain multiple third candidate boxes. In the image block within each third candidate box, the VLM model is used to search for the lifting object to obtain the confidence of each third candidate box. The third candidate box with the best confidence after shrinking the edge of each second candidate box is retained for the localization of the lifting object.

15. A computing device, characterized in that, include, bus; A communication interface, which is connected to the bus; At least one processor is connected to the bus; as well as At least one memory connected to the bus and storing program instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 13.

Images