Load tracking method, electronic equipment, crane and computer readable storage medium
By using video frames to track the minimum rectangular frame and contour of the load in the crane, the problem of unreal-time, accuracy and efficiency of load tracking in the prior art is solved, and long-term real-time tracking and efficient tracking of loads are achieved.
Patent Information
- Application Number
- CN202411553269.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-03-11
AI Technical Summary
Existing load tracking methods cannot track loads in real time for a long time, and are inaccurate and efficient.
By obtaining the current video frame of the crane target load, rectangular box tracking and mask tracking based on the minimum rectangular box and mask in the historical effective video frame, the minimum rectangular box and outline of the target load in the current video frame is determined, thereby real-time tracking of the load.
Real-time tracking of loads for a long time is realized, which reduces computing resource consumption and improves the accuracy and efficiency of load tracking.
Smart Images
Figure CN119600498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of engineering machinery, and in particular to a load tracking method, electronic equipment, a crane and a computer-readable storage medium. Background Art
[0002] When the crane is in operation, the operator needs to pay attention to the surrounding environment of the load at all times to prevent the load from colliding with surrounding obstacles, causing safety accidents or damage to the transported load.
[0003] However, existing load tracking methods cannot track the load in real time for a long time, and the accuracy and efficiency of load tracking are low. Summary of the invention
[0004] The purpose of the present application is to provide a load tracking method, electronic device, crane and computer-readable storage medium, which can realize real-time tracking of loads for a long time, while reducing the computing resources consumed by load tracking and improving the accuracy and efficiency of load tracking.
[0005] To achieve the above objectives:
[0006] In a first aspect, an embodiment of the present application provides a load tracking method, comprising:
[0007] Obtaining a current video frame captured of a target load of the crane;
[0008] When the current video frame is a non-target frame, performing rectangular frame tracking on the target load in the current video frame based on the minimum rectangular frame of the target load in the historical valid video frame to determine the minimum rectangular frame of the target load in the current video frame;
[0009] Based on the mask of the target load in the historical valid video frame, mask tracking is performed on the target load in the current video frame according to the current region of interest to obtain the contour of the target load in the current video frame;
[0010] A mask of the target load in the current video frame is determined according to a minimum rectangular frame and a corresponding outline of the target load in the current video frame.
[0011] Optionally, the method further comprises:
[0012] When the current video frame is a target frame, in response to a marking operation on the current video frame, taking a region marked by the marking operation in the current video frame as an initial region of interest;
[0013] The region of interest is segmented, and based on a minimum rectangular frame and a corresponding contour of the target load in the current video frame obtained by segmentation, a mask of the target load in the current video frame is determined.
[0014] Optionally, before the mask of the target load in the historical valid video frame is used to track the target load in the current video frame according to the current region of interest, and the contour of the target load in the current video frame is obtained, the method includes:
[0015] Check whether the last updated or determined region of interest meets the region of interest update condition;
[0016] If so, the target minimum rectangular frame is expanded in all directions with the center of the target minimum rectangular frame as the center point on the current video frame, and the obtained area is used as the updated current area of interest; the target minimum rectangular frame is the minimum rectangular frame of the target load in the previous valid video frame.
[0017] Optionally, it also includes:
[0018] If the last updated or determined region of interest does not meet the region of interest update condition, the last updated or determined region of interest is used as the current region of interest.
[0019] Optionally, the region of interest update condition includes at least one of the following:
[0020] The time since the last update or determination of the area of interest is greater than or equal to a preset time threshold;
[0021] It is detected that the contour of the target load in the last valid video frame is at the edge of the last updated or determined area of interest.
[0022] Optionally, determining the mask of the target payload in the current video frame according to the minimum rectangular frame and the corresponding outline of the target payload in the current video frame includes:
[0023] An intersection process is performed on the minimum rectangular frame information of the target load in the current video frame and the corresponding contour, and the obtained process result is determined as the mask of the target load in the current video frame.
[0024] Optionally, the method further comprises:
[0025] Obtaining a position deviation between a mask of the target payload in the current video frame and a mask of the target payload in a previous valid video frame;
[0026] When the position deviation meets a preset condition, the current video frame is determined as the valid video frame for tracking the target payload.
[0027] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing a computer program, and when the processor runs the computer program, the above-mentioned load tracking method is implemented.
[0028] In a third aspect, an embodiment of the present application provides a crane, comprising an image acquisition device arranged on the boom head of the crane, and the field of view of the image acquisition device covers the area below the boom head along the direction of gravity; and the electronic device described in the second aspect.
[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned load tracking method is implemented.
[0030] The load tracking method, electronic device, engineering machinery and computer-readable storage medium provided by the embodiment of the present application include: obtaining a current video frame shot of the target load of the crane; when the current video frame is a non-target frame, the target load in the current video frame is tracked based on the minimum rectangular frame of the target load in the historical valid video frame to determine the minimum rectangular frame of the target load in the current video frame; based on the mask of the target load in the historical valid video frame, the target load in the current video frame is masked and tracked according to the current region of interest to obtain the contour of the target load in the current video frame; according to the minimum rectangular frame of the target load in the current video frame and the corresponding contour, the mask of the target load in the current video frame is determined. In this way, the target load in the current video frame is tracked by the minimum rectangular frame of the target load in the historical valid video frame, combined with the mask tracking of the target load in the current video frame in the region of interest, so that the load can be tracked in real time for a long time, while reducing the computing resources consumed by the load tracking, and improving the accuracy and efficiency of the load tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A schematic diagram of a flow chart of a load tracking method provided by an embodiment of the present invention;
[0032] Figure 2 An interactive schematic diagram of a load tracking device provided by an embodiment of the present invention;
[0033] Figure 3 A schematic diagram of the processing process of the load tracking method provided by an embodiment of the present invention;
[0034] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0036] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.
[0037] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.
[0038] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0039] It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.
[0040] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0041] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0042] See also Figure 1 , is a load tracking method provided in an embodiment of the present application. The load tracking method can be executed by a load tracking device provided in an embodiment of the present application. The load tracking device can be implemented in software and / or hardware, such as an electronic device such as a controller or a processor. In this embodiment, the execution subject of the load tracking method is a processor in a crane as an example. The load tracking method provided in this embodiment includes:
[0043] Step S101: Acquire a current video frame shot for a target load of a crane.
[0044] Specifically, the current video frame shot for the target load of the crane may be acquired in real time or periodically.
[0045] It should be noted that an image acquisition device may be provided on the boom of the crane, and the field of view of the image acquisition device covers the area below the boom along the direction of gravity to ensure that the video frames corresponding to the target load of the crane can be acquired. The target load of the crane is the object hoisted by the crane, including but not limited to bricks, steel bars, wooden boards, etc., and the target load is located under the hook of the crane. The image acquisition device is used to acquire images of the target load of the crane, and may include, for example, an RGB camera, etc.
[0046] Step S102: when the current video frame is a non-target frame, the current video frame is tracked based on the minimum rectangular frame of the target load in the historical valid video frame to determine the minimum rectangular frame of the target load in the current video frame.
[0047] The target frame refers to the video frame that needs to manually mark the area of interest, such as the first video frame shot of the target load of the crane. The valid video frame refers to the video frame that can be used to track the target load. The minimum rectangular frame of the target load refers to the coordinate information of the minimum rectangle that can enclose the target load, which can be used to indicate the approximate position of the target load in the video frame.
[0048] It can be understood that, since the acquisition interval between adjacent video frames is short, the position change of the target load in adjacent video frames will be small, that is, the position of the target load in the current video frame will be relatively close to the position in multiple historical video frames before the current video frame, that is, the position of the target load in different video frames will show a certain regularity. Therefore, the current video frame can be tracked based on the minimum rectangular frame of the target load in the historical valid video frame to determine the minimum rectangular frame of the target load in the current video frame. It should be noted that, since the minimum rectangular frame of the target load can be determined by the four vertices corresponding to the minimum rectangular frame, determining the minimum rectangular frame of the target load in the current video frame can be regarded as determining the four vertices corresponding to the minimum rectangular frame of the target load in the current video frame.
[0049] In this embodiment, based on the minimum rectangular frame of the target load in the historical valid video frame, the current video frame can be tracked by a pre-trained global area rectangular frame tracking model to determine the minimum rectangular frame of the target load in the current video frame. The global area rectangular frame tracking model can be established by using a deep learning algorithm such as a deep neural network or a convolutional neural network, and the established global area rectangular frame tracking model is trained based on a training sample set (including image samples of the minimum rectangular frame marked with the load). The specific training process can refer to the prior art and will not be repeated here.
[0050] Step S103: Based on the mask of the target load in the historical valid video frame, the mask tracking is performed on the target load in the current video frame according to the current region of interest to obtain the contour of the target load in the current video frame.
[0051] The target load mask can be regarded as a set of pixel points constituting the target load, and the target load contour can be obtained based on the target load mask. The target load contour can be used to characterize the size and range of the target load in the video frame.
[0052] In this embodiment, based on the mask of the target load in the historical valid video frame, the current video frame can be mask tracked by a pre-trained local area mask tracking model to determine the contour of the target load in the current video frame. The local area mask tracking model can be established by using a deep learning algorithm such as a deep neural network or a convolutional neural network, and the established local area mask tracking model is trained based on a training sample set (including image samples marked with a mask of the load). The specific training process can refer to the prior art and will not be repeated here.
[0053] Step S103: determining a mask of the target load in the current video frame according to the minimum rectangular frame and the corresponding outline of the target load in the current video frame.
[0054] It can be understood that since the position of the target load in the current video frame can be obtained based on the outline of the target load, and the pixel information of the target load in the current video frame can be obtained based on the minimum rectangular frame of the target load, the mask of the target load in the current video frame can be determined based on the minimum rectangular frame and the corresponding outline of the target load in the current video frame.
[0055] In one embodiment, determining a mask of the target payload in the current video frame according to a minimum rectangular frame and a corresponding contour of the target payload in the current video frame includes:
[0056] The minimum rectangular frame of the target load in the current video frame and the corresponding contour are subjected to intersection processing, and the obtained processing result is determined as the mask of the target load in the current video frame.
[0057] It can be understood that since mask tracking uses tracking of the region of interest, it is impossible to obtain global information, and the contour of the target load obtained may have partial overflow and deviation, which can be regarded as noise. At the same time, the minimum rectangular frame of the target load usually contains complete information of the target load. Therefore, in order to accurately extract the mask of the target load in the video frame, the minimum rectangular frame of the target load in the current video frame and the corresponding contour can be processed by intersection, and the obtained processing result is determined as the mask of the target load in the current video frame. The mask of the target load includes all pixel point information that constitutes the target load, and the contour of the target load can be obtained based on the edge pixel point information that constitutes the target load.
[0058] In summary, in the load tracking method provided in the above embodiment, the target load in the current video frame is tracked by using the minimum rectangular frame of the target load in the historical valid video frame, combined with mask tracking of the target load in the current video frame in the area of interest, so that real-time tracking of the load for a long time can be achieved, while reducing the computing resources consumed by load tracking and improving the accuracy and efficiency of load tracking.
[0059] In one embodiment, the method further comprises:
[0060] When the current video frame is the target frame, in response to a marking operation on the current video frame, a region marked by the marking operation in the current video frame is used as an initial region of interest;
[0061] The region of interest is segmented, and based on the minimum rectangular frame and the corresponding contour of the target load in the current video frame obtained by segmentation, a mask of the target load in the current video frame is determined.
[0062] The target frame refers to a video frame that needs to manually mark the region of interest, such as the first video frame shot for the target load of the crane, i.e., the first video frame. When the current video frame is the target frame, the input marking operation for the current video frame can be received, and the region marked or selected by the marking operation in the current video frame is used as the initial region of interest, and then the region of interest in the current video frame is segmented to obtain the minimum rectangular frame and the corresponding contour of the target load in the current video frame, and finally the mask of the target load in the current video frame is determined based on the minimum rectangular frame and the corresponding contour of the target load in the current video frame obtained by segmentation.
[0063] It can be understood that when the current video frame is the target frame, since the current video frame may include other objects in addition to the target load, it is necessary to first mark the area where the target load to be tracked is located, that is, mark the initial region of interest. Here, the region of interest is segmented, and the preliminary outline of the target load is first segmented from the region of interest, and then the preliminary outline is screened and filtered, and the contour edge is smoothed by morphological processing to obtain the contour of the target load. Then, the four vertices of the minimum rectangular box of the target load can be determined based on the contour of the target load to obtain the minimum rectangular box of the target load. In this way, the minimum rectangular box and contour of the target load when the current video frame is the target frame can be quickly and accurately obtained, thereby improving the accuracy and efficiency of load tracking.
[0064] In one embodiment, based on the mask of the target load in the historical valid video frame, the mask tracking of the target load in the current video frame is performed according to the current region of interest, and before obtaining the contour of the target load in the current video frame, the method includes:
[0065] Check whether the last updated or determined region of interest meets the region of interest update condition;
[0066] If yes, the target minimum rectangular frame is expanded in all directions with the center of the target minimum rectangular frame as the center point on the current video frame, and the obtained area is used as the updated current region of interest; the target minimum rectangular frame is the minimum rectangular frame of the target payload in the previous valid video frame.
[0067] It can be understood that since the area of the region of interest is usually large and larger than the area corresponding to the minimum rectangular frame of the target load, considering that the acquisition time interval between different video frames is short, correspondingly, the position of the target load in different video frames is relatively close. At this time, based on the mask of the target load in the historical valid video frame, the mask tracking of the target load in the current video frame is performed according to the current region of interest. Before obtaining the contour of the target load in the current video frame, it can be detected whether the region of interest updated or determined last time meets the update condition of the region of interest. If it does, the center of the target minimum rectangular frame is used as the center point on the current video frame to expand around, and the obtained area is used as the updated current region of interest to achieve the update of the region of interest, and the target minimum rectangular frame is the minimum rectangular frame of the target load in the previous valid video frame. Among them, the expansion around the center of the target minimum rectangular frame on the current video frame as the center point can be based on the center of the target minimum rectangular frame on the current video frame as the center point, and the target minimum rectangular frame is enlarged by a preset multiple, so that the enlarged area is used as the updated current region of interest. For example, the target minimum rectangular frame can be enlarged by 1 or 1.5 times with the center of the target minimum rectangular frame as the center point on the current video frame, and the area corresponding to the enlarged target minimum rectangular frame is used as the updated current region of interest. In this way, the accuracy and efficiency of load tracking are further improved by timely updating the region of interest.
[0068] In one embodiment, the method further comprises:
[0069] If the last updated or determined region of interest does not meet the region of interest update condition, the last updated or determined region of interest is used as the current region of interest.
[0070] Specifically, if the last updated or determined ROI does not meet the ROI update condition, it means that the target payload in the current video frame can continue to be mask tracked according to the last updated or determined ROI, and the last updated or determined ROI is used as the current ROI.
[0071] Exemplarily, assuming that for the first video frame shot of the target load of the crane, the area selected by the marking operation is taken as the area of interest A, and the target load in the first video frame is tracked by mask using the area of interest A; for the second video frame shot of the target load of the crane, if the area of interest A does not meet the area of interest update condition, the target load in the second video frame is tracked by mask using the area of interest A; for the third video frame shot of the target load of the crane, if the area of interest A meets the area of interest update condition, it is necessary to first expand the third video frame with the center of the minimum rectangular box corresponding to the target load in the second video frame as the center point, and take the obtained area as the updated area of interest B, and use the area of interest B to track the target load in the third video frame by mask; for the fourth video frame shot of the target load of the crane, if the area of interest B does not meet the area of interest update condition, the target load in the fourth video frame is tracked by mask using the area of interest B, and so on.
[0072] In this way, if the last updated or determined region of interest does not meet the region of interest update conditions, the last updated or determined region of interest continues to be used as the current region of interest to perform mask tracking on the target load in the current video frame, which can reduce the computational operations required for frequent updates or determinations of the region of interest, and further improve the accuracy and efficiency of load tracking.
[0073] In one embodiment, the region of interest update condition includes at least one of the following:
[0074] The time since the last update or determination of the area of interest is greater than or equal to a preset time threshold;
[0075] It is detected that the outline of the target payload in the last valid video frame is at the edge of the last updated or determined area of interest.
[0076] Among them, the region of interest can be set to be updated periodically. When it is detected that the time length from the last update or determination of the region of interest is greater than or equal to the preset time length threshold, it indicates that the region of interest needs to be updated. The preset time length threshold can be set according to actual needs, such as 3 seconds, 10 seconds, etc. In practical applications, the preset time length threshold can be set in combination with the acquisition time interval of adjacent video frames. When it is detected that the contour of the target load in the last valid video frame is at the edge of the region of interest last updated or determined, it means that the contour of the target load may be about to be located outside the region of interest last updated or determined. At this time, the region of interest needs to be updated in time to avoid the phenomenon of incomplete contour information acquisition when continuing to use the last updated or determined region of interest to perform mask tracking on the target load in the video frame.
[0077] In one embodiment, the method further comprises:
[0078] Obtain the position deviation between the mask of the target payload in the current video frame and the mask of the target payload in the previous valid video frame;
[0079] When the position deviation meets a preset condition, the current video frame is determined as a valid video frame for tracking the target payload.
[0080] It can be understood that under normal circumstances, the positions of the masks of the target loads in adjacent video frames should be relatively close. However, due to the influence of external environmental factors, when a large deviation is detected in the position of the masks of the target loads in adjacent video frames, it means that the current video frame cannot be used as a valid video frame for tracking the target load. The position deviation can refer to the distance of the position, such as the distance between the center of the mask of the target load in the current video frame and the center of the mask of the target load in the previous valid video frame. The position deviation can be set according to the actual situation when it meets the preset conditions, such as the position deviation is less than the preset threshold. In this way, by selecting a valid video frame for tracking the target load, the target load can be effectively tracked, further improving the accuracy and efficiency of load tracking.
[0081] Based on the same inventive concept as the aforementioned embodiment, the load tracking method provided by the present embodiment is described below through a specific example. In this example, the minimum rectangular frame is used as the enclosing frame and the target load is used as the load target.
[0082] See also Figure 2 The load tracking device provided in this embodiment mainly includes: an image acquisition module, an image processing module, and a load target extraction and tracking module, wherein the image acquisition module transmits the acquired original image of the load to the image processing module, the image processing module transmits the processed image to the load target extraction and tracking module, and the load target extraction and tracking module outputs the extracted load information to the perception system so that the perception system can monitor and warn the load, etc.
[0083] See also Figure 3 The main implementation process of the load tracking method provided in this embodiment is as follows:
[0084] 1) Selection of regions of interest: By acquiring the video frame images transmitted by the image acquisition module on the crane boom, such as the camera, the area information of the load in the space below the crane and other surrounding objects can be obtained. In the first frame of the image, the load hoisted in the pre-operation needs to be roughly marked to obtain the region of interest of the load in the initial frame.
[0085] 2) Accurate extraction of load targets: For the region of interest of the load in the first frame image, the segmentation model is used to segment the region of interest to obtain the precise outline of the load target and the corresponding bounding box (i.e., position).
[0086] Here, ROI segmentation is mainly used to obtain the contour within the area, and the contour is filtered through features such as contour area and center position. The contour edge is smoothed through morphological processing, and finally the precise contour and bounding box information of the crane's load target are obtained.
[0087] 3) Global region bounding box tracking: Obtain the coordinate information of the bounding box of the payload target in the global image, and track the bounding box of subsequent frames. Here, more information features are obtained through global information, which can make the payload bounding box tracking more stable and efficient.
[0088] Here, an end-to-end visual object single target tracker can be used to extract and merge features through a mixed attention mechanism between the payload target of the previous frame and the current search area to obtain the correlation matching degree between the current area and the payload target. When the matching degree reaches the benchmark, the highest matching area is selected as the predicted bounding box for output.
[0089] 4) Local area mask tracking: Obtain the contour information of the mask of the payload target, expand it around the mask, and obtain a small local area centered on the mask for mask tracking of subsequent frames. Mask tracking in a small local area makes tracking more efficient.
[0090] When performing mask tracking in subsequent frames, the global area calculation requires more computing resources and a large amount of calculation leads to slow speed. Considering the rationality of the movement of the load target in the real-time frame, the mask and bounding box information of the previous frame is used to obtain the ROI area near it. Through semi-supervised segmentation, the appearance information such as color, texture, shape and motion information of the load target are combined to segment the load target in the ROI area. Through contour processing and smoothing and other operations, the precise contour of the load target in the ROI area is obtained.
[0091] 5) Region update and correction: In the subsequent frame tracking, if the payload target is close to the edge of the local small area (ROI), the ROI area is updated again with the bounding box of the payload target in the previous video frame as the center. Finally, the results are intersected based on bounding box tracking and mask tracking to obtain the final mask of the payload target. If the mask is empty, the mask is repaired using the candidate segmentation algorithm based on the bounding box, and the target position information is updated again through mask tracking.
[0092] In the subsequent frame tracking, a small ROI area is selected. If the payload target moves too much and the payload target extracted in the current frame is close to the edge of the ROI area, the ROI area is updated in time and the position status in the mask tracking is updated.
[0093] Since mask tracking uses local small area tracking and cannot obtain global information, the mask of mask tracking may cause partial overflow and deviation. It is corrected by tracking the bounding box to obtain the final mask information within the bounding box. At the same time, when the mask tracking is empty, the image segmentation of the information within the bounding box can be used to correct the mask tracking and reduce the frequency of frame loss.
[0094] 6) Global feedback correction: Detect the contour position information of the load target through the local area and obtain its corresponding information in the global image. Calculate the deviation between the position information of the load target obtained in the current frame and the previous video frame. When the difference is large or unreasonable, discard the current frame information and perform segmentation and load extraction correction through the position of the previous video frame. In other words, after obtaining the final mask and bounding box, match them with the mask and bounding box of the previous frame, including area, position and other difference matching. If the difference exceeds the threshold, discard the current frame information and re-update the position status of the mask tracking and bounding box tracking with the previous video frame information to ensure the correctness of the tracking.
[0095] In summary, the load tracking method of the crane provided in this embodiment has the following innovations: 1) Using mask tracking of the local ROI area, and timely updating the center of the ROI area and the target position information in the tracking model, which can not only reduce the resources consumed by the calculation, but also ensure the accuracy of the tracking mask of the subsequent load target. 2) Using the complementarity of multiple tracking and segmentation models. In the current frame tracking, they are predicted independently, and in the final calculation, they complement and correct each other, so that the tracking accuracy is higher and the tracking loss of the target load in the image frame is less. At the same time, since the bounding box calculation is faster, the global area is adopted for tracking, which can ensure the stability of the tracking; while the mask tracking calculation is slower, but the obtained contour is more refined. For the phenomenon that the local ROI area is adopted to obtain the accurate contour of the mask, but the local area may cause more frame loss, it is corrected by segmenting the model in the bounding box. The complementarity of the above multiple models, taking advantage of each other's strengths and weaknesses, can obtain better load target perception accuracy and speed. 3) Better robustness and fault tolerance. A variety of deep learning models or algorithms are used, including segmentation, bounding box tracking, and mask tracking, all of which use deep learning models. Through model training with its own data set, the application scope of this application is wide, and it can adapt to various crane load transportation scenarios, with better robustness and more intelligent. At the same time, the remedial measures of various methods make the system's fault tolerance better: when the mask tracking is lost, the load target segmentation in the bounding box is used for remedy; when the whole is lost, the previous frame information is used to timely correct the current frame, so that the system can quickly update and correct the error information. 4) Real-time, using multi-threaded concurrency processing and model acceleration processing, so that this application can perform long-term real-time tracking of load targets in embedded devices. Therefore, compared with the prior art, the present application can achieve a longer tracking time, and can track the load target in real time from the pre-operation state to the operation state to the operation completion state, that is, from more than ten minutes to several hours of real-time tracking of the load target, better assisting the system to perceive the target load in real time. That is to say, the present application has the following advantages: 1) The tracking speed is fast and the accuracy is high, and the operating load can be tracked and sensed in real time. This mainly depends on the complementarity between the calculation method of the local ROI area and the calculation of the global area, so that the present application can quickly and accurately obtain the accurate contour information of the crane load target in the video frame image. 2) It has a good fault tolerance mechanism, stable target tracking, and can reduce the frequency of target load tracking loss in the video frame image. Through remedial correction by various methods, the tracking error information in subsequent frame images can be quickly corrected. 3) There is less manual interaction. The operator only needs to roughly mark the crane target load at the beginning, and it can be automatically tracked in the subsequent process and warned by the perception system.
[0096] Based on the same inventive concept as the above embodiments, an embodiment of the present invention provides an electronic device, such as Figure 4As shown, the electronic device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 4 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 4 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the load tracking method described above is implemented.
[0097] The electronic device may also include: at least one network interface 312. The various components in the electronic device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 313 is not used in the following examples. Figure 4 Various buses are labeled as bus system 313.
[0098] The memory 311 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.
[0099] The memory 311 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer program used to operate on the electronic device, such as an operating system and an application; contact data; phone book data; messages; pictures; videos, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program may include various applications, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. Here, the program that implements the method of the embodiment of the present invention may be included in the application program.
[0100] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention provides a crane, comprising:
[0101] An image acquisition device is provided at the boom head of the crane, and the field of view of the image acquisition device covers the area below the boom head along the direction of gravity; and
[0102] The electronic device as described above is communicatively connected with the image acquisition device.
[0103] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer storage medium, in which a computer program is stored. The computer storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it may also be various devices including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer storage medium is executed by the processor, the load tracking method described above is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.
[0104] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0105] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.
[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A load tracking method, characterized in that: The method comprises: Obtaining a current video frame captured of a target load of the crane; When the current video frame is a non-target frame, performing rectangular frame tracking on the target load in the current video frame based on the minimum rectangular frame of the target load in the historical valid video frame to determine the minimum rectangular frame of the target load in the current video frame; Based on the mask of the target load in the historical valid video frame, mask tracking is performed on the target load in the current video frame according to the current region of interest to obtain the contour of the target load in the current video frame; A mask of the target load in the current video frame is determined according to a minimum rectangular frame and a corresponding outline of the target load in the current video frame.
2. The method according to claim 1, characterized in that The method further comprises: When the current video frame is a target frame, in response to a marking operation on the current video frame, taking a region marked by the marking operation in the current video frame as an initial region of interest; The region of interest is segmented, and based on a minimum rectangular frame and a corresponding contour of the target load in the current video frame obtained by segmentation, a mask of the target load in the current video frame is determined.
3. The method according to claim 1 or 2, characterized in that: The mask of the target load in the historical valid video frame is based on the mask of the target load, and before the mask tracking of the target load in the current video frame is performed according to the current region of interest to obtain the contour of the target load in the current video frame, it includes: Check whether the last updated or determined region of interest meets the region of interest update condition; If so, the target minimum rectangular frame is expanded in all directions with the center of the target minimum rectangular frame as the center point on the current video frame, and the obtained area is used as the updated current area of interest; the target minimum rectangular frame is the minimum rectangular frame of the target load in the previous valid video frame.
4. The method according to claim 3, characterized in that Also includes: If the last updated or determined region of interest does not meet the region of interest update condition, the last updated or determined region of interest is used as the current region of interest.
5. The method according to claim 3, characterized in that: The update condition of the region of interest includes at least one of the following: The time since the last update or determination of the area of interest is greater than or equal to a preset time threshold; It is detected that the contour of the target load in the last valid video frame is at the edge of the last updated or determined area of interest.
6. The method according to claim 1, characterized in that The determining, according to the minimum rectangular frame and the corresponding outline of the target payload in the current video frame, a mask of the target payload in the current video frame comprises: An intersection process is performed on the minimum rectangular frame of the target load in the current video frame and the corresponding contour, and the obtained process result is determined as the mask of the target load in the current video frame.
7. The method according to claim 1, characterized in that The method further comprises: Obtaining a position deviation between a mask of the target payload in the current video frame and a mask of the target payload in a previous valid video frame; When the position deviation meets a preset condition, the current video frame is determined as the valid video frame for tracking the target payload.
8. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the load tracking method according to any one of claims 1 to 7 is implemented.
9. A crane, characterized in that: include: An image acquisition device is arranged on the boom head of the crane, and the field of view of the image acquisition device covers the area below the boom head along the direction of gravity; as well as, The electronic device as claimed in claim 8.
10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the load tracking method according to any one of claims 1 to 7 is implemented.