Bucket tracking method, device and medium based on binocular camera

Abstract

The present application discloses a binocular camera-based bucket tracking method, device and medium, the method comprising: obtaining multiple frames of working images of the flow machine through a binocular camera arranged on the flow machine at preset time intervals; performing target detection on the bucket on the flow machine based on pixel differences between the multiple frames of working images, and determining the target positioning point of the bucket under different frames of working images and the three-dimensional coordinates corresponding to the target positioning point; determining at least one local tracking area corresponding to the bucket, the local tracking area being an unobstructed area in the bucket; for the local tracking area, connecting the target positioning points having the same corresponding three-dimensional coordinates under different frames of working images to obtain a motion trajectory corresponding to the local tracking area; predicting the target motion trajectory of the local tracking area based on the action direction and trajectory curvature of the motion trajectory, so as to track the bucket according to the target motion trajectory.

Description

Technical Field

[0001] The present application relates to the field of engineering machinery, and specifically to a binocular camera-based bucket tracking method, equipment, and medium. Background Art

[0002] The port logistics system is an important part of the logistics and transportation system. Port machinery, as the main tool for loading, unloading and transportation, has been widely used in the port logistics system.

[0003] Port machinery can be broadly divided into two categories: physical cranes and mobile cranes. Physical cranes, also known as large cranes, are used for stationary operations on the dock walls, such as bridge cranes, gantry cranes, and tire cranes. Mobile cranes are mobile machinery, such as forklifts, truck cranes, trailers, straddle carriers, and reach stackers. With the continuous development of mobile machinery, its application scope has also been further expanded.

[0004] However, due to the complex port construction environment, mobile machinery operations require real-time monitoring to ensure efficiency and safety. Currently, manual monitoring is often used to monitor the operating status of mobile machinery. However, this method only provides overall control of the operation of mobile machinery, making it difficult to accurately track specific parts of the machinery and resulting in low efficiency. Summary of the Invention

[0005] In order to solve the above problems, the present application proposes a binocular camera-based bucket tracking method, comprising: obtaining multiple frames of working images of the flow machine through a binocular camera set on the flow machine at preset time intervals;

[0006] Based on the pixel difference between multiple frames of working images, the bucket on the conveyor is detected, and the target positioning point of the bucket in different frames of working images and the corresponding three-dimensional coordinates of the target positioning point are determined;

[0007] Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket;

[0008] For the local tracking area, connect the target positioning points with the same 3D coordinates in different frames of the working image to obtain the motion trajectory corresponding to the local tracking area;

[0009] According to the motion direction and trajectory curvature of the motion trajectory, the target motion trajectory of the local tracking area is predicted so that the bucket can be tracked according to the target motion trajectory.

[0010] In one implementation of the present application, target detection is performed on the bucket of the conveyor according to the pixel difference between multiple frames of working images, specifically including:

[0011] For any two adjacent first operation images in the multiple frames of operation images, grayscale the first operation images to obtain a grayscaled second operation image;

[0012] Subtracting the pixel values ​​corresponding to the same pixel position in the second working image, and taking the absolute value of the result of the subtraction to obtain the pixel difference between the pixel positions;

[0013] Determining the pixel type of the pixel position according to the magnitude relationship between the pixel difference value and the first preset threshold value; the pixel type includes foreground pixel and background pixel;

[0014] The second working image is traversed, and the position corresponding to the bucket is determined according to the pixel type.

[0015] In one implementation of the present application, determining at least one local tracking area corresponding to the bucket specifically includes:

[0016] Determine the foreground target in the working image according to the foreground pixels; the foreground target includes the bucket and other foreground targets;

[0017] Determine whether there is overlap between other foreground objects and the bucket, and if so, remove the overlap from the bucket;

[0018] The bucket after removing the overlapping portion is segmented to obtain at least one local tracking region corresponding to the bucket.

[0019] In one implementation of the present application, the bucket is segmented after the overlapping portion is removed, specifically including:

[0020] Extract the contour corresponding to the bucket and mark the target positioning points on the contour;

[0021] Divide the contour into at least one region, and calculate the density of the target positioning points contained in the region;

[0022] According to the density, a target weight corresponding to the area is determined, so as to determine at least one local tracking area corresponding to the bucket according to the target weight.

[0023] In one implementation of the present application, predicting the target motion trajectory in the local tracking area based on the motion direction and trajectory curvature of the motion trajectory specifically includes:

[0024] Determining a plurality of trajectory points included in the motion trajectory and trajectory curvatures corresponding to the plurality of trajectory points, so as to generate a corresponding trajectory point data sequence according to the trajectory curvature;

[0025] According to the direction of the motion trajectory, from the trajectory point data sequence, determining the last trajectory point whose trajectory curvature exceeds a second preset threshold as the target trajectory point;

[0026] A tangent line is drawn with the target trajectory point as the tangent point, and the target motion trajectory in the local tracking area is predicted according to the direction of the tangent line.

[0027] In one implementation of the present application, tracking the bucket according to the target motion trajectory specifically includes:

[0028] Determine the local displacement information of the local tracking area according to the target motion trajectory;

[0029] Determine a relative positional relationship between the local tracking area and the bucket, where the relative positional relationship represents an offset relationship between a center of a first area and a center of a second area, where the center of the first area corresponds to the local tracking area and the center of the second area corresponds to the bucket;

[0030] The global displacement information of the bucket is determined according to the local displacement information and the relative position relationship, so as to track the bucket through the global displacement information.

[0031] In one implementation of the present application, before obtaining multiple frames of operation images of the flow machine through a binocular camera provided on the flow machine, the method further includes:

[0032] Generate a simulated video stream of the streaming machine, and superimpose a number of equidistant grids on the simulated video stream;

[0033] According to the distance between the equidistant grid and the ground surface, taking the ground surface as the starting point, the equidistant grid is locally densified in sequence to obtain the locally densified grid layer;

[0034] Obtaining surface elevation data, and determining whether there are any spatial obstacles within a preset range of the flow machine based on the surface elevation data;

[0035] For the locally densified grid layer, the grids to which the spatial obstacles belong are locally de-densified to obtain the final grid layer.

[0036] In one implementation of the present application, the method further includes:

[0037] Based on the operation image, determine the location of the material pile at the machine working site and extract the corresponding material pile contour;

[0038] Extending the tangent line of the motion trajectory until the tangent line intersects with the pile;

[0039] Determine multiple intersection points between the tangent line and the stockpile, and determine the corresponding operating area on the stockpile contour based on the intersection points;

[0040] Display the work area in a grid layer.

[0041] The embodiment of the present application provides a bucket tracking device based on a binocular camera, the device comprising:

[0042] at least one processor; and,

[0043] a memory communicatively connected to at least one processor; wherein,

[0044] The memory stores instructions executable by at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0045] At preset time intervals, a binocular camera installed on the flow machine is used to obtain multiple frames of operation images of the flow machine;

[0046] Based on the pixel difference between multiple frames of working images, the bucket on the conveyor is detected, and the target positioning point of the bucket in different frames of working images and the corresponding three-dimensional coordinates of the target positioning point are determined;

[0047] Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket;

[0048] For the local tracking area, connect the target positioning points with the same 3D coordinates in different frames of the working image to obtain the motion trajectory corresponding to the local tracking area;

[0049] According to the motion direction and trajectory curvature of the motion trajectory, the target motion trajectory of the local tracking area is predicted so that the bucket can be tracked according to the target motion trajectory.

[0050] An embodiment of the present application provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0051] At preset time intervals, a binocular camera installed on the flow machine is used to obtain multiple frames of operation images of the flow machine;

[0052] Based on the pixel difference between multiple frames of working images, the bucket on the conveyor is detected, and the target positioning point of the bucket in different frames of working images and the corresponding three-dimensional coordinates of the target positioning point are determined;

[0053] Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket;

[0054] For the local tracking area, connect the target positioning points with the same 3D coordinates in different frames of the working image to obtain the motion trajectory corresponding to the local tracking area;

[0055] According to the motion direction and trajectory curvature of the motion trajectory, the target motion trajectory of the local tracking area is predicted so that the bucket can be tracked according to the target motion trajectory.

[0056] The binocular camera-based bucket tracking method proposed in this application can bring the following beneficial effects:

[0057] By calculating the pixel difference between different working images, the bucket is detected to determine the corresponding position of the bucket, thereby achieving precise positioning of the bucket; through the current motion trajectory of the local tracking area, the future target motion trajectory of the bucket is predicted, and the bucket is tracked according to the target motion trajectory, which reduces the number of features that need to be identified, improves tracking efficiency, and effectively improves tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 A schematic flow chart of a binocular camera-based bucket tracking method provided in an embodiment of the present application;

[0060] Figure 2 A schematic structural diagram of a binocular camera-based bucket tracking device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, an embodiment of the present application provides a binocular camera-based bucket tracking method, comprising:

[0064] S101: According to a preset time interval, a binocular camera provided on the flow machine is used to obtain multiple frames of operation images of the flow machine.

[0065] A binocular camera enables real-time monitoring of moving targets and needs to be placed in a relatively fixed location. For example, during excavation operations, the mobile machine typically remains stationary, and a binocular camera can be placed there. Alternatively, a binocular camera can be placed at any location within the binocular camera's worksite that covers the entire working area of ​​the mobile machine. After the binocular camera is set up, the server can use the binocular camera to capture multiple frames of the mobile machine's operating images at preset intervals. This interval can be set based on specific application requirements and is not limited in this application.

[0066] S102: Target detection is performed on the bucket on the conveyor according to the pixel difference between the multiple frames of working images, and the target positioning point of the bucket in the different frames of working images and the three-dimensional coordinates corresponding to the target positioning point are determined.

[0067] When a mobile crane is operating, the bucket is always in motion. By performing object detection on the working image, the foreground object in the image can be separated and extracted from the background, thereby determining the specific position of the bucket. When the lighting at the work site does not vary much, the background similarity between two adjacent working images is relatively high. Therefore, the pixel difference between multiple working images can be used to detect the bucket.

[0068] Specifically, for any two adjacent first work images in the multi-frame work image, the first work image is grayscaled to obtain a grayscaled second work image. Then, the pixel values ​​corresponding to the same pixel position in the second work image are subtracted, and the absolute value of the subtraction result is taken to obtain the pixel difference between each pixel position. After obtaining the pixel difference, the pixel type of the pixel position is determined based on the size relationship between the pixel difference and the first preset threshold. Pixel types include foreground pixels and background pixels. When the pixel difference is less than the first preset threshold, the pixel position is a background pixel. When the pixel difference is greater than the first preset threshold, the pixel position is a foreground pixel. Finally, by traversing the second work image, the pixel type corresponding to each pixel position in the second work image can be determined, and the position of the bucket can be determined based on the pixel type. At this point, the target detection of the bucket is completed.

[0069] To facilitate subsequent bucket tracking, the detected bucket must be annotated with target positioning points. Specifically, the bucket tip is located based on the detected bucket target. Then, based on the distances between the bucket tip pixels and the bucket target pixels, corresponding target positioning points are annotated, and the corresponding 3D coordinates of each target positioning point are determined. It should be noted that the distance from the bucket tip determines the density of target positioning points; closer distances result in denser target positioning points.

[0070] S103: Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket.

[0071] After detecting the bucket in the work image, the tracking area is further narrowed to ensure accurate tracking of the bucket even when it is partially obscured. This narrows the tracking area to identify at least one local tracking region, which is the unobstructed area of ​​the bucket. This narrowing of the tracking region reduces the number of features required for tracking, improving tracking accuracy and preventing tracking failures when the bucket is obscured.

[0072] In one embodiment, determining at least one local tracking area corresponding to the bucket is determined based on whether the current bucket is partially obscured. First, based on the foreground pixels, the foreground targets in the work image are determined. The foreground targets include the bucket and other foreground targets other than the bucket. After obtaining the foreground targets, determining whether there is overlap between the other foreground targets and the bucket is performed. If there is overlap, it indicates that the bucket has been obscured by the other foreground targets, and the overlapping portion must be removed from the detected bucket target. After removing the overlapping portion, the bucket is subjected to regional segmentation to obtain at least one local tracking area corresponding to the bucket.

[0073] After removing overlapping portions, the bucket is segmented. This specifically includes the following steps: extracting the bucket's contour and marking the target points on the contour. Using a segmentation algorithm, the contour is segmented into at least one region. For example, an irregular bucket contour can be segmented into several regular and irregular regions. Segmentation ensures that the region contour is continuous. If the contour is discontinuous due to occlusion of the bucket, the endpoints of the contour must be determined and extended before segmentation to ensure contour continuity. After segmenting at least one region, the density of target points within each region is determined. Based on this density, a target weight is assigned to each region. When selecting local tracking regions, if multiple regions exist, the corresponding number of local tracking regions is selected in descending order of target weights. Furthermore, in addition to the density of target points, the area of ​​each region can also be considered when selecting local tracking regions. For regions with the same density, the larger region should be prioritized as the local tracking region. This improves local tracking accuracy.

[0074] S104: For the local tracking area, connect the target positioning points corresponding to the same three-dimensional coordinates in different frames of working images to obtain a motion trajectory corresponding to the local tracking area.

[0075] After obtaining at least one local tracking area corresponding to the bucket based on the above step S103, for each local tracking area, the same target positioning points corresponding to the same three-dimensional coordinates in multiple frames of working images are connected to obtain the motion trajectory corresponding to the local tracking area.

[0076] It should be noted that different target positioning points correspond to different trajectory curves, and the final motion trajectory of the bucket can be one or more of the above-mentioned trajectory curves, or it can be a trajectory curve formed by connecting the center points of the local tracking area under different frame operation images.

[0077] S105: Predicting a target motion trajectory of the local tracking area according to the motion direction and trajectory curvature of the motion trajectory, so as to track the bucket according to the target motion trajectory.

[0078] After obtaining the motion trajectory of the local tracking area, the bucket can be tracked in real time by predicting the bucket target motion trajectory.

[0079] Specifically, for each local tracking area, the multiple trajectory points contained in its corresponding motion trajectory are determined, and the trajectory points are distributed on the motion trajectory according to a preset distance. After the trajectory points are determined, the trajectory curvature corresponding to each trajectory point is determined, and based on the trajectory curvature, a corresponding trajectory point data sequence is generated. The trajectory point data sequence includes the position information and trajectory curvature of the trajectory point, and the order of the trajectory points is arranged according to the acquisition time of the operation image. After the trajectory point data sequence is generated, based on the action direction of the motion trajectory, the last trajectory point whose trajectory curvature exceeds the second preset threshold is determined from the trajectory point data sequence as the target trajectory point. The second preset threshold is used to represent the lower limit value of the deviation degree of the trajectory point. When the trajectory curvature of the trajectory point is greater than the second preset threshold, it can be predicted that the next trajectory point of the trajectory point has a turning intention, and the trajectory point with the largest trajectory curvature is the turning point. After determining the target trajectory point, a tangent line is drawn with the target trajectory point as the tangent point, and the target motion trajectory of the local tracking area is predicted based on the direction of the tangent line.

[0080] The target motion trajectory of the local tracking area reflects the tracking information of a portion of the bucket. To track the entire bucket, the local displacement information of the local tracking area must be determined based on the target motion trajectory. The relative position relationship between the local tracking area and the bucket is then determined. This relative position relationship represents the offset between the center of the first area and the center of the second area, where the center of the first area corresponds to the local tracking area and the center of the second area corresponds to the bucket. After determining the relative position relationship between the local tracking area and the bucket, the global displacement information of the bucket is determined based on this relative position relationship using the local displacement information of the local tracking area. This global displacement information allows for global tracking of the bucket.

[0081] By predicting the target motion trajectory in a local tracking area, the bucket is partially tracked, and global tracking of the bucket is achieved based on the relative positional relationship between the local tracking area and the bucket. This reduces the number of features to be recognized and, compared to directly tracking the bucket, small-scale tracking can further improve tracking accuracy.

[0082] During the bucket tracking process, the bucket's landing position can be predicted based on the bucket's target trajectory, and the landing position can be displayed on the remote video stream to indicate the bucket's operation.

[0083] Specifically, a simulated video stream of a flow machine is generated, and a number of equidistant grids are superimposed on the simulated video stream. According to the distance between the equidistant grids and the ground surface, the ground surface is used as the starting point, and the equidistant grids are locally densified in sequence to obtain a locally densified grid layer. It should be noted that the embodiment of the present application can densify the grid in a linearly related manner, such as a power function, a linear function, etc. After the equidistant grids are locally densified, the density between each grid gradually increases according to the distance from the ground surface. In this way, the grid layer is more in line with people's visual habits, and the precise positioning of the grids in the near-ground area is more accurate.

[0084] Furthermore, the surface elevation data is obtained, and then based on the surface elevation data, it is determined whether there are spatial obstacles within the preset range of the flow machine. If the surface elevation data is higher than the preset value, it means that there are spatial obstacles on the current ground. In the case of the presence of spatial obstacles on the surface, the grid to which the spatial obstacle belongs in the grid layer is determined, and the grid is locally de-densified to obtain the final grid layer. The local de-densification here is the same as the local encryption method, and can also be performed in a linearly related manner. The grid layer generated by the above-mentioned de-densification method can intuitively reflect the position of the spatial obstacles on the current surface, so that the flow machine can selectively avoid the obstacle gathering area to choose the appropriate travel direction and working orientation when operating, which is more convenient.

[0085] In one embodiment, after superimposing a grid layer on the video stream, the location of the material pile at the machine's work site can be determined based on the work image, and the corresponding material pile contour can be extracted. After extracting the material pile contour, the tangent line of the motion trajectory is extended until the tangent line intersects with the material pile. Since the target motion trajectory of the local tracking area can correspond to multiple curves, there will also be multiple intersections between its tangent line and the material pile. Based on this, the working area corresponding to the bucket can be determined on the material pile contour according to the intersection of the tangent line and the material pile. This working area can be displayed in real time on the grid layer superimposed on the remote video stream. In this way, when remotely controlling the bucket, the bucket's expected working area can be monitored in real time through the video stream, and when there is a large deviation between the working area and the material pile position, the bucket can be adjusted in time to improve the efficiency of remote control.

[0086] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0087] Figure 2 This is a schematic diagram of the structure of a binocular camera-based bucket tracking device provided in an embodiment of the present application. Figure 2 Shown, including:

[0088] at least one processor; and,

[0089] a memory communicatively connected to at least one processor; wherein,

[0090] The memory stores instructions executable by at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0091] At preset time intervals, a binocular camera installed on the flow machine is used to obtain multiple frames of operation images of the flow machine;

[0092] Based on the pixel difference between multiple frames of working images, the bucket on the conveyor is detected, and the target positioning point of the bucket in different frames of working images and the corresponding three-dimensional coordinates of the target positioning point are determined;

[0093] Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket;

[0094] For the local tracking area, connect the target positioning points with the same 3D coordinates in different frames of the working image to obtain the motion trajectory corresponding to the local tracking area;

[0095] According to the motion direction and trajectory curvature of the motion trajectory, the target motion trajectory of the local tracking area is predicted so that the bucket can be tracked according to the target motion trajectory.

[0096] An embodiment of the present application provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0097] At preset time intervals, a binocular camera installed on the flow machine is used to obtain multiple frames of operation images of the flow machine;

[0098] Based on the pixel difference between multiple frames of working images, the bucket on the conveyor is detected, and the target positioning point of the bucket in different frames of working images and the corresponding three-dimensional coordinates of the target positioning point are determined;

[0099] Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket;

[0100] For the local tracking area, connect the target positioning points with the same 3D coordinates in different frames of the working image to obtain the motion trajectory corresponding to the local tracking area;

[0101] According to the motion direction and trajectory curvature of the motion trajectory, the target motion trajectory of the local tracking area is predicted so that the bucket can be tracked according to the target motion trajectory.

[0102] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0103] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0104] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0110] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0112] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A bucket tracking method based on a binocular camera, characterized in that: The method comprises: At preset time intervals, a binocular camera provided on the flow machine is used to obtain multiple frames of operation images of the flow machine; Performing target detection on the bucket on the mobile machine based on the pixel difference between the multiple frames of working images, and determining the target positioning point of the bucket in different frames of working images and the three-dimensional coordinates corresponding to the target positioning point; Determine at least one local tracking area corresponding to the bucket, where the local tracking area is an unobstructed area in the bucket; For the local tracking area, connecting the target positioning points corresponding to the same three-dimensional coordinates in different frames of work images to obtain a motion trajectory corresponding to the local tracking area; predicting a target motion trajectory of the local tracking area according to the motion direction and trajectory curvature of the motion trajectory, so as to track the bucket according to the target motion trajectory; Target detection of the bucket on the mobile machine is performed based on pixel differences between the multiple frames of operation images, specifically including: For any two adjacent first operation images in the plurality of frames of operation images, grayscale the first operation images to obtain a grayscaled second operation image; Subtracting pixel values ​​corresponding to the same pixel position in the second working image, and taking an absolute value of the result of the subtraction to obtain a pixel difference value between the pixel positions; Determining the pixel type of the pixel position according to the magnitude relationship between the pixel difference value and a first preset threshold value; the pixel type includes a foreground pixel and a background pixel; Traversing the second operation image, and determining a position corresponding to the bucket according to the pixel type; Determining at least one local tracking area corresponding to the bucket specifically includes: Determining a foreground target in the working image according to the foreground pixel; the foreground target includes the bucket and other foreground targets; determining whether there is an overlapping portion between the other foreground objects and the bucket, and if so, removing the overlapping portion from the bucket; Performing region segmentation on the bucket after removing the overlapping portion to obtain at least one local tracking region corresponding to the bucket; Performing regional segmentation on the bucket after removing the overlapping portion specifically includes: Extracting the contour corresponding to the bucket and marking the target positioning point on the contour; Segment the contour into at least one region, and for the region, calculate the density of target positioning points contained in the region; determining a target weight corresponding to the area according to the density, and determining at least one local tracking area corresponding to the bucket according to the target weight; Predicting the target motion trajectory of the local tracking area according to the motion direction and trajectory curvature of the motion trajectory specifically includes: determining a plurality of trajectory points included in the motion trajectory and trajectory curvatures corresponding to the plurality of trajectory points, so as to generate a corresponding trajectory point data sequence according to the trajectory curvature; According to the direction of the motion trajectory, from the trajectory point data sequence, determining the last trajectory point whose trajectory curvature exceeds a second preset threshold as the target trajectory point; Draw a tangent line with the target trajectory point as a tangent point, and predict the target motion trajectory of the local tracking area according to the direction of the tangent line; Before obtaining multiple frames of operation images of the flow machine through a binocular camera provided on the flow machine, the method further includes: generating a simulated video stream of a streaming machine, and superimposing a plurality of equidistant grids on the simulated video stream; According to the distance between the equidistant grid and the ground surface, taking the ground surface as the starting point, locally densifying the equidistant grid in sequence to obtain a locally densified grid layer; Acquiring surface elevation data, and determining whether there are any spatial obstacles within a preset range of the flow machine based on the surface elevation data; For the locally densified grid layer, the grid to which the spatial obstacle belongs is locally de-densified to obtain a final grid layer.

2. The binocular camera-based bucket tracking method according to claim 1, characterized in that: Tracking the bucket according to the target motion trajectory specifically includes: Determining local displacement information of the local tracking area according to the target motion trajectory; Determining a relative positional relationship between the local tracking area and the bucket, the relative positional relationship representing an offset relationship between a center of a first area and a center of a second area, the center of the first area corresponding to the local tracking area, and the center of the second area corresponding to the bucket; The global displacement information of the bucket is determined according to the local displacement information and the relative position relationship, so as to track the bucket through the global displacement information.

3. The binocular camera-based bucket tracking method according to claim 1, characterized in that: The method further comprises: Determine the location of a material pile at the machine working site based on the operation image, and extract a material pile contour corresponding to the material pile; Extending a tangent line of the motion trajectory until the tangent line intersects the stockpile; determining a plurality of intersection points between the tangent line and the stockpile, and determining corresponding operating areas on the stockpile contour according to the intersection points; The operation area is displayed in the grid layer.

4. A bucket tracking device based on a binocular camera, characterized in that the device include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the binocular camera-based bucket tracking method as described in any one of claims 1-3.

5. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are configured to: A binocular camera-based bucket tracking method according to any one of claims 1 to 3.

Images