Discharging method and device of harvesting machine, electronic equipment and storage medium

Abstract

The invention provides an unloading method and device of a harvesting machine, electronic equipment and a storage medium. The unloading method comprises the steps that a depth map, collected by a binocular camera, of a material receiving and transporting vehicle is obtained; based on the depth map, determining coordinates of each pixel point forming the material receiving and transporting vehicle in the depth map under a camera coordinate system; generating a three-dimensional point cloud corresponding to each pixel point in a vehicle body coordinate system based on the coordinate of each pixel point forming the material receiving and transporting vehicle in the depth map in a camera coordinate system; based on the three-dimensional point cloud, determining a region of interest of a car hopper of the material receiving and transporting car; dividing a plurality of grids on the basis of the region of interest, and determining the height of a material in a region corresponding to each grid in the plurality of grids in the car hopper; adjusting a spray cylinder of the harvesting machine based on the height of the material and the observation points, and adjusting a baffle at the tail end of the spray cylinder of the harvesting machine based on the distance between the harvesting machine and the material receiving and transporting vehicle. By adopting the technical scheme provided by the invention, the driving safety and the driving comfort are improved.

Description

Technical Field

[0001] The present application relates to the technical field of unloading, and in particular to an unloading method, device, electronic equipment and storage medium for a harvesting machine. Background Art

[0002] Harvesters produce straw and other materials when they complete operations such as crop harvesting and threshing, and material transport vehicles are responsible for receiving and transporting the materials. During the harvesting process of traditional harvesters, the operator needs to manually operate buttons to adjust the rotation of the spray barrel and the extension and retraction of the baffle at the end of the spray barrel to adapt to different material transport vehicles and driving speeds. In order to improve efficiency, some practitioners let the operator drive exclusively and send another operator to operate remotely outdoors.

[0003] However, the manual operation of the buttons by the operator is inefficient and poses a safety hazard to inexperienced operators. At the same time, some operators use outdoor remote control, which is more dangerous. Therefore, how to unload materials efficiently and safely has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, device, electronic device and storage medium for unloading a harvesting machine, which improves driving safety and driving comfort.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a method for unloading a harvester, wherein a binocular camera is provided at the bottom of a spray barrel of the harvester, and the binocular camera is used to collect a depth map of a scene within a camera field of view, and the unloading method comprises:

[0007] Obtaining a depth map of the material transport vehicle captured by the binocular camera;

[0008] Based on the depth map, determine the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system;

[0009] Based on the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system, a three-dimensional point cloud corresponding to each pixel in the vehicle body coordinate system is generated;

[0010] Based on the three-dimensional point cloud, determine the region of interest of the bucket of the material transport vehicle; wherein the region of interest is the material area in the bucket;

[0011] Divide the region of interest into multiple grids based on the region of interest, and determine the height of the material in the region corresponding to each of the multiple grids in the vehicle box;

[0012] Based on the height of the material and multiple observation points, the spray barrel of the harvester is adjusted, and based on the distance between the harvester and the material transport vehicle, the rear end baffle of the spray barrel of the harvester is adjusted.

[0013] Furthermore, determining the region of interest of the bucket of the material transport vehicle based on the three-dimensional point cloud includes:

[0014] Selecting a target three-dimensional point cloud within a preset distance range from the harvesting machine from the three-dimensional point cloud;

[0015] Based on the target three-dimensional point cloud, determining the outer surface of the material transport vehicle;

[0016] Based on the outer surface, determining the coordinates of at least one edge point of the bucket in the three-dimensional point cloud;

[0017] Based on the coordinates of the at least one edge point in the three-dimensional point cloud, determining the coordinates of a plurality of initial reference positioning points of the bucket in the three-dimensional point cloud;

[0018] Based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determining a straight line where the bucket is close to the upper boundary of the harvesting machinery;

[0019] The straight line where the upper boundary of the bucket is close to the harvester is moved by a preset distance along a set direction to obtain a straight line where the boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle;

[0020] The region of interest of the bucket is determined based on a straight line where the upper boundary of the bucket is close to the harvesting machine and a straight line where the upper boundary of the bucket is far from the harvesting machine.

[0021] Further, the determining of the straight line where the bucket approaches the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud includes:

[0022] Convert the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system;

[0023] Fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line;

[0024] The slope of the initial straight line and the slope of the historical straight line are weighted to obtain the slope of the straight line where the bucket is close to the upper boundary of the harvesting machinery; wherein the historical straight line is a straight line obtained by fitting the coordinates of multiple historical reference positioning points of the bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system;

[0025] Perform weighted processing on the intercept of the initial straight line and the intercept of the historical straight line to obtain the intercept of the straight line where the upper boundary of the hopper approaches the harvesting machine;

[0026] Based on the slope and intercept of the straight line where the upper boundary of the hopper approaches the harvesting machine, obtain the straight line where the upper boundary of the hopper approaches the harvesting machine.

[0027] Further, the dividing a plurality of grids based on the region of interest and determining the height of the material in each grid corresponding region within the hopper includes:

[0028] Based on the region of interest, remove the noise point cloud outside the hopper from the three-dimensional point cloud to obtain the material point cloud of the hopper;

[0029] Map each point in the material point cloud to a two-dimensional plane to obtain the projection points of the material within the hopper on the two-dimensional plane;

[0030] Divide the region where the projection points are located into a plurality of grids;

[0031] Based on the vertical coordinate of each point in the material point cloud, determine the height of the material corresponding to the projection point after the point is mapped to the two-dimensional plane;

[0032] For each grid among the plurality of grids, perform weighted processing on the average value of the material heights corresponding to the projection points included in this grid and the average value of the material heights corresponding to the projection points included in each of the surrounding grids of this grid to determine the height of the material in this grid;

[0033] Determine the height of the material in this grid as the height of the material in the corresponding region within the hopper of this grid;

[0034] Determine the height of the material in each grid corresponding region within the hopper.

[0035] Further, the adjusting the nozzle of the harvesting machine based on the height of the material and a plurality of observation points includes:

[0036] Confirm the landing point of the material ejected by the nozzle of the harvesting machine as the initial observation point;

[0037] Based on the coordinates of the initial observation point in the three-dimensional point cloud and a preset step size, determine the coordinates of a plurality of observation points in the three-dimensional point cloud;

[0038] Based on the height of the material, determine the coordinates of the observation point corresponding to the lowest material height among a plurality of preset observation points in the three-dimensional point cloud;

[0039] Determine the difference between the abscissa of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the abscissa of the landing point of the material ejected by the nozzle of the harvester in the three-dimensional point cloud;

[0040] Determine the difference between the ordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the ordinate of the landing point of the material ejected by the nozzle of the harvester in the three-dimensional point cloud;

[0041] Based on the difference in the abscissa and the difference in the ordinate, adjust the nozzle of the harvester.

[0042] In a second aspect, an embodiment of the present application further provides a discharging device for a harvesting machine, and the discharging device includes:

[0043] An acquisition module that acquires a depth map of the material receiving vehicle collected by the binocular camera;

[0044] A coordinate conversion module that, based on the depth map, determines the coordinates of each pixel point that makes up the material receiving vehicle in the camera coordinate system;

[0045] A point cloud generation module that, based on the coordinates of each pixel point that makes up the material receiving vehicle in the camera coordinate system in the depth map, generates a three-dimensional point cloud corresponding to each pixel point in the vehicle body coordinate system;

[0046] An interested region extraction module that, based on the three-dimensional point cloud, determines an interested region of the hopper of the material receiving vehicle; wherein, the interested region is the material region within the hopper;

[0047] A material height calculation module that, based on the interested region, divides a plurality of grids and determines the height of the material in the corresponding region of each grid within the hopper;

[0048] A nozzle and baffle control module that adjusts the nozzle of the harvesting machine based on the height of the material and a plurality of observation points, and adjusts the baffle at the end of the nozzle of the harvesting machine based on the distance between the harvesting machine and the material receiving vehicle.

[0049] Further, the interested region extraction module is specifically configured to:

[0050] Select target three-dimensional point clouds within a preset distance range from the harvesting machine from the three-dimensional point cloud;

[0051] Based on the target three-dimensional point cloud, determine the outer surface of the material receiving vehicle;

[0052] Based on the outer surface, determine the coordinates of at least one edge point of the hopper in the three-dimensional point cloud;

[0053] Based on the coordinates of the at least one edge point in the three-dimensional point cloud, determining the coordinates of a plurality of initial reference positioning points of the bucket in the current frame in the three-dimensional point cloud;

[0054] Based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determining a straight line where the bucket is close to the upper boundary of the harvesting machinery;

[0055] The straight line where the upper boundary of the bucket is close to the harvester is moved by a preset distance along a set direction to obtain a straight line where the boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle;

[0056] The region of interest of the bucket is determined based on a straight line where the upper boundary of the bucket is close to the harvesting machine and a straight line where the upper boundary of the bucket is far from the harvesting machine.

[0057] Further, when the region of interest extraction module is used to determine the straight line where the bucket approaches the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, it is also specifically used to:

[0058] Convert the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system;

[0059] Fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line;

[0060] The slope of the initial straight line and the slope of the historical straight line are weighted to obtain the slope of the straight line where the bucket is close to the upper boundary of the harvesting machinery; wherein the historical straight line is a straight line obtained by fitting the coordinates of multiple historical reference positioning points of the bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system;

[0061] The intercept of the initial straight line and the intercept of the historical straight line are weighted to obtain the intercept of the straight line where the bucket is close to the upper boundary of the harvesting machinery;

[0062] Based on the slope and intercept of the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is located, the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is obtained.

[0063] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the unloading method of the harvesting machinery described in the first aspect or any possible implementation manner of the first aspect.

[0064] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the unloading steps of the harvesting machinery described in the first aspect or any possible implementation manner of the first aspect are executed.

[0065] The embodiments of the present application provide a method, device, electronic device and storage medium for unloading a harvesting machine. First, a binocular camera is used to collect a depth map of a material transport vehicle. Then, the coordinates of each pixel constituting the material transport vehicle in the camera coordinate system are determined based on the depth map. Then, these coordinates are converted to generate a corresponding three-dimensional point cloud in the vehicle body coordinate system. Secondly, the material area in the bucket of the material transport vehicle is determined as the area of ​​interest based on the three-dimensional point cloud, and then the height of the material in the bucket is determined based on the area. Finally, the spray barrel of the harvesting machine and its tail baffle are adjusted based on the height of the material in the bucket, multiple observation points and the distance between the harvesting machine and the material transport vehicle. In this way, driving safety and driving comfort are improved.

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 One of the flow charts of a method for unloading a harvesting machine provided in an embodiment of the present application is shown;

[0069] Figure 2 An example of a three-dimensional point cloud corresponding to each pixel point constituting the material transport vehicle in the camera coordinate system is shown;

[0070] Figure 3 An example of a three-dimensional point cloud corresponding to each pixel point constituting the material transport vehicle in the vehicle body coordinate system is shown;

[0071] Figure 4 A second flow chart of a method for unloading a harvesting machine provided in an embodiment of the present application is shown;

[0072] Figure 5 An example diagram of initial reference positioning points is shown;

[0073] Figure 6 A third flow chart of a method for unloading a harvesting machine provided in an embodiment of the present application is shown;

[0074] Figure 7 A fourth flowchart of a method for unloading a harvesting machine provided in an embodiment of the present application is shown;

[0075] Figure 8 An example diagram of a material point cloud is shown;

[0076] Fig. 9 An example diagram of material distribution is shown;

[0077] Fig.10 A fifth flow chart showing a method for unloading a harvesting machine provided in an embodiment of the present application is shown;

[0078] Fig.11 A mathematical modeling diagram of a tail baffle of a spray barrel of a harvesting machine is shown;

[0079] Fig.12 An example diagram of the identification result of a material transport vehicle is shown;

[0080] Fig.13 A schematic structural diagram of a unloading device of a harvesting machine provided in an embodiment of the present application is shown;

[0081] Fig.14 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0082] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0083] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0084] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where harvesting machinery needs to unload. The embodiments of the present application are not limited to specific application scenarios. Any scheme of the unloading method and device of the harvesting machinery provided by the embodiments of the present application is within the scope of protection of the present application.

[0085] It is worth noting that when harvesting and threshing crops, harvesters produce materials such as straw, and the material transport vehicle is responsible for taking over and transporting the materials. During the harvesting process of traditional harvesters, the operator needs to manually operate the buttons to adjust the rotation of the spray barrel and the extension and retraction of the baffle at the tail end of the spray barrel to adapt to different material transport vehicles and driving speeds. In order to achieve efficiency, some practitioners let the operator focus on driving and send another operator to operate remotely outdoors. However, the method of manually operating buttons by the operator is inefficient and poses a safety hazard to inexperienced operators. At the same time, some practitioners use outdoor remote control, which has a higher risk factor. Therefore, how to unload materials efficiently and safely has become an urgent problem to be solved.

[0086] In response to the above problems, the embodiments of the present application propose a unloading method, device, electronic device and storage medium for a harvesting machinery, which improves driving safety and driving comfort.

[0087] To facilitate the understanding of the present application, the technical solution provided by the present application is described in detail below in conjunction with specific embodiments.

[0088] See also Figure 1 , Figure 1 This is one of the flow charts of a method for unloading a harvesting machine provided in an embodiment of the present application.

[0089] As examples, harvesting machines may include, but are not limited to, forage harvesters, cotton harvesters, and sugar cane harvesters.

[0090] In order to improve the automation level of agricultural machinery operations, more and more developers are choosing to incorporate machine vision technology into the design of agricultural machinery algorithms. Through vision technology, the system can automatically identify the material transport vehicle, and then determine the best grain throwing position through intelligent decision-making, and finally realize automatic unloading.

[0091] In recent years, deep learning-based methods have been widely used in the agricultural field, such as the YOLO target detection network and the U-Net segmentation network. These technologies are usually used as the preferred algorithms for identifying crops and field obstacles in agricultural scenarios. For the problem of unloading grain from harvesting machinery, the segmentation network can theoretically be used to accurately divide the unloading target and material area. However, deep learning methods have high requirements for training data, requiring not only a large amount of data, but also a rich and diverse data type. The material transport vehicles used in China are of various shapes, with more than three main models, including high-sided vehicles, muck trucks, and tractor buckets. In addition, these material transport vehicles are often equipped with various baffles and baffles, such as camouflage baffles, which puts forward high requirements for the diversity of data sets. In addition, the domestic harvest season is relatively concentrated, the annual operation season is short, and it is also easily affected by rainy weather during this period. The above factors increase the difficulty of data set preparation and training, and raise the threshold of deep learning technology in the application of automatic unloading of harvesting machinery. In addition, the recognition effect of deep learning methods is easily disturbed by the dynamic background of vehicle driving and interference factors such as splashing grain, and the overall recognition stability is poor. In this application scenario, if a deeper network structure is used and the number of training rounds is increased, the network's understanding ability can be enhanced to a certain extent, but overfitting is very likely to occur. Ultimately, although the deep learning method is theoretically feasible in this scenario, it performs poorly in actual testing.

[0092] Traditional image processing methods are highly interpretable, and through reasonable algorithm design, they can accurately capture the direct and effective features of the target detection object. Compared with deep learning methods, traditional image processing methods are more computationally efficient and consume less resources, so their real-time processing capabilities are more prominent. Based on these characteristics, this method can be deployed on low-cost edge devices and is more suitable for promotion and implementation. Traditional matching algorithms in traditional image processing methods have the problem of poor imaging quality. Even if algorithms such as hole filling and spatial filtering are applied to optimize the processing, the generated images often still have a lot of noise. Unlike lidar, it cannot be directly converted into a three-dimensional point cloud for processing. Therefore, a binocular evaluation network is used in this application, that is, various image processing is performed on the depth map through image processing algorithms to complete feature extraction. The binocular evaluation network reduces noise and depth value errors.

[0093] like Figure 1 As shown in the unloading method of the harvester provided in the embodiment of the present application, a binocular camera is arranged at the bottom of the spray barrel of the harvester, and the binocular camera is used to collect a depth map of the scene within the camera field of view. The unloading method of the harvester includes the following steps:

[0094] Step S101, obtaining a depth map of the material transport vehicle captured by the binocular camera.

[0095] Here, the binocular camera can simulate human binocular vision and provide depth information for target identification. The binocular camera can not only collect color images, identify targets based on color, texture and other features, but also obtain disparity maps. By using stereo matching technology to capture the distribution of materials in the bucket of the material transport vehicle, it can assist in judging interference objects such as grain columns and splashes, thereby reducing the probability of misjudgment of the full distribution of grain in the bucket of the material transport vehicle. Among them, the grain column is the material that is ejected from the spray barrel of the harvesting machinery during the operation. On this spatial trajectory from the nozzle to the point where the material falls, many material particles form a column-like material collection form; splashes are the phenomenon that during the process of material ejection from the spray barrel, due to the collision between the inner wall of the bucket, the material particles, or the influence of factors such as air resistance, some materials are splashed outside the normal accumulation area in the bucket.

[0096] Here, the installation position and related parameters of the binocular camera are determined by the length of the material transport vehicle. As an example, the material transport vehicle used with the silage machine has a body length of more than 6 meters. To ensure that the material transport vehicle will not be frequently lost in the camera's field of view, the material transport vehicle and binocular camera are modeled and analyzed to obtain the following: the camera's installation height from the ground is greater than 5 meters, the camera's horizontal field of view angle is greater than 100 degrees, the camera's vertical field of view angle is greater than 80 degrees, and the camera's focal length is 2.1 mm.

[0097] Before step S101, the unloading method of the harvester also includes: when it is determined to enter the automatic mode, performing a heartbeat detection on the communication between preset components.

[0098] Here, as an example, the preset components include: visual main control unit, communication unit, perception component, main screen, sub-screen, spray gun arm, spray gun tail baffle, electronically controlled hydraulic valve and push rod, etc. The perception components include: binocular camera and multiple angle sensors, etc. Heartbeat detection is to check the communication between the component and the visual main control unit during the period when the automatic mode is turned on and in operation, to ensure the normal operation of the function and the safety of assisted driving. Among them, the main screen is the original display screen in the cab, which provides users with various parameter setting items of the automatic mode (such as sensitivity); the sub-screen is an additional screen with only display function, which is used to display the real-time screen with perception decision information through the network port. On the one hand, the communication unit provides CAN bus communication between the visual main control unit and the vehicle controller, and on the other hand, it provides network port push service for the sub-screen through the standard network port or the vehicle Ethernet port to ensure that the sub-screen can obtain and display the corresponding picture information in real time.

[0099] Here, in order to fully ensure the safety of users when using automatic mode, a two-way switching logic between manual mode and automatic mode is designed. In manual mode, users can manually control the spray gun through the remote control and the handle set on the armrest box. If you switch to automatic mode, long press the operation button on the handle set on the armrest box for more than the preset time to complete the switch. As an example, the preset time is 2 seconds. After entering the automatic mode, the user does not need to manually operate the rotation of the spray gun and the extension and retraction of the baffle at the tail end of the spray gun. You only need to pay attention to the alarm tone during driving and regularly check the recognition effect of the sub-screen in the cab. When encountering difficult scenarios (such as the material transport vehicle model does not meet the set standards) or extreme weather, the user only needs to tap any manual operation button, and the system will immediately exit the automatic mode and return to the manual operation state, thereby completing the entire manual override logic.

[0100] In this application, when the automatic mode is turned on and in operation, if the information recognized by the sensing component does not match the expected information, the system will generate a corresponding fault code. At the same time, the system will send a corresponding text reminder message to the main screen or the secondary screen and trigger a buzzer alarm. As an example, if the depth map of the binocular camera is processed and there is no material or no material transport truck bucket, the system will generate a corresponding fault code.

[0101] Step S102: Based on the depth map, determine the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system.

[0102] Specifically, based on the parallax between the two imaging units of the binocular camera, the Z-direction depth value of each pixel constituting the material transport vehicle in the depth map is obtained, and based on the Z-direction depth value and the camera internal parameter matrix, the X-direction and Y-direction depth values ​​of each pixel constituting the material transport vehicle in the depth map are obtained. Through the above method, the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system are obtained.

[0103] Step S103, based on the coordinates of each pixel constituting the material handling vehicle in the depth map in the camera coordinate system, generate a three-dimensional point cloud of each pixel in the vehicle body coordinate system.

[0104] Here, the three-dimensional point cloud can intuitively present the geometric features of an object. For example, information such as the surface undulation, edges, and corners of the object will be reflected in the three-dimensional point cloud. The vehicle body coordinate system is established with the fixed rotation fulcrum of the spray barrel as the origin. The vehicle body coordinate system adopts the standard right-hand system. In the vehicle body coordinate system, each direction is clearly defined. For example, the X-axis, Y-axis, and Z-axis are perpendicular to each other and conform to the right-hand rule, that is, stretch out the right hand, with the thumb pointing in the positive direction of the X-axis, the index finger pointing in the positive direction of the Y-axis, and the middle finger pointing in the positive direction of the Z-axis. Among them, the X-axis direction is perpendicular to the Y direction and parallel to the horizontal ground, the Y-axis direction is the forward direction of the harvesting machine, and the Z direction is perpendicular to the horizontal ground.

[0105] Specifically, first, the coordinates of each pixel point of the material transfer vehicle in the camera coordinate system are brought into the point cloud library to generate the corresponding three-dimensional point cloud of each pixel point of the material transfer vehicle in the camera coordinate system. Through the external camera parameter matrix, the three-dimensional point cloud of each pixel point of the material transfer vehicle in the camera coordinate system can be converted into the corresponding three-dimensional point cloud in the vehicle body coordinate system. Among them, the external camera parameter matrix is determined by the relative position relationship between the three-dimensional point cloud of each pixel point of the material transfer vehicle in the camera coordinate system and the three-dimensional point cloud of each pixel point of the material transfer vehicle in the camera coordinate system. The parameters in the external camera parameter matrix are obtained based on factors such as the normal vector characteristics of the three-dimensional point cloud and the centroid of the three-dimensional point cloud. For more accurate subsequent analysis, the three-dimensional point cloud can be filtered to remove noise and outliers. An example diagram of the three-dimensional point cloud corresponding to each pixel point of the material transfer vehicle in the camera coordinate system is as Figure 2 shown. An example of the conversion of the three-dimensional point cloud of each pixel point of the material transfer vehicle in the camera coordinate system into the corresponding three-dimensional point cloud in the vehicle body coordinate system is as Figure 3 shown.

[0106] In this application, considering the real-time performance of the algorithm, the three-dimensional point cloud corresponding to each pixel point of the material transfer vehicle in the vehicle body coordinate system can be voxelized. Among them, voxelization is to divide the three-dimensional point cloud into multiple tiny cube units according to the set size specifications. In each tiny cube unit, a representative point is selected. In this way, while effectively retaining the important shape features of the point cloud, the number of points to be processed by the subsequent algorithm can be significantly reduced, thereby significantly improving the running efficiency of the algorithm.

[0107] Step S104, based on the three-dimensional point cloud, determine the region of interest of the hopper of the material transfer vehicle; wherein, the region of interest is the material region inside the hopper.

[0108] Next, combined with Figure 4 to illustrate how to determine the region of interest of the hopper of the material transfer vehicle based on the three-dimensional point cloud.

[0109] See also Figure 4 , Figure 4 This is the second structural schematic diagram of a unloading device of a harvesting machinery provided in an embodiment of the present application.

[0110] like Figure 4 As shown in FIG. 1 , regarding step S104, in a specific implementation, as an example, the following steps may be included:

[0111] Step S1041, selecting a target three-dimensional point cloud within a preset distance range from the harvesting machine from the three-dimensional point cloud.

[0112] Here, the 3D point cloud within the preset distance range from the harvester is the 3D point cloud of the near side of the material transport vehicle, where the near side of the material transport vehicle is the side of the material transport vehicle close to the harvester. Since the binocular camera has a higher ranging accuracy of about 3-6 meters, the point cloud of the near side of the body of the material transport vehicle is used as the main reference.

[0113] Step S1042: determining the outer surface of the material transport vehicle based on the target three-dimensional point cloud.

[0114] Here, the outer surface of the bed of the material handling vehicle is determined from a three-dimensional point cloud of the near side portion of the material handling vehicle.

[0115] Step S1043: Based on the outer surface, determine the coordinates of at least one edge point of the truck bucket in the three-dimensional point cloud.

[0116] Here, the edge points on the outer surface of the material transport vehicle usually present a large curvature change or density change feature. Based on this feature, the edge points on the outer surface of the material transport vehicle can be obtained by methods such as normal differential segmentation and three-dimensional boundary extraction.

[0117] Step S1044, based on the coordinates of the at least one edge point in the three-dimensional point cloud, determine the coordinates of multiple initial reference positioning points of the vehicle bucket in the current frame in the three-dimensional point cloud.

[0118] Here, as an example, in the present application, there are two initial reference positioning points. Specifically, screening is performed from the acquired edge point set, the vertical coordinates of each edge point among all the acquired edge points are compared, and the edge points corresponding to the vertical coordinates within the preset range are selected. Among them, the preset range is set according to actual experience, and is used to determine the edge points of the upper boundary of the vehicle bucket. If there are two edge points after selection, the two selected edge points are determined as the two initial reference positioning points of the vehicle bucket in the current frame; if there is one edge point after selection, a point on the image boundary is selected from the three-dimensional point cloud image, and the point satisfies that the depth value of the point is closest to the depth value of the selected edge point, and the selected edge point and a point on the image boundary selected from the three-dimensional point cloud image are determined as the two initial reference positioning points of the vehicle bucket in the current frame; if the number of selected edge points is zero, the system will send a corresponding text reminder message to the main screen or the sub-screen, and trigger a buzzer alarm. As an example, the example diagram of the determined initial reference positioning points is as follows Figure 5 shown.

[0119] Step S1045, based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determine the straight line where the bucket is close to the upper boundary of the harvesting machinery.

[0120] Here, as an example, the previous moment is 2 seconds ago. According to actual experience, the previous moment can also be set to other values, which are not limited here.

[0121] Next, combine Figure 6 To illustrate how to determine the straight line where the bucket approaches the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud.

[0122] See also Figure 6 , Figure 6 This is the third structural schematic diagram of a unloading device of a harvesting machinery provided in an embodiment of the present application.

[0123] like Figure 6 As shown in FIG. 1 , regarding step S1045, in a specific implementation, as an example, the following steps may be included:

[0124] Step S10451, converting the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system.

[0125] Step S10452, fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line.

[0126] Step S10453, weighting the slope of the initial straight line and the slope of the historical straight line to obtain the slope of the straight line where the upper boundary of the truck bucket close to the harvesting machinery is located; wherein the historical straight line is a straight line fitted by the coordinates of multiple historical reference positioning points of the truck bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system.

[0127] Here, as an example, a Kalman filter is used to perform weighted processing on the slope of the initial straight line and the slope of the historical straight line to avoid frequent jitters in the recognition result.

[0128] Step S10454, weighting the intercept of the initial straight line and the intercept of the historical straight line to obtain the intercept of the straight line where the upper boundary of the bucket close to the harvesting machinery is located.

[0129] Step S10455, based on the slope and intercept of the straight line where the boundary of the vehicle bucket close to the harvesting machinery is located, obtain the straight line where the upper boundary of the vehicle bucket close to the harvesting machinery is located.

[0130] return Figure 4 , step S1046, moving the straight line where the upper boundary of the bucket is close to the harvester by a preset distance along the set direction to obtain a straight line where the boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle.

[0131] Here, as an example, assuming that the body width of the material transport vehicle is 2.5 meters, the preset distance is 2.5 meters.

[0132] Step S1047, determining the region of interest of the vehicle bucket based on the straight line where the upper boundary of the vehicle bucket is close to the harvesting machine and the straight line where the upper boundary of the vehicle bucket is far from the harvesting machine.

[0133] Here, first, multiple reference positioning points on the straight line where the upper boundary of the bucket close to the harvester is located are determined. Then, multiple reference positioning points on the straight line where the upper boundary of the bucket is far from the harvester are determined. Secondly, based on the reference positioning points on the straight line where the upper boundary of the bucket close to the harvester is located and the multiple reference positioning points on the straight line where the upper boundary of the bucket is far from the harvester are connected in a preset order, the enclosed area is determined as the area of ​​interest of the bucket. As an example, the preset order is to start from the leftmost reference positioning point on the upper boundary of the bucket close to the harvester, and connect other reference positioning points in a clockwise direction.

[0134] return Figure 1 , step S105, based on the area of ​​interest, divide into multiple grids, and determine the height of the material in the area corresponding to each grid in the multiple grids in the truck bed.

[0135] Here, the most direct way to determine the height of the material in the truck bed is to separate the grain point cloud from the entire point cloud data, and after filtering out the outliers, perform surface fitting operations on the remaining points. In practical applications, commonly used methods include greedy triangle projection, B-spline surface interpolation, etc. However, it has been verified in practice that this type of technical route takes more than 200 milliseconds in the processing process and cannot meet the real-time requirements. In addition, when using this type of algorithm, if you want to obtain a better fitting effect, you need to make targeted parameter adjustments based on the current specific scenario. Due to the diversity and complexity of automatic unloading scenarios, complex parameter adjustments must be made each time, resulting in the low applicability of this type of algorithm in automatic unloading scenarios, which is difficult to meet the needs of practical applications.

[0136] Next, combine Figure 7 To illustrate how to determine the height of the material in the truck bed based on the region of interest.

[0137] See also Figure 7 , Figure 7 This is the fourth structural schematic diagram of a unloading device of a harvesting machinery provided in an embodiment of the present application.

[0138] like Figure 7 As shown in FIG. 1 , regarding step S105, in a specific implementation, as an example, the following steps may be included:

[0139] Step S1051, based on the region of interest, remove the noise point cloud outside the truck bed from the three-dimensional point cloud to obtain the material point cloud of the truck bed.

[0140] Here, as an example, the noise point cloud outside the truck bed is removed by straight-through filtering. Figure 8 As shown. Figure 8 As shown, the distance scale in the figure is used to determine the distance between the harvesting machine and the material transport vehicle.

[0141] Step S1052, mapping each point in the material point cloud onto a two-dimensional plane to obtain a projection point of the material in the truck bed on the two-dimensional plane.

[0142] Here, obvious abnormal points among the projection points of the material in the bucket on the two-dimensional plane may be removed by radius filtering.

[0143] Step S1053: Divide the area where the projection point is located into a plurality of grids.

[0144] Step S1054, based on the vertical coordinate of each point in the material point cloud, determine the material height corresponding to the projection point after the point is mapped to the two-dimensional plane.

[0145] Step S1055, for each grid in the plurality of grids, weighted processing is performed on the average value of the material height corresponding to the projection points included in the grid and the average value of the material height corresponding to the projection points included in each grid in the surrounding grids of the grid to determine the height of the material in the grid.

[0146] Here, as an example, the weighted processing method may adopt Gaussian blur.

[0147] Step S1056, determining the height of the material in the grid as the height of the material in the area corresponding to the grid in the truck bed.

[0148] Step S1057, determining the height of the material in the area corresponding to each of the multiple grids in the truck bed.

[0149] Here, the material distribution example is as follows Fig. 9 As shown. Fig. 9 As shown in the figure, the value and color depth in the figure represent the grain height in the grid, in meters. Here, in the material distribution map, there may be some black holes representing invalid values. These holes may be caused by missing material point cloud data, mistaken deletion during filtering, etc. In order to make the material distribution map more complete and accurate, these black holes need to be further processed. The processing methods can include interpolation, filling, etc., to fill the values ​​of these invalid areas based on the information of the surrounding valid areas.

[0150] return Figure 1 , step S106, adjusting the spray barrel of the harvester based on the height of the material and the multiple observation points, and adjusting the rear end baffle of the spray barrel of the harvester based on the distance between the harvester and the material transport vehicle.

[0151] Here, the observation point is determined according to the landing point of the material ejected by the spray barrel of the harvesting machine. The observation points at different positions are selected to represent the state of the material in the entire bucket.

[0152] Here, the rules for the extension and retraction of the baffle are formulated according to the distance between the harvester and the material transport vehicle. Among them, the baffle at the tail end of the harvester's spray barrel is at the tail end of the harvester's spray barrel. As an example, if the distance is far, in order to ensure that the material can fall smoothly into the material transport vehicle, the baffle needs to be extended. If the distance is close, the baffle at the tail end of the spray barrel can be appropriately shortened. The harvester's spray barrel is adjusted by controlling the push rod.

[0153] Next, combine Fig.10 To illustrate how to adjust the spray barrel of a harvesting machine based on the height of the material and multiple observation points.

[0154] See also Fig.10 This is the fifth structural schematic diagram of a unloading device of a harvesting machinery provided in an embodiment of the present application.

[0155] like Fig.10 As shown in FIG. 1 , regarding adjusting the spray barrel of the harvesting machine based on the height of the material and multiple observation points in step S106, in specific implementation, as an example, the following steps may be included:

[0156] Step S1061, confirming the landing point of the material ejected by the spray barrel of the harvesting machine as the initial observation point.

[0157] Here, the present application obtains the three-dimensional coordinates of the tail end of the nozzle baffle according to the angle values ​​obtained after calibration of the three angle sensors set, combined with the physical modeling relationship of the nozzle. Afterwards, based on the parabolic motion equation and taking into account factors such as the rotation speed of the harvesting machinery's throwing fan, the landing point of the material ejected by the nozzle of the harvesting machinery is predicted. Among them, three angle sensors of the same specifications are installed outside the cab, one angle sensor is installed near the bottom of the nozzle swing arm to measure the approximate unloading direction; one angle sensor is installed near the tail end of the nozzle to measure the nozzle lifting height; and the other angle sensor is installed at the nozzle tail baffle to measure the distance of the ejected material. Each angle sensor is installed and fixed by a corresponding bracket and related mechanical linkage structure. Compared with the existing foreign ones, each angle sensor of the present application is adapted to the domestic harvesting machinery nozzle through a corresponding bracket and related mechanical linkage structure to avoid structural interference. At the same time, it is considered to extend the cover plate at an appropriate position to prevent rainwater from backflowing into the angle sensor and affecting the service life. Specifically, for the angle sensor installed at the baffle at the tail end of the nozzle, such as Fig.11 As shown in the figure, a mathematical modeling diagram of the tail baffle of the spray barrel of the harvesting machinery is established. Fig.11 As shown, point B and point E are the positions of the two rotating axes respectively; CD is a connecting rod. Point A and point F are the inherent positions of the swing arm and the baffle respectively, and these two points form a fixed angle with the point selected on the surface of the nozzle. Based on the above points, the nozzle can be divided into three parts: the part where points A, B, and C are located constitutes the long swing arm of the nozzle, which is defined as section one; the area where points B, C, D, and E are located is the connecting part between the swing arm and the baffle, which is defined as section two; the part where points D, E, and F are located is the tail end baffle, which is defined as section three. After completing the calibration operation according to the characteristic curve of the angle sensor, the structure can measure ∠ABE. For the measurement of related physical quantities at other positions, a similar connecting rod structure is used, and the angle conversion is performed. After a series of measurements and conversions, ∠ADF can be finally obtained. ∠ADF is the angle formed by the tail end baffle and the vertical direction of the ground, and its size represents the exit direction of the projectile material.

[0158] Here, the spray barrel of the harvesting machinery has three throwing modes when throwing materials. The first throwing mode is the front mode. In this mode, the material is thrown in the direction of the front of the vehicle with the front direction as the front reference. In the actual operation process, the landing point is set according to the lateral distribution of the material. Specifically, as long as the observation point in front of the vehicle head has the lowest material height among all the observation points, this point is set as the target landing point; if the observation point has the highest material height among all the observation points, the observation points behind this point are queried one by one, and they are screened in descending order of priority. This is because during the operation of the harvesting machinery, there is relative movement between the harvesting machinery and the material transport vehicle. The use of this front mode can ensure that the material transport vehicle can always follow stably during the overtaking process. At the same time, the range of the thrown materials can fully cover the entire material transport vehicle bucket, fully meeting the needs of actual operations. The second throwing mode is the rear mode. The working principle of the rear mode is similar to that of the front mode. The second casting mode is the standard mode, which is a combination of the front mode and the rear mode. That is, after completing the automatic unloading task in the left half of the camera's field of view, the system will automatically switch from the front mode to the rear mode to achieve a more efficient and comprehensive unloading operation.

[0159] Step S1062, based on the coordinates of the initial observation point in the three-dimensional point cloud and a preset step size, determining the coordinates of multiple observation points in the three-dimensional point cloud.

[0160] Here, as an example, the preset step size can be set to 0.5 meters, which is not limited here. Specifically, after determining the coordinates of the initial observation point in the three-dimensional point cloud, move in sequence along the positive direction of the Y axis and / or the negative direction of the Y axis according to the preset step size to obtain the coordinates of multiple observation points in the three-dimensional point cloud.

[0161] Here, the coordinates of multiple observation points in the three-dimensional point cloud can be converted into coordinates in a two-dimensional coordinate system and drawn in a two-dimensional image, wherein multiple observation points can be represented by different colors in the two-dimensional image with respect to the height of the material. As an example, red represents an observation point with a higher material height; yellow represents an observation point with a moderate material height; and green represents an observation point with a lower material height.

[0162] Step S1063, based on the height of the material in the truck bed, determine the coordinates of the observation point corresponding to the lowest material height among multiple preset observation points in the three-dimensional point cloud.

[0163] Step S1064, determining the difference between the horizontal coordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the horizontal coordinate of the landing point of the material ejected by the spray barrel of the harvester in the three-dimensional point cloud.

[0164] Step S1065, determining the difference between the vertical coordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the vertical coordinate of the landing point of the material ejected by the spray barrel of the harvester in the three-dimensional point cloud.

[0165] Step S1066, adjusting the spray barrel of the harvester based on the difference between the horizontal coordinate and the vertical coordinate.

[0166] Here, the control quantity is outputted by the PID controller, wherein the input deviation of the PID controller is: the coordinates of the observation point at the lowest material height in the three-dimensional point cloud and the coordinates of the landing point of the material ejected by the harvester spray barrel in the three-dimensional point cloud, and the difference between the horizontal coordinates and the vertical coordinates. The PWM output corresponding to the control quantity needs to be speed-limited to obtain the final current output result, and control the electronically controlled hydraulic valve to adjust the harvester spray barrel.

[0167] The following is an example of the identification result of the material transport vehicle displayed on the secondary screen: Fig.12 Here, the original material transport vehicle recognition result example is two-dimensional. The PCA algorithm is used to compress the dimension and obtain the approximate lateral distribution of the material, that is, Fig.12 The one-dimensional image shown. Fig.12 As shown, the dotted line is the straight line where the boundary of the material transport vehicle is located, and the point is the observation point.

[0168] This application does not restrict the model, type and color of the added materials in terms of the parameters related to the material transport vehicle. On this basis, this application has a series of significant advantages: first, the added structure is highly reliable, the modification process is simple and easy, and the added hardware cost is low, and it is easy to carry out optional installation and subsequent upgrade operations; second, there is no need to carry out a large amount of sampling and data training work on the model data of the material transport vehicle, so as to achieve stable recognition of the boundary line of the material transport vehicle, and it is less affected by environmental interference; third, after algorithm optimization, the real-time rate can reach more than 8fps, which can effectively achieve the goal of real-time recognition and control; fourth, at the operational level, the need for frequent operation of the operating buttons on the spray gun handle is reduced, and the operator only needs to pay attention to the alarm prompts on the screen and implement manual intervention for abnormal situations.

[0169] An embodiment of the present application provides a method for unloading a harvesting machine, through which driving safety and driving comfort are improved.

[0170] Based on the same application concept, the embodiment of the present application also provides a unloading device of a harvesting machinery corresponding to the unloading method of the harvesting machinery provided in the above embodiment. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the unloading method of the harvesting machinery in the above embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0171] See also Fig.13 , Fig.13 A schematic structural diagram of a unloading device of a harvesting machinery provided in an embodiment of the present application.

[0172] like Fig.13 As shown in the embodiment of the present application, a discharging device 210 of a harvesting machine is provided, and the discharging device includes:

[0173] An acquisition module 211 acquires a depth map of the material transport vehicle captured by the binocular camera;

[0174] A coordinate conversion module 212 determines the coordinates of each pixel constituting the material transfer vehicle in the depth map in the camera coordinate system based on the depth map;

[0175] The point cloud generation module 213 generates a three-dimensional point cloud corresponding to each pixel point in the vehicle body coordinate system based on the coordinates of each pixel point constituting the material handling vehicle in the depth map in the camera coordinate system;

[0176] An area of ​​interest extraction module 214 determines an area of ​​interest of a bucket of a material transport vehicle based on the three-dimensional point cloud; wherein the area of ​​interest is a material area in the bucket;

[0177] The material height calculation module 215 divides the region of interest into a plurality of grids and determines the height of the material in the region corresponding to each of the plurality of grids in the vehicle bucket;

[0178] The spray barrel and baffle control module 216 adjusts the spray barrel of the harvester based on the height of the material and multiple observation points, and adjusts the baffle at the rear end of the spray barrel of the harvester based on the distance between the harvester and the material transport vehicle.

[0179] Furthermore, the region of interest extraction module 214 is specifically used for:

[0180] Selecting a target three-dimensional point cloud within a preset distance range from the harvesting machine from the three-dimensional point cloud;

[0181] Based on the target three-dimensional point cloud, determining the outer surface of the material transport vehicle;

[0182] Based on the outer surface, determining the coordinates of at least one edge point of the bucket in the three-dimensional point cloud;

[0183] Based on the coordinates of the at least one edge point in the three-dimensional point cloud, determining the coordinates of a plurality of initial reference positioning points of the bucket in the three-dimensional point cloud;

[0184] Based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determining a straight line where the bucket is close to the upper boundary of the harvesting machinery;

[0185] The straight line where the upper boundary of the bucket is close to the harvester is moved by a preset distance along a set direction to obtain a straight line where the upper boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle;

[0186] The region of interest of the bucket is determined based on a straight line where the upper boundary of the bucket is close to the harvesting machine and a straight line where the upper boundary of the bucket is far from the harvesting machine.

[0187] Further, when the region of interest extraction module 214 is used to determine the straight line where the bucket approaches the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, it is also specifically used to:

[0188] Convert the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system;

[0189] Fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line;

[0190] The slope of the initial straight line and the slope of the historical straight line are weighted to obtain the slope of the straight line where the bucket is close to the upper boundary of the harvesting machinery; wherein the historical straight line is a straight line obtained by fitting the coordinates of multiple historical reference positioning points of the bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system;

[0191] The intercept of the initial straight line and the intercept of the historical straight line are weighted to obtain the intercept of the straight line where the upper boundary of the bucket close to the harvesting machinery is located;

[0192] Based on the slope and intercept of the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is located, the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is obtained.

[0193] Furthermore, the material height calculation module 215 is specifically used for:

[0194] Based on the region of interest, removing the noise point cloud outside the truck bed from the three-dimensional point cloud to obtain the material point cloud of the truck bed;

[0195] Mapping each point in the material point cloud onto a two-dimensional plane to obtain a projection point of the material in the truck bucket on the two-dimensional plane;

[0196] Dividing the area where the projection point is located into a plurality of grids;

[0197] Based on the vertical coordinate of each point in the material point cloud, determine the material height corresponding to the projection point after the point is mapped to the two-dimensional plane;

[0198] For each of the multiple grids, weighted processing is performed on an average value of material heights corresponding to the projection points included in the grid and an average value of material heights corresponding to the projection points included in each of the grids surrounding the grid to determine the height of the material in the grid;

[0199] Determine the height of the material in the grid as the height of the material in the area corresponding to the grid in the truck bucket;

[0200] The height of the material in the area corresponding to each of the plurality of grids in the vehicle bucket is determined.

[0201] Furthermore, the nozzle and baffle control module 216 is specifically used for:

[0202] Confirm the landing point of the material ejected by the spray barrel of the harvesting machinery as the initial observation point;

[0203] Determining coordinates of a plurality of observation points in the three-dimensional point cloud based on the coordinates of the initial observation point in the three-dimensional point cloud and a preset step size;

[0204] Based on the height of the material, determining the coordinates of the observation point corresponding to the lowest material height among a plurality of preset observation points in the three-dimensional point cloud;

[0205] Determine the difference between the horizontal coordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the horizontal coordinate of the landing point of the material projected by the spray barrel of the harvester in the three-dimensional point cloud;

[0206] Determine the difference between the ordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the ordinate of the landing point of the material ejected by the spray barrel of the harvester in the three-dimensional point cloud;

[0207] Based on the difference in the abscissa and the difference in the ordinate, a spray barrel of the harvester is adjusted.

[0208] An embodiment of the present application provides a unloading device for a harvesting machine, through which driving safety and driving comfort are improved.

[0209] See also Fig.14 , Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0210] like Fig.14 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0211] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 via the bus 330. When the machine-readable instructions are executed by the processor 310, the above-mentioned Figure 1 , Figure 4 , Figure 6 , Figure 7 and Fig.12 The specific implementation of the steps of the unloading method of the harvesting machinery in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0212] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 , Figure 4 , Figure 6 , Figure 7 and Fig.12 The specific implementation of the steps of the unloading method of the harvesting machinery in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0213] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0217] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for unloading a harvesting machine, characterized in that: A binocular camera is provided at the bottom of the harvester spray barrel, and the binocular camera is used to collect a depth map of the scene within the camera field of view. The unloading method includes: Obtaining a depth map of the material transport vehicle captured by the binocular camera; Based on the depth map, determine the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system; Based on the coordinates of each pixel constituting the material transport vehicle in the depth map in the camera coordinate system, a three-dimensional point cloud corresponding to each pixel in the vehicle body coordinate system is generated; Based on the three-dimensional point cloud, determine the region of interest of the bucket of the material transport vehicle; wherein the region of interest is the material area in the bucket; Divide the region of interest into multiple grids based on the region of interest, and determine the height of the material in the region corresponding to each of the multiple grids in the vehicle box; Based on the height of the material and multiple observation points, the spray barrel of the harvester is adjusted, and based on the distance between the harvester and the material transport vehicle, the rear end baffle of the spray barrel of the harvester is adjusted.

2. The unloading method according to claim 1, characterized in that: The determining of the region of interest of the bucket of the material handling vehicle based on the three-dimensional point cloud comprises: Selecting a target three-dimensional point cloud within a preset distance range from the harvesting machine from the three-dimensional point cloud; Based on the target three-dimensional point cloud, determining the outer surface of the material transport vehicle; Based on the outer surface, determining the coordinates of at least one edge point of the bucket in the three-dimensional point cloud; Based on the coordinates of the at least one edge point in the three-dimensional point cloud, determining the coordinates of a plurality of initial reference positioning points of the bucket in the three-dimensional point cloud; Based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determining a straight line where the bucket is close to the upper boundary of the harvesting machinery; The straight line where the upper boundary of the bucket is close to the harvester is moved by a preset distance along a set direction to obtain a straight line where the upper boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle; The region of interest of the bucket is determined based on a straight line where the upper boundary of the bucket is close to the harvesting machine and a straight line where the upper boundary of the bucket is far from the harvesting machine.

3. The unloading method according to claim 2, characterized in that: The step of determining a straight line where the bucket is close to the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud comprises: Convert the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system; Fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line; The slope of the initial straight line and the slope of the historical straight line are weighted to obtain the slope of the straight line where the bucket is close to the upper boundary of the harvesting machinery; wherein the historical straight line is a straight line obtained by fitting the coordinates of multiple historical reference positioning points of the bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system; The intercept of the initial straight line and the intercept of the historical straight line are weighted to obtain the intercept of the straight line where the bucket is close to the upper boundary of the harvesting machinery; Based on the slope and intercept of the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is located, the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is obtained.

4. The unloading method according to claim 1, characterized in that: The method of dividing the region of interest into a plurality of grids and determining the height of the material in the region corresponding to each of the plurality of grids in the vehicle bucket comprises: Based on the region of interest, removing the noise point cloud outside the truck bed from the three-dimensional point cloud to obtain the material point cloud of the truck bed; Mapping each point in the material point cloud onto a two-dimensional plane to obtain a projection point of the material in the truck bucket on the two-dimensional plane; Dividing the area where the projection point is located into a plurality of grids; Based on the vertical coordinate of each point in the material point cloud, determine the material height corresponding to the projection point after the point is mapped to the two-dimensional plane; For each of the multiple grids, weighted processing is performed on an average value of material heights corresponding to the projection points included in the grid and an average value of material heights corresponding to the projection points included in each of the grids surrounding the grid to determine the height of the material in the grid; Determine the height of the material in the grid as the height of the material in the area corresponding to the grid in the truck bucket; The height of the material in the area corresponding to each of the plurality of grids in the vehicle bucket is determined.

5. The unloading method according to claim 1, characterized in that: The method of adjusting the spray barrel of the harvesting machinery based on the height of the material and the plurality of observation points comprises: Confirm the landing point of the material ejected by the spray barrel of the harvesting machinery as the initial observation point; Determining coordinates of a plurality of observation points in the three-dimensional point cloud based on the coordinates of the initial observation point in the three-dimensional point cloud and a preset step size; Based on the height of the material, determining the coordinates of the observation point corresponding to the lowest material height among a plurality of preset observation points in the three-dimensional point cloud; Determine the difference between the horizontal coordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the horizontal coordinate of the landing point of the material projected by the spray barrel of the harvester in the three-dimensional point cloud; Determine the difference between the ordinate of the observation point corresponding to the lowest material height in the three-dimensional point cloud and the ordinate of the landing point of the material ejected by the spray barrel of the harvester in the three-dimensional point cloud; Based on the difference in the abscissa and the difference in the ordinate, a spray barrel of the harvester is adjusted.

6. A unloading device for a harvesting machine, characterized in that: The unloading device comprises: An acquisition module is used to acquire a depth map of the material transport vehicle captured by the binocular camera; A coordinate conversion module, based on the depth map, determines the coordinates of each pixel constituting the material transfer vehicle in the depth map in the camera coordinate system; A point cloud generation module, based on the coordinates of each pixel constituting the material handling vehicle in the depth map in the camera coordinate system, generates a three-dimensional point cloud corresponding to each pixel in the vehicle body coordinate system; An area of ​​interest extraction module is used to determine an area of ​​interest of a bucket of a material transport vehicle based on the three-dimensional point cloud; wherein the area of ​​interest is a material area in the bucket; A material height calculation module, based on the region of interest, divides the region into a plurality of grids, and determines the height of the material in the region corresponding to each of the plurality of grids in the vehicle bucket; The spray barrel and baffle control module adjusts the spray barrel of the harvester based on the height of the material and multiple observation points, and adjusts the baffle at the rear end of the spray barrel of the harvester based on the distance between the harvester and the material transport vehicle.

7. The unloading device according to claim 6, characterized in that: The region of interest extraction module is specifically used for: Selecting a target three-dimensional point cloud within a preset distance range from the harvesting machine from the three-dimensional point cloud; Based on the target three-dimensional point cloud, determining the outer surface of the material transport vehicle; Based on the outer surface, determining the coordinates of at least one edge point of the bucket in the three-dimensional point cloud; Based on the coordinates of the at least one edge point in the three-dimensional point cloud, determining the coordinates of a plurality of initial reference positioning points of the bucket in the current frame in the three-dimensional point cloud; Based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, determining a straight line where the bucket is close to the upper boundary of the harvesting machinery; The straight line where the upper boundary of the bucket is close to the harvester is moved by a preset distance along a set direction to obtain a straight line where the boundary of the bucket is away from the harvester; wherein the set direction is a direction away from the harvester and parallel to the horizontal ground; the preset distance is the body width of the material transport vehicle; The region of interest of the bucket is determined based on a straight line where the upper boundary of the bucket is close to the harvesting machine and a straight line where the upper boundary of the bucket is far from the harvesting machine.

8. The unloading device according to claim 7, characterized in that: When the region of interest extraction module is used to determine the straight line where the bucket approaches the upper boundary of the harvesting machinery based on the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud, it is also specifically used to: Convert the coordinates of the multiple initial reference positioning points in the three-dimensional point cloud into coordinates in a two-dimensional coordinate system; Fitting the coordinates of the multiple initial reference positioning points in the two-dimensional coordinate system into an initial straight line; The slope of the initial straight line and the slope of the historical straight line are weighted to obtain the slope of the straight line where the bucket is close to the upper boundary of the harvesting machinery; wherein the historical straight line is a straight line obtained by fitting the coordinates of multiple historical reference positioning points of the bucket in each frame of multiple frames at the previous moment in a two-dimensional coordinate system; The intercept of the initial straight line and the intercept of the historical straight line are weighted to obtain the intercept of the straight line where the bucket is close to the upper boundary of the harvesting machinery; Based on the slope and intercept of the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is located, the straight line where the upper boundary of the vehicle bucket closes to the harvesting machinery is obtained.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the unloading method of the harvesting machinery as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the unloading method of the harvesting machinery as described in any one of claims 1 to 5 are executed.

Images