A method and device for automatically tracking a bucket with a throwing tube based on edge intelligence

Abstract

The technical solution of the present invention discloses a method and device for automatically tracking a bucket by a throwing barrel based on edge intelligence, comprising: collecting video data through a camera, measuring the angle information of the camera and the throwing barrel through an angle sensor, and the edge intelligent device obtaining the camera video stream and decoding it to obtain a video frame sequence, and obtaining the angle information; deploying a bucket detection model, a monocular ranging model and a bucket tracking controller in the edge intelligent device, the bucket detection model is used to process the video frame sequence, and detect the pixel coordinates of the center point of the bucket in the video image; the monocular ranging model is used to process the pixel coordinates and angle information of the center point of the bucket in the video image, and obtain the distance from the center of the bucket to the camera in the vertical direction; the bucket tracking controller generates a tracking control instruction according to the pixel coordinates of the center point of the bucket in the video image and the monocular ranging result, controls the rotation of the throwing barrel, and realizes automatic tracking of the bucket by the throwing barrel during agricultural machinery harvesting operations.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for automatically tracking a bucket using a throwing tube based on edge intelligence. Background Art

[0002] In recent years, with the development of hardware devices such as graphics processing units (GPUs), edge intelligent computing has begun to rise. The current popular GPU edge intelligent computing devices can already support the calculation of large deep neural networks and can ensure computing efficiency. With the improvement of the performance of edge intelligent devices, the demand for edge computing in practical applications is becoming more and more diversified, such as video surveillance, autonomous driving, agricultural machinery automatic control, etc.

[0003] Agricultural machinery automatic control is a new intelligent control field in recent years. With the improvement of productivity, agricultural production is becoming more and more mechanized and automated. In the process of agricultural production, the labor cost is gradually reduced, but the application of mechanization also generates some manpower demand. For example, in agricultural machinery harvesting operations, it is often necessary to increase human resources to control the throwing barrel to track the bucket for operation. However, there is no mature and stable automatic tracking method for the throwing barrel in the current market.

[0004] Patent CN 110244710 B discloses an automatic tracking method, including: obtaining an RGB image and a depth image of a target field of view; using the RGB image as an input of a first preset angle network model to obtain a first predicted angle output by the first preset angle network model; using the RGB image and the depth image as input of a preset obstacle avoidance network model to obtain an obstacle avoidance type and a correctness rate corresponding to the obstacle avoidance type output by the preset obstacle avoidance network model; determining a second predicted angle according to the obstacle avoidance type and the correctness rate corresponding to the obstacle avoidance type; determining a target travel angle according to the first predicted angle and the second predicted angle to achieve tracking, but the tracking of the device is to determine the average value of the first predicted angle and the second predicted angle as the target travel angle, and the device only moves in one direction, which cannot meet the longitudinal and lateral movement requirements of the throwing barrel when tracking the bucket. Summary of the invention

[0005] In order to solve the existing technical problems, the present invention provides a method and device for automatically tracking a bucket with a throwing drum based on edge intelligence. By deploying edge intelligent equipment in agricultural machinery, and deploying a bucket detection model, a monocular ranging model and a bucket tracking controller in the edge equipment, the throwing drum can automatically track the bucket during the harvesting operation of the agricultural machinery at the front end.

[0006] The technical solution of the present invention is: a method for automatically tracking a bucket with a throwing tube based on edge intelligence, comprising the following steps: collecting video data through a camera, measuring angle information of the camera and the throwing tube through an angle sensor, and obtaining a video stream of the camera through an edge intelligent device and decoding it to obtain a video frame sequence, and obtaining angle information;

[0007] A vehicle bucket detection model, a monocular ranging model and a vehicle bucket tracking controller are deployed in the edge intelligent device. The vehicle bucket detection model is used to process the decoded video frame sequence and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model is used to process the pixel coordinates and angle information of the center point of the vehicle bucket in the video image to obtain the vertical distance from the center of the vehicle bucket to the camera; the vehicle bucket tracking controller generates a tracking control instruction according to the pixel coordinates of the center point of the vehicle bucket in the video image and the monocular ranging result, and the tracking control instruction is sent to the throwing cylinder to control the throwing cylinder to rotate.

[0008] Furthermore, the modeling method of the monocular ranging model is to establish a monocular ranging model M according to the pixel coordinates of the center point of the bucket in the video image at different camera heights and different camera angles and the actual distance between the center point of the bucket and the camera in the vertical direction. The specific formula is as follows:

[0009] w2=M(h2, h3, α, (x1, y1))

[0010] Among them, w2 is the actual distance between the center point of the truck bed and the camera in the vertical direction, h2 is the height of the truck bed, h3 is the height of the camera, α is the angle between the optical axis of the camera and the vertical direction of the camera, and (x1, y1) is the pixel coordinate of the center point of the truck bed in the video image.

[0011] Furthermore, the method for generating the tracking control instruction comprises the following steps:

[0012] (1) The longitudinal rotation command is generated by calculating the angle γ between the horizontal direction of the center point of the bucket and the throwing direction of the throwing tube according to the following formula:

[0013]

[0014] Among them, h1 is the height of the throwing tube, and w1 is the actual distance between the camera and the throwing tube in the vertical direction;

[0015] Then calculate the longitudinal offset angle P between the throwing tube and the bucket according to the following formula:

[0016] P=90-γ-β

[0017] Among them, β is the angle between the throwing direction of the throwing tube and the vertical direction;

[0018] Generate a longitudinal rotation instruction according to the longitudinal offset angle P between the throwing tube and the bucket;

[0019] (2) Generation of lateral rotation instructions: generating lateral rotation instructions based on the difference between the pixel coordinates (x1, y1) of the center point of the truck bed in the video image and the coordinates (x2, y2) of the center point of the video image.

[0020] Furthermore, the edge intelligent device supports GPU computing.

[0021] Furthermore, the model used to train the vehicle bucket detection model is a deep neural network Faster RCNN, and the vehicle bucket detection model and operating environment after Faster RCNN model training are deployed in the edge intelligent device.

[0022] Furthermore, the edge intelligent device performs frame skipping processing on the video frame sequence according to the running speed of the truck bucket detection model after training the Faster RCNN model, detects the pixel position of the truck bucket in the video image, and obtains the pixel coordinates of the center point of the truck bucket in the video image.

[0023] An edge-intelligence-based throwing barrel automatic tracking bucket device includes a throwing arm, a camera, a throwing barrel and a bucket, wherein the lower end of the throwing arm is mounted on the agricultural machinery, the throwing barrel is mounted on the upper end of the throwing arm, the camera is mounted in the middle of the throwing arm, and angle sensors are mounted on the camera and the throwing barrel. The angle sensor, the camera and the throwing barrel are respectively connected to the edge intelligent device for communication, and the edge intelligent device is deployed in the agricultural machinery control system;

[0024] The angle sensor is used to obtain the angle between the optical axis of the camera and the vertical direction of the camera, as well as the angle between the throwing direction of the throwing tube and the vertical direction, and send the angle information to the edge intelligent device in real time;

[0025] The edge intelligent device is used to process the camera video stream and angle information, generate tracking control instructions, and send tracking control instructions to the delivery tube.

[0026] Furthermore, the edge intelligent device includes a vehicle bucket detection model, a monocular ranging model and a vehicle bucket tracking controller; the vehicle bucket detection model is used to process the camera video stream and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model detects the actual distance between the center point of the vehicle bucket and the camera in the vertical direction through the pixel coordinates of the center point of the vehicle bucket in the video image and the angle sensor data; the vehicle bucket tracking controller generates tracking control instructions in real time through the pixel coordinates of the center point of the vehicle bucket in the video image and the actual distance between the center point of the vehicle bucket and the camera in the vertical direction.

[0027] By adopting the above technical solution, the beneficial effects achieved by the present invention are as follows:

[0028] (1) The present invention deploys edge intelligent devices in agricultural machinery, and deploys deep neural network models, monocular ranging models and bucket tracking controllers in the edge devices to realize automatic tracking of the agricultural machinery throwing barrel at the front end; the edge intelligent device automatically locates the bucket and controls the throwing barrel to automatically track the bucket, filling the gap in the market for automatic tracking methods of the throwing barrel during agricultural machinery harvesting operations, saving labor costs in the throwing barrel tracking operation, and promoting the further development of intelligent control of agricultural machinery.

[0029] (2) The present invention invents a monocular distance measurement model for the scenario where the throwing tube automatically tracks the bucket. This method can quickly and accurately measure the distance between the bucket and the camera.

[0030] (3) The vehicle bed tracking controller of the present invention can calculate and issue control instructions in real time according to the pixel coordinates of the center point of the vehicle bed in the video image and the actual distance between the center point of the vehicle bed and the camera in the vertical direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural block diagram of the present invention;

[0032] Figure 2 It is a schematic diagram of the principle structure of the monocular ranging model of the present invention.

[0033] In the figure, there are a throwing arm 1, a camera 2, a throwing tube 3, and a bucket 4. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with specific embodiments.

[0035] Example 1

[0036] Reference Figure 1-2 As shown, a method for automatically tracking a bucket with a throwing tube based on edge intelligence includes the following steps: collecting video data through a camera, measuring the angle information of the camera and the throwing tube through an angle sensor, and the edge intelligent device obtaining the camera video stream and decoding it to obtain a video frame sequence, and obtaining angle information;

[0037] A vehicle bucket detection model, a monocular ranging model and a vehicle bucket tracking controller are deployed in the edge intelligent device. The vehicle bucket detection model is used to process the decoded video frame sequence and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model is used to process the pixel coordinates and angle information of the center point of the vehicle bucket in the video image to obtain the vertical distance from the center of the vehicle bucket to the camera; the vehicle bucket tracking controller generates a tracking control instruction according to the pixel coordinates of the center point of the vehicle bucket in the video image and the monocular ranging result, and the tracking control instruction is sent to the throwing cylinder to control the throwing cylinder to rotate.

[0038] Furthermore, the modeling method of the monocular ranging model is to establish a monocular ranging model M according to the pixel coordinates of the center point of the bucket in the video image at different camera heights and different camera angles and the actual distance between the center point of the bucket and the camera in the vertical direction. The specific formula is as follows:

[0039] w2=M(h2, h3, α, (x1, y1))

[0040] Among them, w2 is the actual distance between the center point of the truck bed and the camera in the vertical direction, h2 is the height of the truck bed, h3 is the height of the camera, α is the angle between the optical axis of the camera and the vertical direction of the camera, and (x1, y1) is the pixel coordinate of the center point of the truck bed in the video image.

[0041] The monocular ranging model is built based on the characteristics of the camera's tilted shooting, and the actual distance between the center point of the truck bed and the camera in the vertical direction is calculated more accurately.

[0042] Furthermore, the method for generating the tracking control instruction comprises the following steps:

[0043] (1) The longitudinal rotation command is generated by calculating the angle γ between the horizontal direction of the center point of the bucket and the throwing direction of the throwing tube according to the following formula:

[0044]

[0045] Among them, h1 is the height of the throwing tube, and w1 is the actual distance between the camera and the throwing tube in the vertical direction;

[0046] Then calculate the longitudinal offset angle P between the throwing tube and the bucket according to the following formula:

[0047] P=90-γ-β

[0048] Among them, β is the angle between the throwing direction of the throwing tube and the vertical direction;

[0049] Generate a longitudinal rotation instruction according to the longitudinal offset angle P between the throwing tube and the bucket;

[0050] (2) Generation of lateral rotation instructions: generating lateral rotation instructions based on the difference between the pixel coordinates (x1, y1) of the center point of the truck bed in the video image and the coordinates (x2, y2) of the center point of the video image.

[0051] In the present invention, h1, h2, h3 and w1 are obtained by actual measurement, and the data is input into the edge intelligent device. In the process of generating the lateral rotation instruction, the lateral rotation instruction is generated according to the difference between the pixel coordinates (x1, y1) of the center point of the bucket in the video image and the horizontal coordinates of the center point coordinates (x2, y2) of the video image.

[0052] Through the longitudinal rotation instruction, the throwing cylinder is rotated longitudinally to prevent the throwing cylinder from throwing materials to the left and right sides of the truck bucket; through the lateral rotation instruction, the throwing cylinder is rotated laterally until the difference between x1 and x2 is 0, to prevent the throwing cylinder from throwing materials to the front or rear sides of the truck bucket such as the front or rear of the truck bucket; through the longitudinal and lateral rotation of the throwing cylinder, the throwing cylinder is guaranteed to accurately and automatically track the truck bucket. Figure 2 The longitudinal direction is the up-down direction, and the transverse direction is the straight line direction where the front of the vehicle, the cargo box and the rear of the vehicle are located, wherein the front of the vehicle is set in the direction perpendicular to the paper surface and inward.

[0053] Furthermore, the edge intelligent device supports GPU computing, and Jetson Xavier NX is used in this embodiment 1.

[0054] Furthermore, the model used to train the vehicle bucket detection model is a deep neural network Faster RCNN, and the vehicle bucket detection model and operating environment after Faster RCNN model training are deployed in the edge intelligent device.

[0055] Training the truck bed detection model with the high-precision deep neural network Faster RCNN helps accurately detect the center point of the truck bed, and Jetson Xavier NX supports the installation of Faster RCNN.

[0056] Furthermore, the edge intelligent device performs frame skipping processing on the video frame sequence according to the running speed of the truck bucket detection model after training the Faster RCNN model, detects the pixel position of the truck bucket in the video image, and obtains the pixel coordinates of the center point of the truck bucket in the video image.

[0057] By skipping frames, the truck bed detection model can always process the latest camera video image to avoid delays.

[0058] The truck bed detection model trained by Faster RCNN in this embodiment 1 runs at a speed of 170ms / frame in the Jetson Xavier NX device, and the instruction scheduling time is about 30ms / frame. Therefore, the frame skipping process is specifically to obtain the current video frame for analysis every 200ms.

[0059] In the actual application of this embodiment 1, a monocular distance measurement model M is first established by actually measuring different h2 and corresponding h3, α, (x1, y1) and w2, wherein α is measured by an angle sensor and (x1, y1) is obtained by a bucket detection model. When the throwing cylinder automatically tracks the bucket, w2 is calculated by the formula w2=M(h2, h3, α, (x1, y1)) of the monocular distance measurement model according to the currently measured h2, h3, α and (x1, y1) detected by the bucket detection model, as follows:

[0060] Through actual measurement, we get h2=1.3m, h3=3.4m, through the angle sensor measurement, we get α=34.5°, through the truck bed detection model, we get (x1, y1)=(441, 437), and according to the formula of the monocular ranging model M, we get w2=M(1.3, 3.4, 34.5, (441, 437))=4.37m.

[0061] Then, combined with the actual measured h1=4.4m, w1=2.5m, according to the formula calculate We get γ=53.31°; then we get β=37.86° by combining with the angle sensor. According to the formula P=90-γ-β, we get P=90-53.31-37.86=-1.17°. P<0, so the direction of the longitudinal rotation command is downward rotation, that is, the longitudinal rotation command is to rotate downward 1.17°. Figure 2 The corresponding throwing tube in the middle rotates 1.17° counterclockwise around the lower end of the throwing arm.

[0062] According to the current h2, h3, α and the center point coordinates of the video image corresponding to (x2, y2) = (640, 360), the difference x1-x2 = 441-640 = -199 is calculated. If the difference < 0, the direction of the horizontal rotation instruction is backward rotation, that is, the horizontal rotation instruction is backward rotation and the backward rotation distance corresponds to the difference of x1-x2. Figure 2 The corresponding throwing cylinder rotates around the lower end of the throwing arm on a plane perpendicular to the paper surface, so that the throwing cylinder moves in an outward direction perpendicular to the paper surface.

[0063] An edge-intelligence-based throwing barrel automatic tracking bucket device includes a throwing arm, a camera, a throwing barrel and a bucket, wherein the lower end of the throwing arm is mounted on the agricultural machinery, the throwing barrel is mounted on the upper end of the throwing arm, the camera is mounted in the middle of the throwing arm, and angle sensors are mounted on the camera and the throwing barrel. The angle sensor, the camera and the throwing barrel are respectively connected to the edge intelligent device for communication, and the edge intelligent device is deployed in the agricultural machinery control system;

[0064] The angle sensor is used to obtain the angle between the optical axis of the camera and the vertical direction of the camera, as well as the angle between the throwing direction of the throwing tube and the vertical direction, and send the angle information to the edge intelligent device in real time;

[0065] The edge intelligent device is used to process the camera video stream and angle information, generate tracking control instructions, and send tracking control instructions to the delivery tube.

[0066] The agricultural machine in this embodiment 1 is a harvester.

[0067] Furthermore, the edge intelligent device includes a vehicle bucket detection model, a monocular ranging model and a vehicle bucket tracking controller; the vehicle bucket detection model is used to process the camera video stream and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model detects the actual distance between the center point of the vehicle bucket and the camera in the vertical direction through the pixel coordinates of the center point of the vehicle bucket in the video image and the angle sensor data; the vehicle bucket tracking controller generates tracking control instructions in real time through the pixel coordinates of the center point of the vehicle bucket in the video image and the actual distance between the center point of the vehicle bucket and the camera in the vertical direction.

Claims

1. A method for automatically tracking a bucket with a throwing tube based on edge intelligence, characterized in that: The following steps are involved: The camera collects video data, and the angle sensor measures the angle information between the camera and the throwing tube. The edge smart device obtains the camera video stream and decodes it to obtain the video frame sequence and angle information. The vehicle bucket detection model, monocular ranging model and vehicle bucket tracking controller are deployed in the edge intelligent device. The vehicle bucket detection model is used to process the decoded video frame sequence and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model is used to process the pixel coordinates and angle information of the center point of the vehicle bucket in the video image to obtain the vertical distance from the center of the vehicle bucket to the camera; the vehicle bucket tracking controller generates a tracking control instruction according to the pixel coordinates of the center point of the vehicle bucket in the video image and the monocular ranging result, and the tracking control instruction is sent to the throwing cylinder to control the throwing cylinder to rotate; The modeling method of the monocular ranging model is to establish a monocular ranging model M according to the pixel coordinates of the center point of the bucket in the video image at different camera heights and different camera angles and the actual distance between the center point of the bucket and the camera in the vertical direction. The specific formula is as follows: Wherein, w2 is the actual distance between the center of the truck bed and the camera in the vertical direction, h2 is the height of the truck bed, h3 is the height of the camera, α is the angle between the optical axis of the camera and the vertical direction of the camera, and (x1, y1) is the pixel coordinate of the center of the truck bed in the video image; The method for generating the tracking control instruction comprises the following steps: (1) The longitudinal rotation command is generated by calculating the angle γ between the horizontal direction of the center point of the truck bed and the throwing direction of the throwing tube according to the following formula: Among them, h1 is the height of the throwing tube, and w1 is the actual distance between the camera and the throwing tube in the vertical direction; Then calculate the longitudinal offset angle P between the throwing tube and the bucket according to the following formula: Among them, β is the angle between the throwing direction of the throwing tube and the vertical direction; Generate a longitudinal rotation instruction according to the longitudinal offset angle P between the throwing tube and the bucket; (2) Generation of lateral rotation instructions: The lateral rotation instructions are generated based on the difference between the pixel coordinates (x1, y1) of the center point of the truck bed in the video image and the coordinates (x2, y2) of the center point of the video image.

2. According to the method of automatic tracking bucket of the throwing tube based on edge intelligence in claim 1, it is characterized in that: The edge intelligent device supports GPU computing.

3. The method for automatically tracking a bucket with a throwing tube based on edge intelligence according to claim 1 is characterized in that: The model used to train the truck bed detection model is the deep neural network Faster RCNN, and the truck bed detection model and operating environment trained by the Faster RCNN model are deployed in the edge intelligent device.

4. The method for automatically tracking a bucket with a throwing tube based on edge intelligence according to claim 1 or 3 is characterized in that: The edge intelligent device performs frame skipping processing on the video frame sequence according to the running speed of the truck bucket detection model after training the Faster RCNN model, detects the pixel position of the truck bucket in the video image, and obtains the pixel coordinates of the center point of the truck bucket in the video image.

5. An automatic tracking bucket device for a throwing tube based on edge intelligence, characterized in that: It includes a throwing arm, a camera, a throwing barrel and a bucket. The lower end of the throwing arm is installed on the agricultural machinery, the throwing barrel is installed on the upper end of the throwing arm, the camera is installed in the middle of the throwing arm, and angle sensors are installed on the camera and the throwing barrel. The angle sensor, camera and throwing barrel are respectively connected to the edge intelligent device for communication, and the edge intelligent device is deployed in the agricultural machinery control system; The angle sensor is used to obtain the angle between the optical axis of the camera and the vertical direction of the camera, as well as the angle between the throwing direction of the throwing tube and the vertical direction, and send the angle information to the edge intelligent device in real time; The edge intelligent device is used to process the camera video stream and angle information, execute the edge intelligence-based automatic tracking bucket method for the throwing tube as described in any one of claims 1-4 to generate tracking control instructions, and send tracking control instructions to the throwing tube.

6. The device for automatically tracking a bucket with a throwing tube based on edge intelligence according to claim 5 is characterized in that: The edge intelligent device includes a vehicle bucket detection model, a monocular ranging model and a vehicle bucket tracking controller; the vehicle bucket detection model is used to process the camera video stream and detect the pixel coordinates of the center point of the vehicle bucket in the video image; the monocular ranging model detects the actual distance between the center point of the vehicle bucket and the camera in the vertical direction through the pixel coordinates of the center point of the vehicle bucket in the video image and the angle sensor data; the vehicle bucket tracking controller generates tracking control instructions in real time through the pixel coordinates of the center point of the vehicle bucket in the video image and the actual distance between the center point of the vehicle bucket and the camera in the vertical direction.

Images