Forage baler control method and control system based on visual navigation

Abstract

The invention relates to the field of agricultural machinery intelligent control, in particular to a pasture baler control method and system based on visual navigation, and the method comprises the steps: obtaining an operation site image, and determining the curvature, moisture content and sectional area of a to-be-baled strip based on the operation site image; based on the water content and the sectional area of the straw strips, determining the packaging advancing speed; a dynamic Bessel tracking trajectory is generated based on the curvature of the strip, and a movement control instruction is generated based on the dynamic Bessel tracking trajectory and the packaging advancing speed; determining a compression resistance parameter and a roller rotating speed based on the grass strip curvature and the grass strip water content, and generating a compression control instruction based on the compression resistance parameter and the roller rotating speed; and controlling the pasture packaging machine to carry out pasture packaging operation based on the movement control instruction and the compression control instruction. The forage grass packing efficiency and packing quality can be improved conveniently.

Description

Technical Field

[0001] The present application relates to the field of intelligent control of agricultural machinery, and in particular to a control method and control system for a forage baler based on visual navigation. Background Art

[0002] With the rapid development of modern agriculture, hay baling is an important part of animal husbandry production. The improvement of its efficiency and quality is of great significance to reducing production costs and improving feed utilization. Traditional hay baling operations mainly rely on manual operation. The relevant driver needs to visually judge the direction of the grass strips to be baled, and estimate the curvature of the grass strips based on experience, so as to manually adjust the steering angle of the steering wheel. It is also necessary to observe the cross-sectional area of ​​the grass strips to be baled, and manually adjust the baler gear to change the vehicle speed. At the same time, it is also necessary to judge the sound of the compressor by hearing, so as to manually adjust the pressure valve of the compression mechanism. That is, the relevant driver needs to synchronously control the vehicle speed, steering and compression mechanism to perform a complete hay baling operation.

[0003] However, this manual operation mode is highly dependent on personal experience, and the relevant drivers need to adjust multiple operating parameters at the same time. This is not only inefficient, but also easily affected by human factors, resulting in unstable packaging quality. Summary of the invention

[0004] In order to improve the efficiency and quality of forage baling, the present application provides a forage baler control method and control system based on visual navigation.

[0005] In a first aspect, the present application provides a forage baler control method based on visual navigation, which adopts the following technical solution: A forage baler control method based on visual navigation, comprising: Acquire an image of the work site, and determine the curvature, moisture content and cross-sectional area of ​​the grass strips to be baled based on the image of the work site; Determining a baling speed based on the moisture content of the grass strips and the cross-sectional area of ​​the grass strips; Generate a dynamic Bezier tracking trajectory based on the curvature of the grass strip, and generate a movement control instruction based on the dynamic Bezier tracking trajectory and the packing travel speed; Determining a compression resistance parameter and a roller speed based on the curvature of the grass strip and the moisture content of the grass strip, and generating a compression control instruction based on the compression resistance parameter and the roller speed; The forage baler is controlled to perform a forage baling operation based on the movement control instruction and the compression control instruction.

[0006] By adopting the above technical solution, the dynamic Bezier tracking trajectory generated based on the curvature of the grass strip can ensure that the baler performs baling operations along the path that best fits the shape of the grass strip, which improves the accuracy of the baling operation and facilitates the avoidance of material waste or mechanical wear caused by improper baling paths. The compression mechanism of the baler is controlled by the compression resistance parameters determined according to the curvature and moisture content of the grass strip, so as to guide the strength and method of the compression mechanism in the process of compressing forage, which helps to avoid excessive compression damage to the forage while ensuring the baling density, thereby facilitating the maintenance of the quality of the forage. Through multi-dimensional intelligent control and adjustment, it is convenient to comprehensively consider various influencing factors and realize the dual improvement of baling efficiency and quality. Through intelligent decision-making, it is convenient to reduce errors and uncertainties caused by human operation, thereby facilitating the improvement of efficiency and quality in the baling process.

[0007] In one possible implementation, the method further includes: Acquire forward sensing reflection signals, top sensing reflection signals, thermal imaging data, and millimeter wave radar data; Determine a corresponding environment point cloud map based on the front perception reflection signal and the top perception reflection signal, and perform plane fitting on the environment point cloud map to obtain a target contour map; Based on the thermal imaging data and the millimeter wave radar data, the obstacle features and the occurrence position of each obstacle feature at the forage baling site are determined, and based on each occurrence position, the corresponding obstacle feature is superimposed on the target contour map to obtain a three-dimensional digital twin map; Identifying undulation parameters in the three-dimensional digital twin map, and dynamically adjusting vibration suppression parameters based on the undulation parameters and the curvature of the grass strip; The compression control command is adjusted based on the vibration suppression parameter.

[0008] By adopting the above technical solution, by integrating the front sensing reflection signal and the top sensing reflection signal, it is convenient to comprehensively and accurately perceive the terrain undulations and grass distribution at the work site. By analyzing the thermal imaging data and millimeter wave radar data, it is convenient to accurately perceive the location of obstacles, and by superimposing the obstacle features on the target contour map, it is convenient for relevant drivers to accurately avoid various obstacles when performing visual navigation based on the three-dimensional digital twin map, thereby improving the safety of the baling operation. At the same time, by intelligently adjusting the compression control instructions, it is convenient to reduce the failure and downtime of the forage baler, thereby improving the continuity of the baling operation.

[0009] In one possible implementation, the method further includes: When the three-dimensional digital twin map contains a preset control feature, identifying control parameters corresponding to the movement control instruction and the compression control instruction; The control parameters are superimposed on the three-dimensional digital twin map to obtain an AR control map, and the AR control map is fed back to an AR device worn by a relevant driver; Acquiring actual control parameters of the relevant driver, comparing the actual control parameters with corresponding AR control parameters, and determining parameter correction values; When the parameter correction value is higher than a preset correction threshold, a correction prompt instruction is generated.

[0010] By adopting the above technical solution, by superimposing the control parameters into the three-dimensional digital twin map and generating an AR control map, the relevant drivers can intuitively see the operation path, obstacle location, control parameters and other information through the AR device they wear under the current driving situation, thereby reducing the difficulty of operation. In addition, the difference between the actual control parameters and the AR control parameters can also be identified, thereby providing the driver with accurate operation guidance.

[0011] In one possible implementation, the method further includes: Counting all the correction prompt instructions generated within the preset observation period and the generation time of each correction prompt instruction, and determining the correction prompt instruction with a generation number higher than the preset correction number and / or a generation frequency higher than the preset correction frequency as an abnormal correction prompt instruction; Determine the damaged object based on the abnormal correction prompt instruction, and determine the single damage value based on the AR control parameter and the abnormal control parameter corresponding to the abnormal correction prompt instruction; Acquire the amount of work to be packaged, and determine, based on the amount of work to be packaged and the single damage value, a simulated damage result of the damaged object after completing the amount of work to be packaged; The simulated damage results are superimposed on the AR control map to obtain a damage assessment view.

[0012] By adopting the above technical scheme, by analyzing the number of times and frequency of correction prompt instructions generated within a period of time, it is convenient to determine abnormal correction situations of multiple corrections or frequent corrections from multiple correction prompt instructions, and determine the damaged object corresponding to the abnormal correction situation. By combining the AR control parameters and abnormal control parameters corresponding to the abnormal correction prompt instructions, it is convenient to calculate the single damage value, so as to quantify the degree of damage. Then, through the amount of work to be packaged and the single damage value, it is convenient to simulate the damage result of the damaged object after completing the amount of work to be packaged. By superimposing the simulated damage result on the AR control map, it is convenient for relevant drivers to understand the consequences that may result from inaccurate operation, thereby indirectly improving the operation accuracy of relevant drivers.

[0013] In one possible implementation, the method further includes: Obtaining the actual packing speed corresponding to the current moment and the friction coefficient corresponding to the current road section, and determining the initial obstacle emergency braking distance based on the actual packing speed and the friction coefficient; Identifying an obstacle size and an obstacle type from the three-dimensional digital twin map, and determining a distance adjustment parameter based on the obstacle size and the obstacle type; A target obstacle emergency braking distance is determined based on the initial obstacle emergency braking distance and the distance adjustment parameter, and an emergency braking instruction is determined based on the target obstacle emergency braking distance and the three-dimensional digital twin map.

[0014] By adopting the above technical solution, by identifying the size and type of obstacles from the three-dimensional digital twin map, determining the distance adjustment parameters based on this information, and further refining the emergency braking distance, this adaptive adjustment facilitates providing more accurate braking measures for obstacles of different types and sizes, thereby facilitating the effective avoidance of collision risks. By accurately calculating the emergency braking distance and generating emergency braking instructions, the system can warn the driver in advance when necessary, giving him enough time to react or take preventive measures, which helps to reduce the tension of the relevant drivers and facilitates improving control accuracy.

[0015] In one possible implementation, the method further includes: When the three-dimensional digital twin map contains a grass quantity difference feature, determining a grass quantity difference value and a grass quantity difference direction based on the three-dimensional digital twin map; Based on the mapping relationship between the grass amount difference value and the preset lifting parameter, determining the target lifting parameter corresponding to the grass amount difference value; An adjustment instruction is generated based on the forage difference direction and the target lifting parameter to control the operating parameters of a picker in a forage baler.

[0016] By adopting the above technical solution, by accurately identifying the difference in grass amount and optimizing the operating parameters of the picker based on the difference in grass amount, it is convenient to reduce missed or repeated picking, thereby improving the working efficiency. At the same time, through the intelligent adjustment mechanism, it is convenient to control the picker to automatically adjust the operating parameters according to the actual grass amount, so as to ensure that the density of forage grass after packaging is uniform, which helps to improve packaging efficiency and packaging quality.

[0017] In a second aspect, the present application provides a control system, which adopts the following technical solution: A control system, the control system comprising: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the above-mentioned vision-based navigation forage baler control method.

[0018] In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium includes: a computer program that can be loaded by a processor and execute the above-mentioned vision-navigation-based forage baler control method.

[0019] In a fourth aspect, the present application provides a computer program product, which adopts the following technical solution: A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the control method of a forage baler based on visual navigation is implemented.

[0020] In summary, the present application includes at least one of the following beneficial technical effects: The dynamic Bezier tracking trajectory generated based on the curvature of the grass strip can ensure that the baler performs baling operations along the path that best fits the shape of the grass strip, which improves the accuracy of the baling operation and avoids material waste or mechanical wear caused by improper baling paths. The compression mechanism of the baler is controlled by the compression resistance parameters determined according to the curvature and moisture content of the grass strip, so as to guide the strength and method of the compression mechanism in the process of compressing forage. It helps to avoid excessive compression damage to the forage while ensuring the baling density, thereby maintaining the quality of the forage. Through multi-dimensional intelligent control and adjustment, it is convenient to comprehensively consider various influencing factors and realize the dual improvement of baling efficiency and quality. Through intelligent decision-making, it is convenient to reduce errors and uncertainties caused by human operation, thereby improving the efficiency and quality of the baling process.

[0021] By integrating the front sensing reflection signal and the top sensing reflection signal, it is convenient to comprehensively and accurately perceive the terrain undulations and grass distribution at the work site. By analyzing the thermal imaging data and millimeter-wave radar data, it is convenient to accurately perceive the location of obstacles. By superimposing the obstacle features on the target contour map, the relevant drivers can accurately avoid various obstacles when performing visual navigation based on the three-dimensional digital twin map, thereby improving the safety of the baling operation. At the same time, through intelligent adjustment of the compression control instructions, it is convenient to reduce the failure and downtime of the forage baler, thereby improving the continuity of the baling operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a forage baler control method based on visual navigation in an embodiment of the present application; Figure 2 It is a schematic diagram of an adjustment flow of a compression control instruction in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a control system in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following is combined with Figures 1 to 3 This application is described in further detail.

[0024] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the object's permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0027] Specifically, the embodiment of the present application provides a control method for a forage baler based on visual navigation, which is executed by a control system, and the control system can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application does not limit this.

[0028] refer to Figure 1 , Figure 1 : is a flow chart of a method for controlling a forage baler based on visual navigation in an embodiment of the present application, the method comprising steps S110 to S150, wherein: Step S110: Acquire the work site image, and determine the curvature, moisture content and cross-sectional area of ​​the grass strips to be baled based on the work site image.

[0029] Specifically, the operation site is the actual site where the grass strip baling operation is required. The operation site can be a production site of agriculture, animal husbandry or related processing industries. The on-site operation images can be collected by image acquisition equipment set up at the operation site and uploaded to the control system. Grass strips are formed by manually driving a lawn mower to cut the grass along a predetermined route to form uneven grass stubble, and then using a grass rake to gather the grass stubble. Since there may be uneven gullies or stones at the operation site, the grass strips formed by using a grass rake are generally serpentine or curved.

[0030] Before determining the curvature of the grass strips to be packaged based on the work site image, the work site image can be pre-processed by denoising, enhancing contrast, adjusting brightness, etc. to improve the quality of the on-site work image, and then the grass strips to be packaged can be identified from the work site image according to a preset feature recognition algorithm. The identification can be performed by features such as color, shape, and texture. The specific preset feature recognition algorithm is not specifically limited in the embodiments of the present application. After identifying the grass strips to be packaged, the improved YOLO (You Only Look Once) algorithm can be used to perform real-time detection of the bounding box of the grass strips to be packaged in the work site image, and then the DeepSORT (Simple Online and Realtime Tracking with a Deep Association Metric) algorithm can be used to track the grass strips to be packaged in real time. The center point of the grass strips to be packaged can be extracted through the tracked bounding box. The center line of the grass strips to be packaged is fitted by using curve fitting algorithms such as polynomial fitting and spline fitting. The fitted center line is discretized into a series of points, each point represents a position on the center line, and the curvature of each point on the discretized center line is calculated. Specifically, for a point P (x, y) on the center line, its curvature κ can be approximately calculated by the following formula. The formula for calculating the curvature of the grass strip is: ; Among them, κ is the curvature of the grass strip; are the first and second derivatives at that point on the center line, namely the slope of the tangent and the rate of change of the slope of the tangent.

[0031] A multispectral camera or sensor can be used to illuminate the grass strips to be baled at the work site, and the reflectivity of the grass strips to be baled at different wavelengths can be measured. By measuring the multispectral reflectivity data of multiple bands, the response characteristics of the grass strips to be baled in different spectral ranges can be understood. By importing the measured multispectral reflectivity data into the NDVI calculation formula, the NDVI value of the grass strips to be baled can be obtained. The NDVI value is a commonly used vegetation index used to evaluate the coverage and health of vegetation. After determining the NDVI value of the grass strips to be baled, the moisture content corresponding to the NDVI value can be determined based on the preset moisture content mapping relationship, so as to obtain the moisture content of the grass strips to be baled. The calculation formula of NDVI is: ; Among them, NIR is near infrared reflectivity, RED is red light reflectivity; The NDVI value range is from -1 to 1. The closer the NDVI value is to 1, the higher the vegetation coverage and the better the growth status. The NDVI value close to 0 indicates that the surface is mainly bare soil or rock. A negative NDVI value usually indicates water bodies or other non-vegetated surfaces.

[0032] The cross-sectional area of ​​the grass strips to be packaged can be identified from the work site image by using a preset edge detection algorithm, wherein the preset edge detection algorithm can be a Canny edge detection algorithm, and the specific algorithm is not specifically limited in the embodiments of the present application.

[0033] Since the curvature of the grass strips to be baled may affect the moving path of the forage baler, and the change in the moisture content of the grass strips may affect the physical properties of the grass, such as elasticity and toughness, thereby affecting the compression effect and baling density of the baler, the cross-sectional area of ​​the grass strips determines the amount of forage that the baler can pick up and feed each time during the baling and compression process. A grass strip cross-sectional area that is too large may cause the baler to overload, while a grass strip cross-sectional area that is too small may reduce the baling efficiency. Therefore, when controlling the forage baler, it is necessary to consider the curvature of the grass strips to be baled, the moisture content of the grass strips and the cross-sectional area of ​​the grass strips at the same time.

[0034] Step S120: Determine the baling speed based on the moisture content and cross-sectional area of ​​the grass strips.

[0035] Specifically, the moisture content of the grass strips is one of the key factors affecting the efficiency and quality of forage baling. Too high or too low moisture content of the grass strips may cause problems in the forage baling process. For example, when the moisture content of the grass strips is too high, it is easy to suffer from greater compression resistance during the baling and compression process of the forage, which may affect the baling speed. The cross-sectional area of ​​the grass strips can directly reflect the volume or weight of the forage per unit length. A larger cross-sectional area of ​​the grass strips means that more forage needs to be compressed and baled per unit length. If the baling speed does not correspond to the cross-sectional area of ​​the grass strips, there may be insufficient compression time, which may result in insufficient compression density or irregular shape. Therefore, it is necessary to determine the baling speed based on the moisture content of the grass strips and the cross-sectional area of ​​the grass strips. The specific implementation method for determining the baling speed can be: The mechanical efficiency coefficient, grass strip cross-sectional area and moisture content correction factor corresponding to the forage baler are obtained, and the mechanical efficiency parameters, grass strip cross-sectional area and moisture content correction factor are introduced into the baling speed calculation formula, wherein the mechanical efficiency coefficient is used to reflect the power of the forage baler, and can be directly retrieved from the control system through the number or characteristics of the forage baler. The control system contains mechanical efficiency coefficients corresponding to different forage balers, and the moisture content correction factor corresponding to the moisture content of the grass strip can be determined according to the preset factor correspondence relationship. The preset factor correspondence relationship contains moisture content correction factors corresponding to different moisture contents of the grass strips. The preset factor correspondence relationship can be determined by relevant staff according to historical experimental data and uploaded to the control system. The specific content is not specifically limited in the embodiment of this application. The calculation formula for the baling speed is: ; Among them, v is the packing speed; A is the cross-sectional area of ​​the grass strip; k is the mechanical efficiency coefficient; ρ is the moisture content correction factor.

[0036] Step S130: Generate a dynamic Bezier tracking trajectory based on the curvature of the grass strip, and generate a movement control instruction based on the dynamic Bezier tracking trajectory and the packing travel speed.

[0037] Specifically, a suitable Bezier curve type is selected according to the curvature of the grass strip. Different grass strip curvatures are adapted to different Bezier curve types, which can be determined according to a preset curve type mapping relationship. The preset curve type mapping relationship is the correspondence between the grass strip curvature and the Bezier curve type. The specific content is not specifically limited in the embodiment of the present application, and can be determined by relevant staff based on historical experimental data and uploaded to the control system. Among them, the Bezier curve type can be a quadratic Bezier curve, a cubic Bezier curve, etc., and then the starting point and end point of the grass strip to be packaged are identified from the work site image, and the control point of the Bezier curve is determined based on the starting point and the key point. The Bezier curve generation function in the preset software tool or the preset programming environment can be used to generate a Bezier curve according to each control point. The method of generating a Bezier curve according to each control point is not specifically limited in the embodiment of the present application, as long as it can ensure that the generated Bezier tracking path matches the curvature distribution of the grass strip.

[0038] Since the work site image is acquired in real time, the corresponding grass strips to be baled in the work site image also change in real time, so the generated Bessel tracking path is also dynamic. Mobile control instructions are generated based on the dynamic Bessel tracking path and the baling travel speed. The generated mobile control instructions can be sent by the control system to the controller of the forage baler to control the forage baler to perform baling operations on the grass strips to be baled according to the dynamic Bessel tracking path and the baling travel.

[0039] Step S140: Determine a compression resistance parameter and a roller speed based on the curvature and moisture content of the grass strips, and generate a compression control instruction based on the compression resistance parameter and the roller speed.

[0040] Specifically, different compression pressures are applied to the baling process for baling grass strips with different moisture contents. The higher the moisture content of the grass strips, the softer the corresponding grass. Therefore, a higher compression pressure may be required to achieve the target compression density. The compression resistance parameters adapted to the grass corresponding to different baling positions can be determined by the curvature of the grass strips and the moisture content of the grass strips. In addition, when the curvature of the grass strips is large, it indicates that the grass strips to be baled may be unevenly distributed. At this time, it may be necessary to adjust the compression resistance parameters corresponding to different baling positions according to the curvature of the grass strips to avoid local overpressure or underpressure. The compression resistance parameters adapted to the grass corresponding to different baling positions can be determined based on a preset compression resistance parameter mapping relationship, which is the correspondence between the moisture content of the grass strips and the compression resistance parameters. The pressure adjustment values ​​adapted to the grass corresponding to different baling positions can be determined based on a preset pressure adjustment value mapping relationship, which is the pressure adjustment value corresponding to the curvature of the grass strips. The specific preset compression pressure mapping relationship and the preset pressure adjustment value mapping relationship are not specifically limited in the embodiments of the present application, and can be determined by relevant staff based on historical experimental data and uploaded to the control system.

[0041] The roller speed is dynamically adjusted according to the cross-sectional area and moisture content of the straw strips. When the cross-sectional area of ​​the straw strips is large or the moisture content of the straw strips is high, the roller speed may need to be reduced to ensure sufficient compression; otherwise, the roller speed may be increased to improve the baling efficiency. The roller speeds corresponding to different parameter combinations of straw strip cross-sectional areas and straw strip moisture contents may be determined based on a preset roller speed mapping relationship. The preset roller speed mapping relationship is the roller speeds corresponding to different parameter combinations of straw strip cross-sectional areas and straw strip moisture contents. The specific content of the preset roller speed mapping relationship is not specifically limited in the embodiments of the present application and may be determined by relevant staff based on historical experimental data and uploaded to the control system.

[0042] Step S150: Control the forage baler to perform the forage baling operation based on the movement control instruction and the compression control instruction.

[0043] Specifically, the forage baler mainly includes a traveling operation area, a compression and baling area, a knotting area and a bale placing and weighing area, among which the movement control instructions are mainly used to control the traveling operation area, the compression control instructions are mainly used to control the compression and baling area, the knotting area and the bale placing and weighing area are used to cooperate with the traveling operation area and the compression and baling area to pack the forage.

[0044] For the embodiment of the present application, the dynamic Bezier tracking trajectory generated based on the curvature of the grass strip can ensure that the baler performs the baling operation along the path that best fits the shape of the grass strip, thereby improving the accuracy of the baling operation and avoiding material waste or mechanical wear caused by improper baling path. The compression mechanism of the baler is controlled by the compression resistance parameters determined according to the curvature and moisture content of the grass strip, so as to guide the strength and method of the compression mechanism in the process of compressing forage grass, which helps to avoid excessive compression damage to the forage grass while ensuring the baling density, thereby facilitating the maintenance of the quality of the forage grass. Through multi-dimensional intelligent control and adjustment, it is convenient to comprehensively consider various influencing factors and realize the dual improvement of baling efficiency and quality. Through intelligent decision-making, it is convenient to reduce errors and uncertainties caused by human operation, thereby facilitating the improvement of efficiency and quality in the baling process.

[0045] Furthermore, in order to improve the continuity of the packaging operation, the method provided in the embodiment of the present application further includes steps S210 to S250, such as Figure 2 As shown, where: Step S210: Acquire the front sensing reflection signal, the top sensing reflection signal, the thermal imaging data and the millimeter wave radar data.

[0046] Specifically, the front sensing reflection signal can be collected by a sensing device arranged in front of the forage baler. The front sensing device can be a binocular camera with a high dynamic range, wherein the high dynamic range technology can capture and display a wider range of brightness and color details than the standard dynamic range, and the binocular camera can capture two images of the same scene through two lenses placed side by side, and use the difference between the images to calculate the three-dimensional coordinates of the object. This technology is convenient for improving the accuracy of image recognition and enables the machine to better understand the spatial relationship. Therefore, the front sensing reflection signal collected by the binocular camera with a high dynamic range can better reflect the actual situation in front. The top sensing reflection signal can be collected by a sensing device arranged on the top of the forage baler, wherein the top sensing device can be a rotating laser radar, which can realize three-dimensional perception by simultaneously emitting and receiving multiple laser beams. The specific front sensing device and the top sensing device are not specifically limited in the embodiments of this application, and can be set by relevant staff according to actual needs. The thermal imaging data can be collected by the thermal imaging device of the equipment at the operation site and uploaded to the control system. The specific thermal imaging device is not specifically limited in the embodiments of this application. Millimeter-wave radar can accurately measure the distance between the target object and the radar. Its ranging accuracy is usually high. It can penetrate adverse weather conditions such as fog, rain, and snow and is not affected by light. The electromagnetic waves emitted by the millimeter-wave radar will be reflected back after hitting the object, and the data acquisition equipment can send the received reflected signal to the control system.

[0047] Step S220: Determine the corresponding environment point cloud map based on the front perception reflection signal and the top perception reflection signal, and perform plane fitting on the environment point cloud map to obtain a target contour map.

[0048] Specifically, point cloud data processing is performed on the front perception reflection signal and the top perception reflection signal to obtain an environmental point cloud map; the environmental point cloud map is divided into regions based on a preset clustering algorithm to obtain multiple divided point cloud region images, wherein the preset clustering algorithm may be DBSCAN, K-means, etc., and the specific clustering algorithm is not specifically limited in the embodiment of the present application; plane fitting is performed on each divided point cloud region image, and based on each divided point cloud region image and each corresponding plane fitting result, a three-dimensional contour map of each divided point cloud region image is determined; each three-dimensional contour map is spliced ​​to obtain a target contour map.

[0049] Step S230: Based on the thermal imaging data and millimeter wave radar data, determine the obstacle features and the location of each obstacle feature at the forage baling site, and superimpose the corresponding obstacle features onto the target contour map based on each location to obtain a three-dimensional digital twin map.

[0050] Specifically, the temperature abnormality area in the forage baling site can be identified and located from the thermal imaging data by a first preset feature recognition algorithm, and then the shape, size, temperature distribution and other features of the extracted and identified temperature abnormality area are used to determine the first obstacle feature information, and the first obstacle feature information includes the location of each first obstacle and each first obstacle feature. The clustering area in the forage baling site can be identified from the millimeter-wave radar data by a second preset feature recognition algorithm, and then the second obstacle feature information is determined according to the shape, size, position and other features corresponding to the clustering area. The second obstacle feature information includes the location of each second obstacle and each second obstacle feature. Since there are differences in the data source characteristics corresponding to the thermal imaging data and the millimeter-wave radar data, it is necessary to use two different feature recognition algorithms for obstacle feature recognition. The specific first preset feature recognition algorithm and the second preset feature recognition algorithm are not specifically limited in the embodiment of the present application. All obstacle feature information appearing at the forage baling site can be obtained by fusing the first obstacle feature information and the second obstacle feature information.

[0051] After obtaining the characteristic information and location of each obstacle, it can be superimposed on the previously generated target contour map. By fusing the three-dimensional characteristic information of the obstacle with the target contour map, a three-dimensional digital twin map containing the obstacle information can be generated.

[0052] Step S240: Identify the undulation parameters in the three-dimensional digital twin map, and dynamically adjust the vibration suppression parameters based on the undulation parameters and the curvature of the grass strips.

[0053] Step S250: adjusting the compression control command based on the vibration suppression parameter.

[0054] Specifically, terrain undulation parameters can be identified from a three-dimensional digital twin map based on a preset terrain analysis technology. The terrain undulation parameters include, but are not limited to, maximum height difference, average slope, local curvature, etc. The specific parameters are not specifically limited in the embodiments of the present application and can be set by relevant technical personnel according to actual needs. When the vibration suppression parameters are dynamically adjusted based on the undulation parameters and the curvature of the grass strips, the undulation parameters and the curvature of the grass strips can be input into a trained dynamic adjustment model, and the vibration suppression parameters can be gradually adjusted based on the undulation parameters corresponding to each packaging position to achieve dynamic adjustment of the vibration suppression parameters, wherein the vibration suppression parameters include, but are not limited to, vibration frequency, amplitude limit, shock absorber stiffness, etc. The trained dynamic adjustment model can simulate the forage packaging process. The model training process is not specifically limited in the embodiments of the present application, as long as the dynamic adjustment of the vibration suppression parameters can be achieved. The compression control instructions can be adjusted by adjusting the relevant compression control parameters in the compression control instructions according to the vibration suppression parameters.

[0055] For the embodiments of the present application, by integrating the front sensing reflection signal and the top sensing reflection signal, it is convenient to comprehensively and accurately perceive the terrain undulations and grass distribution at the work site. By analyzing the thermal imaging data and the millimeter wave radar data, it is convenient to accurately perceive the location of obstacles. By superimposing the obstacle features on the target contour map, it is convenient for relevant drivers to accurately avoid various obstacles when performing visual navigation based on the three-dimensional digital twin map, thereby improving the safety of the baling operation. At the same time, by intelligently adjusting the compression control instructions, it is convenient to reduce the failure and downtime of the grass baler, thereby facilitating the improvement of the continuity of the baling operation.

[0056] Furthermore, in order to provide accurate operation guidance for relevant drivers, the technical solution provided in the embodiment of the present application also includes: When the three-dimensional digital twin map contains preset control features, the control parameters corresponding to the movement control instructions and the compression control instructions are identified; the control parameters are superimposed on the three-dimensional digital twin map to obtain an AR control map, and the AR control map is fed back to the AR device worn by the relevant driver; the actual control parameters of the relevant driver are obtained, and the actual control parameters are compared with the corresponding AR control parameters to determine the parameter correction value; when the parameter correction value is higher than the preset correction threshold, a correction prompt instruction is generated.

[0057] Specifically, when visual navigation is performed based on the three-dimensional digital twin map, automatic control operations can be performed, and semi-automatic auxiliary operations can also be performed. When the three-dimensional digital twin map contains preset control features, it indicates that the forage baler performs semi-automatic auxiliary operations. That is, based on the manual control of the relevant driver, the control system can provide auxiliary control according to the three-dimensional digital twin map, such as path planning, obstacle avoidance, etc., thereby reducing the burden of manual operation, while also facilitating the improvement of the efficiency and accuracy of forage baling operations.

[0058] The control parameters contained in the movement control instructions and the compression control instructions can be identified based on the preset feature recognition algorithm. By operating based on the various control parameters, the forage baler can be controlled to perform baling operations according to the movement control instructions and the compression control instructions. The control parameters include but are not limited to the steering angle of the steering wheel, the throttle, the opening of the pressure valve, etc. The specific preset feature recognition algorithm is not specifically limited in the embodiment of the present application, as long as the control parameters contained in the movement control instructions and the compression control instructions can be identified. Since the three-dimensional digital twin map is dynamic, the determined control parameters are also dynamic.

[0059] After identifying the dynamic control parameters, the corresponding control parameters can be superimposed on the corresponding positions of the three-dimensional digital twin map according to the actual situation of each packaging position, so that the relevant drivers can directly perform packaging operations according to the AR control map received in the AR device after wearing the AR device, without the need to perform packaging operations through the control parameters provided by other third-party devices or materials. Among them, the AR device can be AR glasses, and the specific AR device is not specifically limited in the embodiment of this application. The control parameters input by the relevant drivers in the actual operation, such as actual speed, steering angle, compression setting, etc., are obtained by sensors, input devices or directly from the control system set on the forage baler, and the actual control parameters obtained are compared with the corresponding control parameters in the AR control map, the difference between the control parameters is calculated, and the parameter correction value corresponding to the operation difference is determined. When the parameter correction value is higher than the preset correction threshold, it indicates that the control operation of the relevant driver may have a large deviation. At this time, the relevant driver can be reminded by generating a correction prompt instruction. The correction prompt instruction may include specific warning information or operation instructions, etc. The specific content is not specifically limited in the embodiment of this application. The specific preset correction threshold can be determined by the relevant staff based on historical experimental data.

[0060] Furthermore, in order to improve the operating accuracy of relevant drivers, the technical solution provided in the embodiment of the present application may also include: Statistics are collected on all correction prompt instructions generated within a preset observation period and the generation time of each correction prompt instruction, and correction prompt instructions with a generation number higher than a preset correction number and / or a generation frequency higher than a preset correction frequency are determined as abnormal correction prompt instructions; the damaged object is determined based on the abnormal correction prompt instruction, and the single damage value is determined based on the AR control parameters and the abnormal control parameters corresponding to the abnormal correction prompt instruction; the amount of work to be packaged is obtained, and based on the amount of work to be packaged and the single damage value, the simulated damage result of the damaged object after the completion of the amount of work to be packaged is determined; the simulated damage result is superimposed on the AR control map to obtain a damage assessment view.

[0061] Specifically, the preset observation period is a period of time before the current moment. The duration corresponding to the preset observation period is not specifically limited in the embodiment of the present application, and can be determined by relevant staff based on historical experimental data and uploaded to the control system. By counting all correction prompt instructions generated within the preset observation period and the generation time of each correction prompt instruction, it is convenient to evaluate the operation of the relevant driver. If the number of times all correction prompt instructions are generated within the preset observation period is higher than the preset correction number, and / or the generation frequency is higher than the preset correction frequency, it may indicate that the relevant driver has poor operating performance during the actual operation process, which may be because the relevant driver relies too much on the control system or the operating guidance of others, and ignores his own operating skills and judgment, or it may be because judgments or processing are often based on personal experience during the actual operation.

[0062] The correction prompt instruction whose generation times in the preset observation period are higher than the preset correction times, and / or whose generation frequency is higher than the preset correction frequency, is determined as an abnormal correction prompt instruction, wherein the preset correction times and preset correction frequency are not specifically limited in the embodiment of the present application, and the number of abnormal correction prompt instructions is also not limited, and the damaged object corresponding to the abnormal correction prompt instruction is the control object corresponding to the control parameter with control difference, and the damaged object can be a steering wheel, a tire, a roller, etc. After the damaged object is determined, the control difference value can be determined based on the AR control parameter and the abnormal control parameter corresponding to the damaged object, and then the single damage degree is quantified according to the control difference value to obtain the single damage value.

[0063] The amount of work to be packaged refers to the amount of packaging work that the damaged object needs to participate in or complete in the future. It can be obtained through packaging work plans, task allocations or historical data. When determining the simulated damage result based on a single damage value and the amount of work to be packaged, the simulated damage model corresponding to the damaged object can be used to perform simulation operations to obtain a damaged result image. It is also possible to obtain the damaged value by quantifying the damage, and then attach the damaged value to the current damaged object to obtain a damaged result image to obtain the final simulated damaged result. The method for determining the simulated damaged result is not specifically limited in the embodiments of this application. By superimposing the simulated damaged result on the AR control map, it is convenient for relevant drivers to understand the consequences that may result from inaccurate operations, thereby indirectly improving the operating accuracy of relevant drivers.

[0064] Furthermore, in order to warn the driver in advance when necessary so that he or she has enough time to react or take preventive measures, the technical method provided in the embodiment of the present application also includes: Obtain the actual packaged travel speed corresponding to the current moment and the friction coefficient corresponding to the current road section, and determine the initial obstacle emergency braking distance based on the actual packaged travel speed and the friction coefficient; identify the obstacle size and obstacle type from the three-dimensional digital twin map, and determine the distance adjustment parameter based on the obstacle size and obstacle type; determine the target obstacle emergency braking distance based on the initial obstacle emergency braking distance and the distance adjustment parameter, and determine the emergency braking instruction based on the target obstacle emergency braking distance and the three-dimensional digital twin map.

[0065] Specifically, the actual packing speed corresponding to the current moment can be obtained by the relevant sensors, the road section features of the current road section can be located by the preset feature recognition algorithm, and then the friction coefficient corresponding to the current road section can be determined based on the preset friction coefficient mapping relationship, wherein the preset friction coefficient mapping relationship is the correspondence between the road section features and the friction coefficient, and the specific content is not specifically limited in the embodiments of this application. When determining the initial obstacle emergency braking distance based on the actual packing speed and the friction coefficient, the actual packing speed and the friction coefficient can be introduced into the initial obstacle emergency braking distance calculation formula, wherein the initial obstacle emergency braking distance calculation formula is: Dmin≤ ; Among them, μ is the friction coefficient; g is the acceleration of gravity; v is the actual packing speed.

[0066] In order to facilitate the improvement of braking timeliness, the obstacle size and obstacle type can be identified from the three-dimensional digital twin map based on the preset feature recognition algorithm, and the distance adjustment parameter corresponding to the current moment can be determined based on the obstacle size, obstacle type and preset adjustment parameter mapping relationship. The preset adjustment parameter mapping relationship is the distance adjustment parameter corresponding to the combination parameter of the obstacle size and obstacle type. The specific content is not specifically limited in the embodiment of this application. After determining the distance adjustment parameter, the initial obstacle emergency braking distance obtained in the above embodiment can be optimized based on the distance adjustment parameter, that is, the target obstacle emergency braking distance is D final ≤ Dmin + k, where k is the distance adjustment parameter. By accurately calculating the emergency braking distance and generating an emergency braking command, the system can warn the driver in advance when necessary.

[0067] Furthermore, in order to improve packaging efficiency and packaging quality, the method provided in the embodiment of the present application also includes: When the three-dimensional digital twin map contains grass amount difference features, the grass amount difference value and the forage difference direction are determined based on the three-dimensional digital twin map; based on the mapping relationship between the grass amount difference value and the preset lifting parameter, the target lifting parameter corresponding to the grass amount difference value is determined; based on the forage difference direction and the target lifting parameter, an adjustment instruction is generated to control the operating parameters of the picker in the forage baler.

[0068] Specifically, when the difference in the amount of forage on the left and right corresponding to a certain packing position in the three-dimensional digital twin map is greater than a preset difference value, it indicates that the three-dimensional digital twin map contains a forage amount difference feature. For example, when the difference in the amount of forage on the left and right is greater than 15%, it can be determined that there is a forage amount difference feature. At this time, the forage difference direction corresponding to the forage amount difference feature can be determined by identifying and analyzing the spatial distribution characteristics of the grass strips to be packed in the three-dimensional digital twin map.

[0069] According to the preset lifting parameter mapping relationship, the calculated grass amount difference value can be converted into the corresponding target lifting parameter. The preset lifting parameter mapping relationship is the correspondence between the grass amount difference value and the target lifting parameter. The target lifting parameter can be the tilt angle, lifting direction, lifting height, etc. of the picker. For example, when the difference in grass amount on the left and right is >15%, the picker tilt angle is automatically adjusted to the right. By accurately identifying the difference in grass amount and optimizing the picker's operating parameters based on the difference in grass amount, it is easy to reduce missed or repeated picking, thereby improving operating efficiency. At the same time, through the intelligent adjustment mechanism, it is easy to control the picker to automatically adjust the operating parameters according to the actual grass amount, so as to ensure that the density of the grass after packaging is uniform, which helps to improve packaging efficiency and packaging quality.

[0070] In an embodiment of the present application, a control system is provided, such as Figure 3 As shown, Figure 3 The control system 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the control system 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the control system 300 does not constitute a limitation on the embodiments of the present application.

[0071] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0072] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0073] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0074] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0075] The control system includes, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. Figure 3 The control system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0076] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0077] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in any of the above embodiments is implemented.

[0078] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0079] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A forage baler control method based on visual navigation, characterized in that: include: Acquire an image of the work site, and determine the curvature, moisture content and cross-sectional area of ​​the grass strips to be baled based on the image of the work site; Determining a baling speed based on the moisture content of the grass strips and the cross-sectional area of ​​the grass strips; Generate a dynamic Bezier tracking trajectory based on the curvature of the grass strip, and generate a movement control instruction based on the dynamic Bezier tracking trajectory and the packing travel speed; Determining a compression resistance parameter and a roller speed based on the curvature of the grass strip and the moisture content of the grass strip, and generating a compression control instruction based on the compression resistance parameter and the roller speed; The forage baler is controlled to perform a forage baling operation based on the movement control instruction and the compression control instruction.

2. The method for controlling a forage baler based on visual navigation according to claim 1, characterized in that: Also includes: Acquire forward sensing reflection signals, top sensing reflection signals, thermal imaging data, and millimeter wave radar data; Determine a corresponding environment point cloud map based on the front perception reflection signal and the top perception reflection signal, and perform plane fitting on the environment point cloud map to obtain a target contour map; Based on the thermal imaging data and the millimeter wave radar data, the obstacle features and the occurrence position of each obstacle feature at the forage baling site are determined, and based on each occurrence position, the corresponding obstacle feature is superimposed on the target contour map to obtain a three-dimensional digital twin map; Identifying undulation parameters in the three-dimensional digital twin map, and dynamically adjusting vibration suppression parameters based on the undulation parameters and the curvature of the grass strip; The compression control command is adjusted based on the vibration suppression parameter.

3. The control method of a forage baler based on visual navigation according to claim 2 is characterized in that: Also includes: When the three-dimensional digital twin map contains a preset control feature, identifying control parameters corresponding to the movement control instruction and the compression control instruction; The control parameters are superimposed on the three-dimensional digital twin map to obtain an AR control map, and the AR control map is fed back to an AR device worn by a relevant driver; Acquiring actual control parameters of the relevant driver, comparing the actual control parameters with corresponding AR control parameters, and determining parameter correction values; When the parameter correction value is higher than a preset correction threshold, a correction prompt instruction is generated.

4. The method for controlling a forage baler based on visual navigation according to claim 3, characterized in that: Also includes: Counting all the correction prompt instructions generated within the preset observation period and the generation time of each correction prompt instruction, and determining the correction prompt instruction with a generation number higher than the preset correction number and / or a generation frequency higher than the preset correction frequency as an abnormal correction prompt instruction; Determine the damaged object based on the abnormal correction prompt instruction, and determine the single damage value based on the AR control parameter and the abnormal control parameter corresponding to the abnormal correction prompt instruction; Acquire the amount of work to be packaged, and determine, based on the amount of work to be packaged and the single damage value, a simulated damage result of the damaged object after completing the amount of work to be packaged; The simulated damage results are superimposed on the AR control map to obtain a damage assessment view.

5. The method for controlling a forage baler based on visual navigation according to claim 2, characterized in that: Also includes: Obtaining the actual packing speed corresponding to the current moment and the friction coefficient corresponding to the current road section, and determining the initial obstacle emergency braking distance based on the actual packing speed and the friction coefficient; Identifying an obstacle size and an obstacle type from the three-dimensional digital twin map, and determining a distance adjustment parameter based on the obstacle size and the obstacle type; A target obstacle emergency braking distance is determined based on the initial obstacle emergency braking distance and the distance adjustment parameter, and an emergency braking instruction is determined based on the target obstacle emergency braking distance and the three-dimensional digital twin map.

6. The method for controlling a forage baler based on visual navigation according to claim 2, characterized in that: Also includes: When the three-dimensional digital twin map contains a grass quantity difference feature, determining a grass quantity difference value and a grass quantity difference direction based on the three-dimensional digital twin map; Based on the mapping relationship between the grass amount difference value and the preset lifting parameter, determining the target lifting parameter corresponding to the grass amount difference value; An adjustment instruction is generated based on the forage difference direction and the target lifting parameter to control the operating parameters of a picker in a forage baler.

7. A control system, characterized in that: The control system includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute a forage baler control method based on visual navigation as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: include: A computer program is stored which can be loaded by a processor and executes a method for controlling a forage baler based on visual navigation as described in any one of claims 1-6.

9. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the steps of a forage baler control method based on visual navigation as described in any one of claims 1 to 6.

Images