A method and device for constructing a multidimensional image database

By constructing a multidimensional image database and subdividing broccoli head image data, the problem of insufficient accuracy in broccoli head detection and grading was solved, enabling efficient detection and grading under diverse conditions and supporting the automation and intelligence of broccoli selective harvesting robots.

CN118628942BActive Publication Date: 2024-11-12ZHEJIANG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411083707.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-12
Estimated Expiration
2044-08-08

Smart Images

  • Figure CN118628942B_ABST
    Figure CN118628942B_ABST
Patent Text Reader

Abstract

A method for constructing a multidimensional image database belongs to the field of agricultural information technology, and includes the following steps: step 1, a division strategy for multidimensional image data of broccoli curd; step 2, selection and design of a collection device for multidimensional image data of broccoli curd; step 3, constructing multidimensional image data of broccoli curd. The present invention also provides a device for constructing a multidimensional image database. The present invention aims to obtain multidimensional image data of curds for training and verifying a target detection model based on YOLOv8, so that the model can meet the detection accuracy of curds in a variety of practical application scenarios, and at the same time verify the generalization ability of the algorithm model, which is crucial to improving the robustness of the visual system under diverse conditions, and provides key technical support for the research and development, application and promotion of air-ground collaborative broccoli selective harvesting robot systems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of agricultural informatization, and in particular relates to a method and device for constructing a multidimensional image database. Background Art

[0002] In order to solve the problems of high cost, low efficiency and difficulty in standardization of manual inspection and harvesting of broccoli, an air-ground collaborative broccoli selective harvesting technology is proposed. The high-throughput remote sensing visible light image data of broccoli is obtained by the aerial drone to estimate the yield of broccoli, which is crucial for making harvesting decisions; the ground-side harvesting robot obtains the visible light image and depth image information of broccoli to determine whether the broccoli in the field of view meets the standard for harvesting. Through air-ground collaboration, the unification of harvesting decision and harvesting execution can be effectively completed, fundamentally solving the automation and intelligent problems of broccoli selective harvesting. Whether it is the aerial visible light image obtained by the drone gimbal or the visible light image obtained by the depth camera of the harvesting robot, it needs to be processed by the visual algorithm model to obtain the position, diameter, coloring and morphology of the flower ball. The quality of the visual detection algorithm directly determines the breadth of the application scenarios of the air-ground collaborative broccoli selective harvesting vision system, which is crucial for the subsequent promotion of the industrial application of the air-ground collaborative broccoli selective harvesting robot system.

[0003] The images of flower heads obtained from the air-ground will be different under different varieties, planting patterns, land soil, agronomic operations, weather conditions, etc. The planting pattern of broccoli will make the posture and degree of occlusion of the flower heads different; the land soil and agronomic operations will affect the growth status and situation of broccoli; the color and grain morphology of flower heads of different varieties and qualities will also vary to varying degrees; different lighting conditions will affect the quality, brightness, contrast, etc. of the obtained images. The broccoli flower head images obtained under the above different conditions show diverse characteristics, which makes the detection and classification of broccoli flower heads difficult, and seriously hinders the research, development, application and promotion of the vision system of broccoli harvesting robots in air-ground collaboration.

[0004] Among the current target detection methods, there are unsupervised learning that does not require labels, mainly including Haar feature cascade classifiers, HOG features combined with SVM, and traditional threshold segmentation; semi-supervised learning that requires a small number of labels, such as Semi-Supervised GAN, consistency regularization, etc.; supervised learning that requires a large number of labels, mainly including YOLO series, RCNN series, DETR series, etc. Due to the complexity of broccoli images, most of the target detection algorithms in agricultural applications are currently developed based on deep learning. The newly launched YOLOv8 has shown strong capabilities in recognition accuracy and efficiency, and is very suitable for the detection of flower heads in diverse broccoli data. There are significant differences between the diverse image data of broccoli heads, and mixed training is difficult to obtain effective results. If the flower heads in the obtained images are divided into multiple dimensions according to their characteristics, the fine classification of the data can be better completed. Each model can focus on the image data under specific conditions, which will effectively improve the accuracy of target detection under different dimensions. In addition, the performance of each model under the change of a single or a few variables can be controlled, making it easier to diagnose model problems and improvement directions. Therefore, it is urgent to build a multi-dimensional image database of broccoli heads to provide sufficient training sets and verification sets for developing and optimizing the YOLOv8-based segmented target detection algorithm model.

[0005] In short, the current harvesting decision and harvesting execution of open-air broccoli rely on manual field inspections and visual judgment to ultimately facilitate the harvesting operation. The air-ground collaborative broccoli selective harvesting vision system can effectively replace manpower to provide a data basis for the automation of harvesting decision and execution. The target detection algorithm model based on YOLOv8 requires multi-dimensional broccoli flower head image data to complete training and verification in order to meet the requirements of flower head detection and grading under different conditions. Therefore, it is necessary to invent a systematic method and device for constructing a multi-dimensional image database of broccoli flower heads. Summary of the invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention provides a method and device for constructing a multidimensional image database, aiming to obtain multidimensional image data of flower curds for training and verifying a target detection model based on YOLOv8, so that the model can meet the detection accuracy of flower curds in a variety of practical application scenarios, and at the same time verify the generalization ability of the algorithm model, which is crucial to improving the robustness of the visual system under diverse conditions, and provides key technical support for the research and development, application and promotion of the air-ground collaborative broccoli selective harvesting robot system.

[0007] To achieve the above technical objectives, the present invention adopts a method for constructing a multidimensional image database, comprising the following steps:

[0008] Step 1, a division strategy for multi-dimensional image data of broccoli heads; by using the multi-scale image data obtained from the air-ground end, the broccoli heads are further subdivided in multiple dimensions based on the presentation characteristics of the broccoli heads, the multiple dimensions include dimension 1, i.e. different planting patterns of the aerial end images; dimension 2, i.e. different growth potentials of the aerial end images; dimension 3, i.e. different varieties of the aerial end images, including tall plants and short plants; dimension 4, i.e. different lighting conditions of the aerial end images, further divided into two conditions of different lighting intensities and different solar altitude angles and quantitatively grouped; dimension 5, i.e. different lighting conditions of the ground end images, further divided into three conditions of sunlight, after sunset, and shading and artificial light sources and quantitatively grouped; dimension 6, i.e. different varieties of the ground end images;

[0009] Step 2, selecting and designing a device for collecting multi-dimensional image data of broccoli heads;

[0010] Step 3. Construct multi-dimensional image data of broccoli head. For broccoli images of different dimensions, formulate a method for collecting head image data. For aerial images, according to the research of the planting market, the sparse leaves after the growth control operation are set as a quantitative value when studying different planting modes and varieties. When studying different varieties and growth potentials, the staggered planting of two rows in a single ridge is set as a quantitative value. Under different planting modes, the sparse leaves of short plants and lush leaves of tall plants, as well as the different varieties planted in two rows in a single ridge are completed. Broccoli image data is collected under different lighting conditions. For ground images, the image data of broccoli under different lighting conditions is obtained, as well as the image data of different sizes and quality grades of different varieties under shading conditions.

[0011] Furthermore, in the step 1, after field investigation, data pre-collection and analysis, the characteristics of the broccoli head images obtained under different conditions were summarized, and a division strategy for the broccoli head multidimensional image database was given, which provided direction for the model segmentation and optimization of the subsequent target detection algorithm. The broccoli head multidimensional image database summarized the characteristics of multidimensional data from four levels: multi-scale, multi-dimensional, multi-condition, and multi-variable.

[0012] The multi-scale image data includes large-scale surface data obtained by the aerial drone and small-scale point data obtained by the ground cart. The image data obtained by the drone aerial photography contains the position and diameter information of flower balls in a large area. Due to the shooting distance of m and the complex lighting and planting conditions, it is difficult to obtain more accurate detailed information about the flower balls. m≥10 meters. The position and diameter information of the flower balls of the entire broccoli field can be obtained later through two-dimensional reconstruction software; the image information obtained by the ground cart contains the information of several flower balls in a small area of ​​view. Due to the shooting distance of n, 0.5 meters ≤n≤1.5 meters, in addition to obtaining position and size information, detailed information such as flower ball color, shape and grain morphology can also be obtained.

[0013] Aerial images are divided into four dimensions based on field research and analysis due to different lighting conditions, planting patterns, planting varieties, and agronomic operations. Ground images are divided into two dimensions based on field research and analysis due to different lighting conditions and planting varieties. The division strategies of these six dimensions are introduced separately, among which dimensions 1 to 4 are about aerial images, and dimensions 5 to 6 are about ground images.

[0014] Dimension 1, different planting modes, single ridge and single row means planting one row of broccoli in one ridge, which is usually used for breeding experiments or mechanized automatic harvesting experiments. This planting mode is convenient for corresponding mechanical and experimental operations, but the average yield per mu is lower than other modes; multiple rows and scattered planting means that multiple rows of broccoli are planted on each large ridge of broccoli, and the middle spacing is less than the preset limit. It is a small area managed by individual households, usually dozens of acres. This planting mode is more suitable for manual water and fertilizer management and can effectively reduce costs; single ridge and two rows staggered means that two rows of broccoli are planted alternately on each ridge. This planting mode is usually adopted by large-scale cooperatives, and the planting area is usually hundreds of acres or even thousands of acres. Laying smart drip irrigation pipes on the center line of the ridge for smart water and fertilizer control is the current development trend. Under different planting modes, the influence of adjacent broccoli plants on each other may cause different imaging effects of the flower heads in the image. In the single-ridge and single-row mode, the distance between adjacent broccoli is large and the influence on each other is small; in the multi-row scattered planting mode, the distance between adjacent broccoli is relatively small, and some leaves of adjacent plants may block the flower heads; when the single-ridge and two-row staggered planting is carried out, the plants are close to each other in the direction of the ridge line, and the plant growth may squeeze the flower heads, causing them to tilt slightly, and the leaves in the direction of the ridge line will be slightly blocked.

[0015] Dimension 2, different growth, is mainly caused by agronomic operations (fertilization, weeding, irrigation, growth control, etc.), soil and climate conditions. The reasons are complicated, so we mainly focus on the results shown in the pictures. The result is generally manifested as different relative growth of leaves and flower heads of plants. The most typical is to take growth control measures, generally apply potions (Magic Fengling or potassium dihydrogen phosphate), mainly to reduce the nutrients absorbed by the leaves, so that the nutrients are more concentrated in the flower heads to increase yields. As a result, the flower heads will be relatively larger, the leaves will be smaller and the number will be reduced. In the picture, the flower heads are more prominent than the leaves, which is conducive to the detection of the flower heads. Therefore, according to the different growth of broccoli, this dimension is divided into two conditions: lush leaves and sparse leaves.

[0016] Dimension 3, different varieties, mainly including tall plants such as Meiqing and Hanxiu, and short plants such as Tailu 630 and Xiyingmen. Although the color and shape of the flower balls of different qualities may be different, it is difficult to capture these detailed features because the height of remote sensing images is generally greater than 10 meters, so more attention is paid to the overall shape of the plant. The characteristics of tall plants are large, tall and compact leaves, while the characteristics of short plants are small, short and loose leaves. In the aerial photos, the characteristic of tall plants is that when the flower balls in the center receive external light, if the light intensity is high and direct (the solar altitude angle is large, generally greater than 75 degrees), the surface of the flower balls receives sufficient light, which is manifested as high brightness and relatively simple detection; at other times, the brightness is dim and detection is relatively difficult. The characteristic of short plants is that the flower balls in the center are more thoroughly exposed. Compared with tall plants, they are more likely to receive external light when the solar altitude angle is low, and they can be clearly displayed even when the light intensity is low.

[0017] Dimension 4, different lighting conditions, mainly include different light intensity and solar altitude angle. Different light intensity will directly affect the brightness of aerial images. Although the gimbal camera will automatically adjust the various parameters of the camera to adapt to different light intensities, when the light intensity is too low, the gimbal camera will first increase the aperture. However, when the aperture is adjusted to the maximum and the brightness is still low, the exposure time will be increased. If the exposure time is too long, shooting the target during the flight will cause motion blur, which will affect the accuracy of target detection. Therefore, it is crucial to find the critical natural light intensity suitable for remote sensing aerial photography of broccoli fields while ensuring a certain target detection accuracy. Therefore, under different light intensity conditions, the cloudy data (natural light intensity greater than or equal to 800lx) is selected to illustrate the detection rate of flower balls at high light intensity, while the gradual light intensity (natural light intensity range is 0~800lx) data in the early morning and evening is used to illustrate the detection rate of flower balls at different light intensities. Different solar altitude angles will directly affect the relative brightness of the flower ball and leaves. When the solar altitude angle is greater than 75 degrees, the sunlight is direct and the entire flower ball is exposed to the sunlight. The brightness of the flower ball photographed by the gimbal is high and very obvious compared to the leaves. When the solar altitude angle is 15-75 degrees, the sunlight is oblique, and the leaves may block part of the flower ball. The lower the angle, the larger the blocking area, making some leaves obviously bright, while some flower balls appear to be alternating between light and dark, which increases the difficulty of detection. When the solar altitude angle is less than 15 degrees, the sunlight is oblique and the light intensity is low, some flower balls appear to be alternating between light and dark and the overall brightness is darker, which increases the difficulty of detection.

[0018] Dimension 5, a dataset of broccoli heads under different lighting conditions obtained on the ground side, mainly includes three conditions; Condition 1, when there is sunlight (the solar altitude angle is greater than 0), the broccoli head image is obtained by using a ground-side cart with a depth camera. Because the shadows of the leaves and the structure of the ground-side cart produced by the sun may cause the head to appear light and dark in the photo, which will increase the difficulty of detection and image segmentation; Condition 2, when there is no sunlight (the solar altitude angle is less than or equal to 0), such as after the sun sets, the light intensity (natural light intensity is less than or equal to 800lx) gradually decreases. The brightness of the image will decrease, and the detection of the flower head will become difficult. Studying the detection accuracy of the flower head under different light intensities is crucial to guide when to add artificial light sources, so that the harvesting robot can perform the harvesting operation normally at night; Condition 3, shading boards and shading cloth strips are laid around and on the top of the data collection cart frame to create a closed environment for data collection space, which can effectively isolate the influence of external natural light. The top frame is equipped with lamps with continuously adjustable light intensity. They are used as a stable light source to quantitatively obtain pictures of broccoli flower heads under different light intensities (artificial light intensity 0~3000lx). The data set obtained by condition 3 is mainly to study the influence of different light intensities on the detection accuracy of broccoli flower heads under the conditions of shading and artificial lighting, and to select the minimum light intensity to save energy while ensuring a certain accuracy of flower head detection; at the same time, the comparison between condition 3 and condition 1 can show whether shading and lighting are needed; the comparison between condition 3 and condition 2 can show how much the ambient light intensity is reduced to, and how much lighting intensity after shading can meet the night work of the harvesting robot.

[0019] Dimension 6, the ground end obtains a data set of broccoli heads of different varieties, mainly to enable the target detection algorithm to complete transfer learning between different varieties, to verify the adaptability and robustness of the algorithm to different varieties, and to optimize the algorithm to improve the generalization ability of the algorithm model. For different varieties, the size classification is first completed, that is, the diameter of the head is less than 12cm, the diameter is between 12cm and 14cm, and the diameter is greater than 14cm. This is one of the standards for harvesting broccoli. Size classification is based on target detection, image segmentation, circular fitting algorithm and depth information, and it is necessary to collect visible light images and depth images of the head. On the basis of size classification, it is necessary to complete quality classification for heads with a diameter greater than or equal to 12cm, mainly including extracting detailed features such as the color and morphology of the head grains, and measuring the roundness of the head contour to determine the quality of the head, corresponding to different harvesting operations. After being picked by the robot, the superior and qualified products are placed in the superior box and qualified box respectively, while the unqualified heads are picked and thrown into the field. The heads with a diameter less than 12cm are not harvested for the time being. Therefore, it is necessary to complete the graded data collection of broccoli of different varieties and qualities.

[0020] Furthermore, in step 2, the device for collecting the multi-dimensional image data of the broccoli head is composed of an aerial drone and a ground cart;

[0021] The aerial drone includes a high-precision RTK equipped with drones, which is used for spatial positioning of the drone; the drone body includes the drone flight control, frame, rotor and motor, which is the carrier for performing flight missions; the gimbal camera is used to obtain high-throughput remote sensing images of the broccoli field;

[0022] The ground-side cart is a three-dimensional frame structure built by several aluminum profiles of different sizes relying on corner pieces. Its structure is easy to transform and can adapt to broccoli plots with different planting patterns. It is also inexpensive and easy to build, making it very suitable for field data collection in broccoli fields. The overall cart structure is a basic structure with a length of 1 meter, a width of 1 meter, and a height of 1 meter. Other accessories are installed and placed on the cart structure. Four adjustable LED lights of the same specifications are connected in parallel and installed on the four frames on the top of the cart and tilted downward at 45 degrees, just so that the light is projected to the ground on the opposite side, which can maximize the utilization of the light; four adjustable LED lights of the same specifications are connected in parallel to ensure that the brightness can change consistently, and they are installed in symmetrical positions, which can theoretically ensure that the light intensity in the space is symmetrical about the vertical line of the center of the entire structure; the built-in program of the control board Jetson Xavier NX adjusts the duty cycle of the PWM signal to control the intensity of the LED light. The universal wheels of the cart can adapt to the movement of broccoli fields; portable lithium batteries can power the entire ground-side data collection system. Control board Jetson Xavier NX is the controller of the ground-side data acquisition system, which controls the depth camera, all light intensity sensors and RTK to collect and store data; the keyboard and mouse are used to check and operate the program; the small display is used to visualize the collected light intensity and image data during the data acquisition process to check the availability of the data; after shading, the inner light intensity sensor is used to measure the ambient light intensity inside the cart, and the outer light intensity sensor is used to measure the natural ambient light intensity outside the cart; the center light intensity sensor is used for calibration after data acquisition is completed, and the value of the inner light intensity sensor is calibrated to the center light intensity sensor; the horizontal center profile is used to install the center light intensity sensor in the center of the structure during the calibration process. The intersection of the line and the horizontal plane of the average height of the flower head is used to complete the light intensity calibration operation; the central light intensity sensor and the central profile are only used during light intensity calibration. When collecting flower head image data on the ground, they need to be removed, otherwise they will block the line of sight of the depth camera and may scratch the broccoli during the movement of the cart; a depth camera is installed on each side of the top of the cart, and the lens is facing downward to obtain RGB images and depth information. The dual depth camera can obtain images of the same broccoli from different angles, effectively improving the rate of collecting broccoli images and depth data. At the same time, the dual depth cameras obtain the same broccoli flower head image data from different angles, and the accuracy of the algorithm can be verified later.The control board Jetson Xavier NX, the internal light intensity sensor and the adjustable LED light form a closed-loop light intensity control system for shading and artificial lighting. After setting different target light intensity values, the control board Jetson Xavier NX controls the brightness of the LED light through the built-in Python script, and the internal light intensity sensor monitors the ambient light intensity in real time and feeds the data back to Jetson XavierNX, the control panel compares the current light intensity with the target light intensity, and adjusts the brightness of the LED light according to the difference until the ambient light intensity approaches the target value, automatically and quantitatively controlling the ambient light intensity in the cart frame when shading and manually lighting; the shading plate is made of lightweight aluminum alloy sheet, which is fixed to the surrounding frame by bolts to block the natural ambient light around the outside world. The left and right sheets in the moving direction are longer, and the front and rear sheets are shorter. The reason is that sufficient height space needs to be reserved in the front and rear to allow the broccoli plants in the field to pass normally during the movement of the cart; the shading plate (aluminum alloy sheet) is fixed to the top of the frame and the upper part of the frame by bolts to block the natural light coming from the top and upper part; the shading cloth strips are located at the bottom of the frame around The shading board and shading cloth strips can block the natural light from the bottom to a great extent, which is very important for quantitatively studying the influence of artificial light intensity on flower head recognition rate. The network version of RTK is used to measure the spatial location of the ground cart. The spatial location of the flower head in the picture can be solved by the RGB image and depth image information obtained by the depth camera, and the relative installation position between the two. This is convenient for the subsequent ground cart and the aerial drone to obtain the data of broccoli in the same plot, and can be verified with each other to evaluate the accuracy of the target detection, image segmentation and contour fitting algorithm model for the flower head. ;

[0023] In step 3, based on the divided broccoli curd multi-dimensional image data and the constructed air-ground end data acquisition system, a method for collecting curd image data is formulated for broccoli images of different dimensions;

[0024] In terms of the dimensions of the aerial image, according to current research on the planting market, the planting pattern of two rows staggered in a single ridge and sparse leaves after growth control operations are more common. Therefore, these two conditions are set as quantitative when considering changes in conditions in other dimensions.

[0025] The image data acquisition method of dimension 1 uses a drone equipped with a gimbal camera and RTK to scan and photograph broccoli plots in different planting patterns (single ridge and single row, multiple rows of scattered planting, and single ridge and two rows staggered) with short plants and sparse leaves and tall plants with lush leaves, and obtain image data under different lighting conditions. Data was not collected for short plants with lush leaves and tall plants with sparse leaves under different planting patterns because these two situations are neither as exposed to the sun as when the short plants have sparse leaves, nor as dark as when the tall plants have lush leaves, so they were excluded to save data collection and analysis costs.

[0026] The method for acquiring dimension 2 image data has been completed when acquiring dimension 1 data. That is, a drone equipped with a gimbal camera and RTK is used to scan and photograph broccoli plots with different growth conditions (lush leaves and sparse leaves) planted in two rows in a staggered manner in a single ridge to obtain image data under different lighting conditions.

[0027] The 3D image data acquisition method uses a drone equipped with a gimbal camera and RTK to complete aerial scanning of broccoli plots with different varieties (greater than or equal to 3 types of tall and short plants) planted in two rows in a staggered manner on a single ridge, and obtains image data under different lighting conditions for training and verifying the generalization ability of the target detection model between different varieties.

[0028] The acquisition method of image data in dimension 4 uses a drone equipped with a gimbal camera and RTK to complete the acquisition of image data under different light intensity conditions under the above conditions. This has already acquired image data in the first three dimensions and does not require additional acquisition; the following describes more data acquisition details, which are divided into two conditions of different light intensities and different solar altitude angles. There are three acquisition times for different light intensities: 1 is cloudy; 2 is early morning, before the sun rises; 3 is evening, after the sun sets. These three time periods are selected to exclude the influence of the solar altitude angle and only consider the influence of the brightness of the natural environment. The light intensity of cloudy image data is relatively stable, and there is basically no obvious shadow. Compared with other light intensity conditions, flower balls are easier to detect. At present, major research institutions have completed corresponding experiments. The two time periods of early morning before the sun rises and evening after the sun sets are selected because pictures of the gradual change of natural light intensity can be measured, which is helpful to study the influence of natural light intensity on the flower ball recognition rate, and derive the critical light intensity to ensure the flower ball recognition rate, which can provide a reference for the application of drone remote sensing technology in field flower ball recognition. When acquiring aerial images of broccoli fields, the control board Jetson Xavier NX was used to connect to the light intensity sensor BH1750 and set it to continuous H resolution mode 2 (0.5lx resolution, the highest resolution mode). The data of natural light intensity during the shooting process and the corresponding Beijing time were recorded throughout the process. The timestamp was then used to complete the correspondence between light and image data, to complete the grouping of images in different light intensity ranges, and to quantitatively study the effect of light intensity on the accuracy of flower head detection.

[0029] Data acquisition under different solar altitude conditions. The range of solar altitude angles every day is related to the geographical location and season (for example, the solar altitude angle of the broccoli planting base in Taizhou, Zhejiang Province is no higher than 38 degrees in winter, while the solar altitude angle of the broccoli planting base in Yancheng can reach up to 78 degrees in summer). By consulting the solar altitude angle website (https: / / www.suncalc.org / ), obtain the highest solar altitude angle of the day in that season (generally at noon of the day) and the corresponding time point, and determine the other solar altitude angles and corresponding time points of the day by subtracting 15 degrees from the solar altitude angle, and complete the aerial photography flight before and after the determined time point to obtain the image data of broccoli under different solar altitude conditions, and organize the aerial photos according to different solar altitudes for subsequent research. Taking Xiangshui County, Yancheng, Jiangsu Province, where broccoli was planted in the first half of the year as an example, it is mainly divided into three solar altitude angle intervals, greater than 75 degrees at noon, 15~75 degrees in the morning and afternoon, and 0~15 degrees in the early morning and evening.

[0030] The data acquisition method of dimension 5 is to use the above-mentioned ground-end cart to obtain the broccoli head data set under different lighting conditions, and obtain pictures containing detailed information of broccoli head at a distance of nearly 1 meter. The planting pattern (dimension 1), plant growth (dimension 2) and height (dimension 3) of broccoli have little effect on the detection of ground-end head, so the ground-end data set no longer considers these dimensions. This dimension contains 3 different lighting conditions. Condition 1 is the acquisition of broccoli head data set under natural sunlight. Choose sunny weather, modify the ground-end cart into a size suitable for moving in the target broccoli field, remove the shading board and shading cloth strips, and obtain broccoli head picture data under different natural lighting conditions; Condition 2 is the acquisition of broccoli head data set under different natural light intensity conditions. In order to isolate the influence of the solar altitude angle, choose the time period of cloudy days, before the sun rises and after the sun sets, modify the ground-end cart into a size suitable for moving in the target broccoli field, remove the shading board and shading cloth strips, and install an external light intensity sensor to measure the external ambient light intensity. The light intensity data on cloudy days is relatively stable. The two time periods of early morning before the sun rises and evening after the sun sets are selected because we can collect pictures of the gradual change of light intensity. This is helpful to study the influence of different light intensities on the flower head recognition rate under natural light conditions, and derive the critical natural light intensity to ensure the flower head recognition rate. It can provide a reference for the automatic harvesting robot to find a suitable light intensity working range without artificial light source when working on cloudy days, early morning and evening. Condition 3: Acquisition of broccoli flower head data set under shading and adding artificial light source. The ground-end cart is modified into a size suitable for moving in the target broccoli field, and a shading plate and shading cloth strips are installed. The internal light intensity sensor and adjustable LED light are installed to measure the changes in the ambient light intensity of the internal space of the cart after shading and artificial lighting. The light intensity control closed-loop system consisting of the control board Jetson Xavier NX, the internal light intensity sensor and the adjustable LED light is used to control the change of the ambient light intensity in the interior space of the cart, so as to obtain the image data set under different light intensities under the conditions of shading and using artificial light sources, which is used for subsequent training and verification of the target detection algorithm under the conditions of shading and using artificial light sources, and is compared with conditions 1 and 2 respectively to illustrate the importance of shading and artificial lighting. In addition, under the conditions of shading and using artificial light sources, the image data set under different light intensities can be used to verify the accuracy of the target detection algorithm model for flower ball detection under different light intensities, which is of great significance for selecting the minimum light intensity to save lighting energy while ensuring a certain accuracy rate.

[0031] In addition, the internal light intensity sensor cannot be installed on the central axis of the frame during the data collection process to measure the light intensity of the plane of average growth height of the flower balls, because this will block the flower balls in the field of view of the depth camera and may damage the flower balls during the movement of the cart; after the data collection is completed, in order to obtain the light intensity of the plane of average growth height of the flower balls with more reference value, it is necessary to add a light intensity calibration process; this calibration process is to keep the position of the internal light intensity sensor unchanged, install the central profile for installing the central light intensity sensor at the point where the central vertical line of the frame intersects the growth plane of the flower balls, and obtain and record the values ​​of the internal light intensity sensor and the central light intensity sensor by changing the brightness of the adjustable LED lamp, and then the data relationship between the two light intensity sensors can be fitted to obtain the light intensity value of the internal light intensity sensor corresponding to the light intensity value of the central flower ball surface.

[0032] The data acquisition method of dimension 6 uses the aforementioned ground-end cart to obtain data of different sizes and quality grades of different varieties. The cart needs to be shaded and artificially illuminated, which helps to eliminate the influence of external light and create stable lighting conditions, which is crucial for the extraction of features such as broccoli head coloring and morphology. First, the image data of the head of broccoli of the same variety of size and quality grading is acquired, and then the method is applied to other varieties (at least 3 kinds) to facilitate the transfer learning and generalization ability verification of the algorithm model in the later stage. The size grading mainly focuses on the diameter size of the broccoli head, which is generally divided into three grades: less than 12cm, 12~14cm, and greater than 14cm. The visible light image and depth image data of the head of the three size grades and the corresponding real measurement data manually obtained with a caliper are obtained respectively, which are used to verify the accuracy of the head detection, image segmentation, and contour fitting algorithm model in the future. The quality grading is judged based on the coloring and morphology of the broccoli head grains. High-quality broccoli florets are generally dark green with uniform color, firm, and uniformly sized buds without yellowing, blackening, or flowering. Low-quality broccoli florets usually have one or more of the following manifestations: uneven color, blackening or purple color in some parts of the florets, cat's eyes (uneven flower size), scattered flowers, yellowing flowers, or small flowers. The identification of broccoli quality requires the invitation of broccoli quality identification experts to conduct on-site identification in order to standardize the standards for subsequent target detection model algorithm training and verification set production.

[0033] The present invention also discloses a device for constructing a multidimensional image database, including a partitioning strategy module, a collection device selection and design module, and a broccoli head multidimensional image data construction module, each module sequentially executes the processing of steps 1 to 3 of the construction method.

[0034] The beneficial effects of the present invention are mainly manifested in: obtaining multi-dimensional image data of flower bulbs for training and verifying a target detection model based on YOLOv8, so that the model can meet the detection accuracy of flower bulbs in a variety of practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the partitioning strategy of the multidimensional image database.

[0036] Figure 2 It is a schematic diagram of a device for collecting multi-dimensional image data. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings.

[0038] Reference Figure 1 and Figure 2 , a method for constructing a multidimensional image database in this embodiment includes steps 1 to 3.

[0039] Step 1, a division strategy for multi-dimensional image data of broccoli heads; by obtaining multi-scale image data from the air-ground end, the broccoli heads are further subdivided in multiple dimensions based on the presentation characteristics of the broccoli heads, and the multiple dimensions include dimension 1, i.e., different planting patterns in the aerial end images; dimension 2, i.e., different growth potentials in the aerial end images; dimension 3, i.e., different varieties in the aerial end images, including tall plants and short plants; dimension 4, i.e., different lighting conditions in the aerial end images, further divided into two situations of different lighting intensities and different solar altitude angles and quantitatively grouped; dimension 5, i.e., different lighting conditions in the ground end images, further divided into three situations of sunlight, after sunset, and shading and artificial light sources and quantitatively grouped; dimension 6, i.e., different varieties in the ground end images.

[0040] After field research, data pre-collection and analysis, the characteristics of broccoli head images obtained under different conditions were summarized, such as Figure 1 The figure shows the partitioning strategy of the broccoli head multidimensional image database, which provides a direction for the model segmentation and optimization of the subsequent target detection algorithm. The broccoli head multidimensional image database summarizes the characteristics of multidimensional data from four levels: multi-scale, multi-dimensional, multi-conditional, and multi-variable.

[0041] Multi-scale data includes large-scale surface data obtained by drones in the air and small-scale point data obtained by carts on the ground. The image data obtained by drone aerial photography contains the location and size information of flower heads in a large area, and the location and size information of flower heads in the entire broccoli field can be obtained through two-dimensional reconstruction software; the image information obtained by the cart on the ground contains the information of several flower heads in a small area, in addition to the location and size information, it also contains detailed information such as flower head color, shape and grain morphology.

[0042] Aerial images are divided into four dimensions after field investigation and analysis due to differences in lighting conditions, planting patterns, planting varieties, and agronomic operations. Ground images are divided into two dimensions after field investigation and analysis due to differences in lighting conditions and planting varieties. The division strategies of these six dimensions are introduced below. Dimensions 1 to 4 are about aerial images, and dimensions 5 to 6 are about ground images.

[0043] Dimension 1, different planting modes, single ridge and single row means planting one row of broccoli in one ridge, which is usually used for breeding experiments or mechanized automatic harvesting experiments. This planting mode is convenient for corresponding mechanical and experimental operations, but the average yield per mu is lower than other modes; multiple rows and scattered planting means that multiple rows of broccoli are planted on each large ridge of broccoli, and the middle spacing is less than the preset limit. It is a small area managed by individual households, usually dozens of acres. This planting mode is more suitable for manual water and fertilizer management and can effectively reduce costs; single ridge and two rows staggered means that two rows of broccoli are planted alternately on each ridge. This planting mode is usually adopted by large-scale cooperatives, and the planting area is usually hundreds of acres or even thousands of acres. Laying smart drip irrigation pipes on the center line of the ridge for smart water and fertilizer control is the current development trend. Under different planting modes, the influence of adjacent broccoli plants on each other may cause different imaging effects of the flower heads in the image. In the single-ridge and single-row mode, the distance between adjacent broccoli is large and the influence on each other is small; in the multi-row scattered planting mode, the distance between adjacent broccoli is relatively small, and some leaves of adjacent plants may block the flower heads; when the single-ridge and two-row staggered planting is carried out, the plants are close to each other in the direction of the ridge line, and the plant growth may squeeze the flower heads, causing them to tilt slightly, and the leaves in the direction of the ridge line will be slightly blocked.

[0044] Dimension 2, different growth, is mainly caused by agronomic operations (fertilization, weeding, irrigation, growth control, etc.), soil and climate conditions. The reasons are complicated, so we mainly focus on the results shown in the pictures. The result is generally manifested as different relative growth of leaves and flower heads of plants. The most typical is to take growth control measures, generally apply potions (Magic Fengling or potassium dihydrogen phosphate), mainly to reduce the nutrients absorbed by the leaves and let the nutrients be more concentrated in the flower heads to increase yield. As a result, the flower heads will be relatively larger in size, the leaves will be smaller in size and fewer in number, and the flower heads will be more prominent in the pictures relative to the leaves, which is conducive to the detection of flower heads.

[0045] Dimension 3, different varieties, mainly including tall plants such as Meiqing and Hanxiu, and short plants such as Tailu 630 and Xiyingmen. Although the color and shape of the flower balls of different qualities will be different, it is difficult to capture these detailed features because the height of remote sensing images is generally greater than 10 meters, so more attention is paid to the overall shape of the plant. The characteristics of tall plants are large, tall and compact leaves, while the characteristics of short plants are small, short and loose leaves. In the aerial photos, the characteristic of tall plants is that when the flower balls in the center receive external light, if the light intensity is high and direct (the solar altitude angle is large, generally greater than 75 degrees), the surface of the flower balls receives sufficient light, which is manifested as high brightness and relatively simple detection; at other times, the brightness is dim and detection is more difficult. The characteristic of short plants is that the flower balls in the center are more thoroughly exposed, and compared with tall plants, they are more likely to receive external light, and can be clearly displayed even when the light intensity is low.

[0046] Dimension 4, different lighting conditions, mainly include different light intensities and solar altitude angles. Different light intensity conditions will directly affect the brightness of aerial images. Although the gimbal camera will automatically adjust the various parameters of the camera to adapt to different light intensities, when the light intensity is too low, the gimbal camera will first increase the aperture. However, when the aperture is adjusted to the maximum and the brightness is still low, the exposure time will be increased. If the exposure time is too long, shooting the target during flight will cause motion blur, which will affect the accuracy of target detection. Therefore, it is crucial to find the critical natural light intensity while ensuring a certain target detection accuracy. Therefore, under different lighting conditions, the cloudy data is selected to illustrate the flower ball detection rate at high light intensity, and the data obtained by the natural gradual light intensity in the early morning and evening is used to illustrate the flower ball detection rate at different light intensities. Different solar altitude angles will directly affect the relative brightness of the flower ball and leaves. When the solar altitude angle is greater than 75 degrees, the sunlight is direct and the entire flower ball is exposed to the sunlight. The brightness of the flower ball photographed by the gimbal is high and very obvious compared to the leaves. When the solar altitude angle is 15-75 degrees, the sunlight is oblique, and the shadow of the leaves may block part of the flower ball. The lower the angle, the larger the blocked area may be. At this time, the brightness of some leaves will be more obvious, and some flower balls will appear alternating between light and dark, which increases the difficulty of detection. When the solar altitude angle is less than 15 degrees, the sunlight is oblique and the light intensity is low. Some flower balls will appear alternating between light and dark and the overall brightness is darker, which increases the difficulty of detection.

[0047] Dimension 5, the broccoli head dataset under different lighting conditions obtained on the ground side, mainly includes three conditions; Condition 1, when there is sunlight (the solar altitude angle is greater than 0), the broccoli head image is obtained by using the ground-side cart with a depth camera, because the shadow of the leaves and the shadow of the ground-side cart structure caused by sunlight may cause the head to appear light and dark in the photo, which will increase the difficulty of detection and image segmentation; Condition 2, when there is no sunlight (the solar altitude angle is less than or equal to 0), such as after the sun sets, the light intensity gradually decreases, which will cause the brightness of the picture to decrease, and the detection of the head will become difficult. Studying the detection accuracy of the head under different light intensities is crucial to guide when to add artificial light sources, so that the harvesting robot can perform the harvesting operation normally at night; Condition 3, shading boards and shading cloth strips are laid around and on the top of the data collection cart frame to create a closed environment for data collection space, which can effectively isolate the influence of external light; lamps with continuously adjustable light intensity are installed around the top frame as a stable light source to quantitatively obtain pictures of broccoli heads under different light intensities. The data set obtained under condition 3 is mainly used to study the effect of different light intensities on the accuracy of broccoli head detection under the conditions of shading and artificial lighting, and to select the minimum light intensity to save energy while ensuring a certain accuracy of head detection. At the same time, the comparison between condition 3 and condition 1 can indicate whether shading and lighting are needed. The comparison between condition 3 and condition 2 can indicate how much the ambient light intensity is reduced and how much light intensity of artificial light source can meet the night work of the harvesting robot.

[0048] Dimension 6, the ground end obtains a data set of broccoli heads of different varieties, mainly to enable the target detection algorithm to complete transfer learning between different varieties, to verify the adaptability and robustness of the algorithm to different varieties, and to optimize the algorithm to improve the generalization ability of the algorithm model. For different varieties, the size classification is first completed, that is, the diameter of the head is less than 12cm, the diameter is between 12cm and 14cm, and the diameter is greater than 14cm. This is one of the standards for harvesting broccoli. Size classification is based on target detection, image segmentation, circular fitting algorithm and depth information, and it is necessary to collect visible light images and depth images of the head. On the basis of size classification, it is necessary to complete quality classification for heads with a diameter greater than or equal to 12cm, mainly including extracting detailed features such as the color and morphology of the head grains, and measuring the roundness of the head contour to determine the quality of the head, corresponding to different harvesting operations. After being picked by the robot, the superior and qualified products are placed in the superior box and qualified box respectively, while the unqualified heads are picked and thrown into the field. The heads with a diameter less than 12cm are not harvested for the time being. Therefore, it is necessary to complete data collection on graded broccoli of different varieties.

[0049] Step 2: Select and design a device for collecting multi-dimensional image data of broccoli florets.

[0050] The device for collecting multi-dimensional image data of broccoli florets consists of an aerial drone and a ground cart.

[0051] The aerial drone includes a high-precision RTK101 equipped with the drone, which is used for the spatial positioning of the drone; the drone body 102 includes basic structures such as drone flight control, frame, rotor and motor, and is the carrier for performing flight missions; the gimbal camera 103 is used to obtain high-throughput remote sensing image data of the broccoli field.

[0052] The structure 204 of the ground-end cart is a three-dimensional frame structure constructed by several aluminum profiles of different sizes relying on corner pieces. The structure is easy to modify to adapt to broccoli plots with different planting patterns, and is inexpensive and easy to build, making it very suitable for field data collection in broccoli fields. Figure 2The cart structure 204 shown in the figure is a basic structure with a length of 1 meter, a width of 1 meter and a height of 1 meter. Other accessories are installed and placed on the cart structure 204. After four adjustable LED lights 203 of the same specifications are connected in parallel, they are installed on the four frames on the top of the cart and tilted downward at 45 degrees, just so that the light is projected to the ground and the edges, which can maximize the utilization of the light. The four adjustable LED lights 203 of the same specifications are connected in parallel, which means that the brightness can change consistently and are installed in symmetrical positions, which can theoretically ensure that the light intensity in the space is symmetrical about the vertical line of the center of the entire structure. The control board Jetson Xavier NX214 is used to adjust the duty cycle of the PWM signal to control the intensity of the LED light; the universal wheel 208 of the cart can adapt to the movement of broccoli fields. The portable lithium battery 201 can power the entire ground-end data acquisition system. The control board Jetson Xavier NX214 is the controller of the ground-end data acquisition system, which controls the depth camera 202, all light intensity sensors and RTK210 to collect and store data. Keyboard 211 and mouse 212 are used to check and operate the program when necessary. Small display 209 is used to visualize the collected light intensity and image data during the data collection process to check the availability of the data. 205 and 213 are the internal light intensity sensor BH1750 and the external light intensity sensor BH1750 respectively. After shading, the internal light intensity sensor 205 is used to measure the ambient light intensity inside the cart, and the external light intensity sensor 213 is used to measure the natural ambient light intensity outside the cart. The central light intensity sensor 215 is used for calibration operations after data collection is completed, and the value of the internal light intensity sensor 205 is calibrated to the central light intensity sensor 215. During the calibration process, the horizontal central profile 216 is used to install the central light intensity sensor 215 at the intersection of the vertical line of the center of the structure and the horizontal plane of the average height of the flower ball to complete the light intensity calibration operation. The central light intensity sensor 215 and the central profile 216 are only used during light intensity calibration. They need to be removed when collecting flower head image data on the ground side, otherwise they will block the depth camera's line of sight and may scratch the broccoli during the movement of the cart. The model of the depth camera 202 is D435i. A depth camera 202 is installed on each side of the top of the cart, and the lens is facing downward to obtain RGB images and depth information. The dual depth cameras can obtain images of the same broccoli from different angles, effectively improving the rate of obtaining broccoli images and depth images. At the same time, the dual depth cameras obtain the same broccoli flower head image data from different angles, and the accuracy of the algorithm can be verified later.The control board Jetson Xavier NX214, the internal light intensity sensor 205 and the adjustable LED light 203 constitute a closed-loop light intensity control system for shading and artificial lighting. After setting different target light intensity values, the control board Jetson Xavier NX214 controls the brightness of the LED light through a built-in Python script. The internal light intensity sensor 205 monitors the ambient light intensity in real time and feeds the data back to Jetson Xavier NX. The control board compares the current light intensity with the target light intensity and adjusts the brightness of the LED light according to the difference until the ambient light intensity approaches the target value. In this way, the ambient light intensity in the cart frame can be automatically and quantitatively controlled during shading and artificial lighting. The shading plate 206 is made of a lightweight aluminum alloy sheet for lightweight considerations. It is fixed to the surrounding frame by bolts to block the natural ambient light from the outside. The left and right sheets in the moving direction are longer, and the front and rear sheets are shorter. The reason is that sufficient height space needs to be reserved in the front and rear to allow the broccoli plants in the field to pass normally during the movement of the cart. The shading plate 206 (aluminum alloy sheet) is fixed to the top of the frame and the upper part of the frame by bolts to block the natural light coming from the top and upper part. The shading cloth strip 207 is located at the lower part of the frame. Due to its flexible characteristics, it can well adapt to various environments when the cart is moving in the field, and is used to block the natural light coming from the bottom. The combination of the shading plate 206 and the shading cloth strip 207 can block the natural ambient light to a great extent, which is crucial for quantitatively studying the effect of the intensity of artificial light on the recognition rate of flower balls. The network version of RTK209 is used to measure the spatial location of the ground cart. The spatial location of the flower head in the picture can be solved by the RGB image and depth image information obtained by the depth camera 202 and the installation position between the two. This facilitates the subsequent mutual verification when the ground cart and the aerial drone obtain the data of broccoli in the same plot, so as to evaluate the accuracy of the target detection, image segmentation and contour fitting algorithm model for the flower head.

[0053] Step 3. Construct multi-dimensional image data of broccoli head. For broccoli images of different dimensions, formulate a method for collecting head image data. For aerial images, according to the research of the planting market, the sparse leaves after the growth control operation are set as a quantitative value when studying different planting modes and varieties. When studying different varieties and growth potentials, the staggered planting of two rows in a single ridge is set as a quantitative value. Under different planting modes, the sparse leaves of short plants and lush leaves of tall plants, as well as the different varieties planted in two rows in a single ridge are completed. Broccoli image data is collected under different lighting conditions. For ground images, the image data of broccoli under different lighting conditions is obtained, as well as the image data of different sizes and quality grades of different varieties under shading conditions.

[0054] Based on the divided multi-dimensional image data of broccoli head and the constructed air-ground data acquisition system, a method for collecting head image data is developed for broccoli images of different dimensions.

[0055] In terms of the dimensions of the aerial image, according to current research on the planting market, the planting pattern of two rows staggered in a single ridge and sparse leaves after growth control operations are more common. Therefore, these two conditions are set as quantitative when considering changes in conditions in other dimensions.

[0056] The image data acquisition method of dimension 1 uses a drone equipped with a gimbal camera and RTK to scan and photograph broccoli plots in different planting patterns (single ridge and single row, multiple rows of scattered planting, and single ridge and two rows staggered) with short plants and sparse leaves and tall plants with lush leaves, and obtain image data under different lighting conditions. Data was not collected for short plants with lush leaves and tall plants with sparse leaves under different planting patterns because these two situations are neither as exposed to the sun as when the short plants have sparse leaves, nor as dark as when the tall plants have lush leaves, so they were excluded to save data collection and analysis costs.

[0057] The method for acquiring dimension 2 image data has been completed when acquiring dimension 1 data. That is, a drone equipped with a gimbal camera and RTK is used to scan and photograph broccoli plots with different growth conditions (lush leaves and sparse leaves) planted in two rows in a staggered manner in a single ridge to obtain image data under different lighting conditions.

[0058] The 3D image data acquisition method uses a drone equipped with a gimbal camera and RTK to complete aerial scanning of broccoli plots with different varieties (3 or more types of tall and short plants) planted in two rows in a staggered manner on a single ridge, and obtains a number of pictures under different lighting conditions to train and verify the generalization ability of the target detection model between different varieties.

[0059] The acquisition method of image data in dimension 4 uses a drone equipped with a gimbal camera and RTK to complete the acquisition of image data under different light intensity conditions under the above conditions. This has already acquired image data in the first three dimensions and does not require additional acquisition. The following describes more details of data acquisition, which is divided into two conditions: different light intensities and different solar altitude angles. There are three acquisition times for different light intensities: 1 is cloudy; 2 is early morning, before the sun rises; 3 is evening, after the sun sets. These three time periods are selected to exclude the influence of the solar altitude angle and only consider the influence of the brightness of the natural environment. The light intensity of cloudy image data is relatively stable, and there is basically no obvious shadow. Compared with other light intensity conditions, flower balls are easier to detect. At present, major research institutions have completed corresponding experiments. The two time periods of early morning before the sun rises and evening after the sun sets are selected because pictures in the process of gradual light intensity change can be measured, which is helpful to study the influence of light intensity on flower ball recognition rate, and derive the critical light intensity to ensure flower ball recognition rate, which can provide a reference for the application of drone remote sensing images. When acquiring aerial images of broccoli fields, the control board Jetson Xavier NX was used to connect to the light intensity sensor BH1750 and set it to continuous H resolution mode 2 (0.5lx resolution, the highest resolution mode). The data of natural light intensity during the shooting process and the corresponding Beijing time were recorded throughout the process. The timestamp was then used to complete the correspondence between light and image data, to complete the grouping of images in different light intensity ranges, and to quantitatively study the effect of light intensity on the accuracy of flower head detection.

[0060] Data acquisition under different solar altitude conditions. The range of solar altitude angles per day is related to the geographical location and season (for example, the solar altitude angle of the broccoli planting base in Taizhou, Zhejiang Province is no higher than 38 degrees in winter, while the solar altitude angle of the broccoli planting base in Yancheng can reach up to 78 degrees in summer). By consulting the solar altitude angle website (https: / / www.suncalc.org / ), obtain the highest solar altitude angle of the day in that season (generally at noon of the day) and the corresponding time point, and determine the other solar altitude angles and corresponding time points of the day by subtracting 15 to 20 degrees from the solar altitude angle, and complete the aerial photography flight before and after the determined time point, obtain the image data of broccoli under different solar altitude conditions, and organize the aerial photos according to different solar altitude angles for subsequent research.

[0061] The data acquisition method of dimension 5 is to use the above-mentioned ground-end cart to obtain the broccoli head data set under different lighting conditions, and obtain pictures containing detailed information of broccoli head at a distance of nearly 1 meter. The planting mode (dimension 1), plant growth (dimension 2) and height (dimension 3) of broccoli have little effect on the detection of ground-end head, so the ground-end data set no longer considers these dimensions. This dimension includes 3 different lighting conditions. Condition 1 is the acquisition of broccoli head data set under the natural lighting conditions of the sun. Select sunny weather, modify the ground-end cart into a size suitable for moving in the target broccoli field, remove the shading plate 206 and the shading cloth strip 207, and obtain data under different natural lighting conditions; Condition 2 is the acquisition of broccoli head data set under different natural light intensity conditions. In order to isolate the influence of the solar altitude angle, select the time period of cloudy days, before the sun rises and after the sun sets, modify the ground-end cart into a size suitable for moving in the target broccoli field, remove the shading plate 206 and the shading cloth strip 207, and install the external light intensity sensor 213 to measure the external ambient light intensity. The light intensity data on cloudy days is relatively stable. The two time periods of early morning before the sun rises and evening after the sun sets are selected because pictures of the gradual change of light intensity can be collected. This is helpful to study the influence of different light intensities on the flower head recognition rate under natural light conditions, and to derive the critical natural light intensity to ensure the flower head recognition rate. This can provide a reference for the automated harvesting robot to find a suitable light intensity working range without the help of artificial light sources when working on cloudy days, early mornings and evenings; Condition 3: Obtaining the broccoli flower head data set under shading and adding artificial light sources, modifying the ground-end cart into a size suitable for moving in the target broccoli field, installing shading plates 206 and shading cloth strips 207, installing internal light intensity sensors 205 and adjustable LED lights 203, to measure the changes in ambient light intensity in the interior space of the cart after shading and artificial lighting, and using the control board Jetson Xavier The light intensity control closed-loop system composed of NX214, internal light intensity sensor 205 and adjustable LED 203 lamp controls the change of ambient light intensity in the interior space of the cart to obtain image data sets at different light intensities under the conditions of shading and using artificial light sources, which are used for subsequent training and verification of the target detection algorithm under the conditions of shading and using artificial light sources, and are compared with conditions 1 and 2 respectively to illustrate the importance of shading and artificial lighting; in addition, under the conditions of shading and using artificial light sources, the image data sets at different light intensities can be used to verify the accuracy of flower ball detection under different light intensities of the target detection algorithm model, which is of great significance for selecting the minimum light intensity to save lighting energy while ensuring a certain accuracy rate. In addition, the internal light intensity sensor cannot be installed on the central axis of the frame during the data collection process to measure the light intensity of the flower ball average growth height plane, because this will block the flower ball in the field of view of the depth camera. Therefore, after the data collection is completed, in order to obtain the light intensity of the flower ball average growth height plane with more reference value, a light intensity calibration process needs to be added.The calibration process is to keep the position of the inner light intensity sensor 205 unchanged, and install the central profile 216 for installing the central light intensity sensor 215 at the point where the vertical line of the frame center intersects the flower ball growth plane, such as. Figure 2 By changing the brightness of the adjustable LED light 203 to obtain and record the values ​​of the inner light intensity sensor 205 and the central light intensity sensor 215, the data relationship between the two light intensity sensors can be fitted to obtain the light intensity value of the inner light intensity sensor 205 corresponding to the light intensity value of the central flower ball surface.

[0062] The data acquisition method of dimension 6 is to use the above-mentioned ground-end cart to obtain data of different sizes and quality grades of different varieties. The cart needs to be shaded and artificially illuminated, which helps to eliminate the influence of external light intensity and create stable lighting conditions, which is crucial for the extraction of features such as broccoli coloring. First, the image data of the flower heads of the same variety of broccoli for size and quality grading is acquired, and then the method is applied to other varieties (at least 3 kinds) to facilitate the transfer learning and generalization ability verification of the algorithm model in the later stage. The size grading mainly focuses on the diameter size of the broccoli flower head, which is generally divided into three grades: less than 12cm, 12~14cm, and greater than 14cm. The visible light image and depth image data of the flower heads of the three size grades and the corresponding real measurement data of the flower head diameter obtained by calipers are obtained respectively, which are used for the subsequent verification of the accuracy of the flower head detection, image segmentation, and contour fitting algorithm models. The quality grading is judged based on the coloring and morphology of the broccoli flower heads. High-quality broccoli florets are generally dark green with uniform color, firm, and uniform bud size without yellowing, blackening, or blooming. Low-quality broccoli florets usually have one or more of the following: uneven floret color, blackening or purpleness in some parts of the floret, cat's eyes (uneven floret size), scattered flowers, yellowing florets, or small flowers. Broccoli quality identification requires the invitation of broccoli quality identification experts to conduct on-site identification to standardize the standards for subsequent target detection model algorithm training and validation set production.

[0063] This embodiment also discloses a device for constructing a multidimensional image database, including a partitioning strategy module, a collection device selection and design module, and a broccoli head multidimensional image data construction module, each module sequentially executes the processing of steps 1 to 3 of the construction method.

[0064] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for constructing a multidimensional image database, characterized in that: The method comprises the following steps: Step 1, a division strategy for multi-dimensional image data of broccoli heads; by using the multi-scale image data obtained from the air-ground end, the broccoli heads are further subdivided in multiple dimensions based on the presentation characteristics of the broccoli heads, wherein the multiple dimensions include dimension 1, i.e., different planting patterns of the aerial end images; dimension 2, i.e., different growth potentials of the aerial end images; dimension 3, i.e., different varieties of the aerial end images, and the different varieties of the aerial end include tall plants and short plants; dimension 4, i.e., different lighting conditions of the aerial end images, further divided into two conditions of different lighting intensities and different solar altitude angles and quantitatively grouped; dimension 5, i.e., different lighting conditions of the ground end images, further divided into three conditions of sunlight, after sunset, and shading and artificial light sources and quantitatively grouped; dimension 6, i.e., different varieties of the ground end images; Step 2, selecting and designing a device for collecting multi-dimensional image data of broccoli heads; Step 3, construct multi-dimensional image data of broccoli head; formulate a method for collecting head image data for broccoli images of different dimensions; for aerial images, according to the research of the planting market, set the sparse leaves after the control operation as quantitative when studying different planting modes and varieties; set the single ridge and two rows staggered planting as quantitative when studying different varieties and growth of the aerial end; complete broccoli image data collection under different lighting conditions for sparse leaves of short plants and lush leaves of tall plants under different planting modes, and for different varieties of aerial end of single ridge and two rows staggered planting; for ground images, obtain broccoli image data collection under different lighting conditions, as well as image data of different sizes and quality grades of different varieties of the ground end under shading conditions; obtain multi-dimensional image data of the head for training and verification of the YOLOv8-based target detection model, so that the model can meet the detection accuracy of the head in a variety of practical application scenarios.

2. The method for constructing a multidimensional image database according to claim 1, characterized in that: In the step 1, the multi-scale image data includes large-scale surface data obtained by the aerial drone and small-scale point data obtained by the ground cart. The image data obtained by the drone aerial photography contains the position and diameter information of large-area flower balls. Due to the shooting distance of m and the complex lighting and planting conditions, it is difficult to obtain more accurate detailed information of the flower balls, m≥10 meters. Subsequently, the position and diameter information of the flower balls of the entire broccoli field are obtained through two-dimensional reconstruction software; the image information obtained by the ground cart contains the information of several flower balls in a small area of ​​view. Due to the shooting distance of n, 0.5 meters≤n≤1.5 meters, in addition to obtaining the position and size information, the color, shape and grain morphology of the flower balls can also be obtained; the aerial image is divided into 4 dimensions due to different lighting conditions, planting patterns, planting varieties and agronomic operations through field investigation and analysis; the ground image is divided into 2 dimensions due to different lighting conditions and planting varieties through field investigation and analysis.

3. The method for constructing a multidimensional image database according to claim 2, characterized in that: In step 1, the division strategy of the six dimensions is as follows: dimensions 1 to 4 are about the aerial end images, and dimensions 5 to 6 are about the ground end images; Dimension 1, different planting modes. Single ridge and single row means planting one row of broccoli in one ridge, which belongs to the breeding experiment land or mechanized automatic harvesting experiment land; multi-row scattered planting means that multiple rows of broccoli are planted on each large ridge of broccoli, and the middle distance is less than the preset limit, which belongs to the small area managed and planted by individual households; single ridge and two rows staggered means that two rows of broccoli are staggered on each ridge line, which belongs to the large-scale cooperatives adopting this planting mode; under different planting modes, the influence of adjacent broccoli plants on each other may cause different imaging effects of the flower heads in the image. In the single ridge and single row mode, the distance between adjacent broccoli is relatively large, and the mutual influence is small; in the multi-row scattered planting mode, the distance between adjacent broccoli is relatively small, and some leaves of adjacent plants block the flower heads; when single ridge and two rows are staggered, due to the close distance between the plants in the ridge line direction, the plant growth may squeeze and cause the flower heads to tilt, and the leaves in the ridge line direction will be slightly blocked; Dimension 2, different growth, is affected by agronomic operations, soil quality and climate conditions, and focuses on the results shown in the photos. The results are manifested in the different relative growth of leaves and flower heads of plants. This dimension is divided into two conditions: lush leaves and sparse leaves; Dimension 3, different varieties in the aerial end, the characteristics of tall plants are large, tall and compact leaves, and the characteristics of short plants are small, short and loose leaves. In the aerial photos, the characteristic of tall plants is that when the flower ball in the center receives external light, if the light intensity is high and direct, the surface of the flower ball receives sufficient light, which is manifested as high brightness and simple detection; at other times, the brightness is dim and detection is difficult; the characteristic of short plants is that the flower ball in the center is more thoroughly exposed. Compared with tall plants, it is easier to receive external light when the sun's altitude angle is low, and it can be clearly displayed even when the light intensity is low; Dimension 4, different lighting conditions, including different lighting intensities and solar altitude angles. Under different lighting intensities, cloudy data is selected to illustrate the detection rate of flower balls under high light intensity, while data from early morning and evening are used to illustrate the detection rate of flower balls under different light intensities. Different solar altitude angles will directly affect the relative brightness of flower balls and leaves. When the solar altitude angle is greater than 75 degrees, the sun is directly shining, and the entire flower ball is exposed to the sun. The brightness of the flower ball photographed by the gimbal is high, which is very obvious compared to the leaves. When the solar altitude angle is 15-75 degrees, the sun is oblique, and the leaves will block part of the flower ball. The lower the angle, the larger the blocking area, making some leaves obviously bright, while some flower balls appear to be bright and dark. When the sun altitude angle is less than 15 degrees, the sun is oblique and the light intensity is low, and some flower balls appear to be bright and dark and the overall brightness is dark. Dimension 5: a dataset of broccoli heads under different lighting conditions obtained by the ground end, including three conditions; Condition 1: In the case of sunlight, the broccoli head image is obtained using a ground-end cart and a depth camera. Because the shadows of the leaves and the structure of the ground-end cart produced by the sun may cause the head to appear light and dark in the photo, which increases the difficulty of detection and image segmentation. Condition 2: When there is no sunlight, the light intensity gradually decreases after the sun sets, which will cause the image brightness to decrease, and the detection of flower heads will become difficult. Studying the detection accuracy of flower heads under different light intensities is crucial to guide when to add artificial light sources so that the harvesting robot can perform harvesting operations normally at night; Condition 3: Sunshades and shading strips are laid around and on the top of the data collection cart frame to create a closed environment for data collection space, which can effectively isolate the influence of external light. Light tubes with continuously adjustable light intensity are installed around the top frame to serve as a stable light source to quantitatively obtain images of broccoli heads under different light intensities; The data set obtained by condition 3 is to study the effect of different light intensities on the accuracy of broccoli head detection under the conditions of shading and artificial lighting. The minimum light intensity is selected to save energy while ensuring a certain accuracy of head detection. At the same time, the comparison between condition 3 and condition 1 can show whether shading and lighting are needed. The comparison between condition 3 and condition 2 can show how much the ambient light intensity is reduced and how much lighting intensity can meet the nighttime work of the harvesting robot. Dimension 6, the ground end obtains a data set of broccoli heads of different varieties on the ground end, so that the target detection algorithm can complete the transfer learning between different varieties on the ground end, to verify the adaptability and robustness of the algorithm to different varieties on the ground end, and optimize the algorithm to improve the generalization ability of the algorithm model; for different varieties on the ground end, first complete the size classification, that is, the diameter of the head is less than 12cm, the diameter is between 12cm and 14cm, and the diameter is greater than 14cm, which is one of the standards for broccoli harvesting operations; size classification is based on target detection, image segmentation, circle fitting algorithm and depth information , it is necessary to collect visible light images and depth images of the flower balls. On the basis of size classification, it is necessary to complete quality classification for flower balls with a diameter greater than or equal to 12 cm, including extracting the color and morphological details of the flower balls and measuring the roundness of the flower ball contour to determine the quality of the flower balls, corresponding to different harvesting operations; after being picked by the robot, the superior and qualified products should be placed in the superior box and the qualified box respectively, while the unqualified flower balls can be thrown into the field after being picked. The flower balls with a diameter less than 12 cm will not be harvested for the time being. Therefore, it is necessary to complete data collection for broccoli of different varieties and qualities on the ground.

4. A method for constructing a multidimensional image database as claimed in any one of claims 1 to 3, characterized in that: In step 2, the device for collecting multi-dimensional image data of broccoli heads is composed of an aerial drone and a ground cart; The aerial drone includes a high-precision RTK equipped with drones, which is used for spatial positioning of the drone; the drone body includes the drone flight control, frame, rotor and motor, which is the carrier for performing flight missions; the gimbal camera is used to obtain high-throughput remote sensing images of the broccoli field; The ground-side cart includes a three-dimensional frame structure, with four adjustable LED lights installed on the top of the cart so that the light is directed to the sides of the ground. The four adjustable LED lights are connected in parallel and installed in symmetrical positions. The control board Jetson Xavier NX is used to adjust the duty cycle of the PWM signal to control the intensity of the LED light; the universal wheels of the cart can adapt to the movement of broccoli in the field; The portable lithium battery can power the entire ground-side data acquisition system. The control board Jetson Xavier NX is the controller of the ground-side data acquisition system, controlling the depth camera, several light intensity sensors and RTK to collect and store data; the keyboard and mouse are used to detect and operate the program; A small display is used to visualize the acquired light intensity and image data during data acquisition to check the usability of the data; The network version of RTK is used to measure the spatial geographic location of the ground cart. The spatial location of the flower head in the image can be solved through the RGB image and depth image information obtained by the depth camera, as well as the installation position between the two. This facilitates the subsequent mutual verification when the ground cart and the aerial drone obtain the data of broccoli in the same plot, and is used to evaluate the accuracy of the target detection, image segmentation and contour fitting algorithm model for the flower head.

5. The method for constructing a multidimensional image database according to claim 4, characterized in that: In the step 2, after shading, the internal light intensity sensor is used to measure the ambient light intensity inside the cart, and the external light intensity sensor is used to measure the natural ambient light intensity outside the cart; the central light intensity sensor is used for calibration operation after data collection is completed, and the value of the internal light intensity sensor is calibrated to the central light intensity sensor; the horizontal central profile frame is used to install the central light intensity sensor at the intersection of the vertical line of the center of the framework and the horizontal plane of the average height of the flower ball during the calibration process to complete the light intensity calibration operation; the central light intensity sensor and the central profile frame are only used during light intensity calibration, and need to be removed when collecting flower ball image data on the ground side, otherwise they will block the line of sight of the depth camera and may scratch the broccoli during the movement of the cart; a depth camera is installed on each side of the top of the cart, and the lens is facing down to obtain RGB images and depth information by overhead shooting. The dual depth camera can obtain images of the same broccoli at different angles, effectively improving the rate of obtaining broccoli images and depth data. At the same time, the image data of the same broccoli flower ball obtained by the dual depth camera from different angles can verify the accuracy of the algorithm later; control board Jetson XavierNX, internal light intensity sensor and adjustable LED light constitute a closed-loop light intensity control system for shading and artificial lighting. After setting different target light intensity values, the control board Jetson Xavier NX controls the brightness of the LED light through the built-in Python script. The internal light intensity sensor monitors the ambient light intensity in real time and feeds the data back to Jetson Xavier NX. The control board compares the current light intensity with the target light intensity and adjusts the brightness of the LED light according to the difference until the ambient light intensity approaches the target value. The ambient light intensity in the cart frame is automatically and quantitatively controlled during shading and artificial lighting. The shading plate is installed on the top of the frame and the upper part of the frame to block the natural light coming in from the top and upper part; the shading cloth strip is located at the lower part of the frame, which can adapt to various environments when the cart is moving in the field and block the natural light coming in from the bottom.

6. The method for constructing a multidimensional image database according to claim 3, characterized in that: In step 3, based on the divided broccoli curd multi-dimensional image data and the constructed air-ground end data acquisition system, a method for collecting curd image data is formulated for broccoli images of different dimensions; For the dimensions of the aerial image, based on the research of the planting market, the planting pattern of two rows staggered in a single ridge and the situation of sparse leaves after the growth control operation are set as quantitative; Dimension 1 image data acquisition method, using a drone equipped with a gimbal camera and RTK to complete aerial scanning of broccoli plots with different planting patterns of short plants with sparse leaves and tall plants with lush leaves, and obtain image data under different lighting conditions; The method for acquiring dimension 2 image data is to use a drone equipped with a gimbal camera and RTK to scan and photograph broccoli plots with different growth conditions, i.e., lush leaves and sparse leaves, in a single ridge and two rows of staggered planting to obtain image data under different lighting conditions. The 3D image data acquisition method uses a drone equipped with a gimbal camera and RTK to scan and photograph a single ridge with two rows of broccoli of different varieties planted in an aerial position, and obtains the number of images under different lighting conditions, which are used to train and verify the generalization ability of the target detection model between different varieties. The 4-dimensional image data acquisition method uses a drone equipped with a gimbal camera and RTK to complete the acquisition of image data under different lighting conditions under the aforementioned conditions; Dimension 5 data acquisition method, using the ground-end cart to obtain broccoli head data sets under different lighting conditions, and obtain pictures containing broccoli head details at a distance of nearly 1 meter; Dimension 5 includes 3 different lighting conditions. Condition 1 is the acquisition of the broccoli head data set under natural sunlight conditions. In sunny weather, the ground-end cart is modified into a size suitable for moving in the target broccoli field, and the sunshade and sunshade cloth strips are removed to obtain data under different natural lighting conditions; Condition 2: Acquisition of broccoli head dataset under different natural light intensity conditions. In order to isolate the influence of the solar altitude angle, we selected the time periods of cloudy days, before sunrise and after sunset, modified the ground-end cart into a size suitable for moving in the target broccoli field, removed the shading board and shading cloth strips, and installed an external light intensity sensor to measure the external ambient light intensity. Condition 3: Acquisition of broccoli head dataset under shading and adding artificial light conditions. We modified the ground-end cart into a size suitable for moving in the target broccoli field, installed shading boards and shading cloth strips, installed internal light intensity sensors and adjustable LED lights to measure the changes in ambient light intensity in the interior space of the cart after shading and artificial lighting, and used the control board Jetson Xavier The light intensity control closed-loop system composed of NX, internal light intensity sensor and adjustable LED light controls the change of ambient light intensity in the interior space of the cart to obtain image data sets at different light intensities under the conditions of shading and using artificial light sources, which are used for subsequent training and verification of the target detection algorithm under the conditions of shading and using artificial light sources, and are compared with conditions 1 and 2 respectively to illustrate the importance of shading and artificial lighting; under the conditions of shading and using artificial light sources, the image data sets at different light intensities can be used to verify the accuracy of the target detection algorithm model in flower ball detection under different light intensities; Dimension 6 data acquisition method, using the ground-end cart to obtain different sizes and quality grading data of different varieties on the ground end. The cart needs to be shielded from light and manually illuminated. First, the image data of the flower heads of the same variety of broccoli is acquired for size and quality grading. The size grading focuses on the diameter size of the broccoli flower heads, which are divided into three grades: less than 12 cm, 12-14 cm, and greater than 14 cm. The visible light image and depth image data of the flower heads of the three size grades and the corresponding real measurement data of the flower head diameter obtained by calipers are obtained for subsequent verification of flower head detection, image segmentation, and wheel The accuracy of the contour fitting algorithm model; the quality grading is based on the coloring and morphology of the broccoli florets. High-quality broccoli florets are dark green with uniform color, firm, and uniform-sized buds without yellowing, blackening, or flowering. Low-quality broccoli florets have one or more of the following manifestations: uneven color of the florets, abnormal blackening or purple color in some parts of the florets, uneven floret sizes, scattered flowers, yellowing florets, or small flowers. The identification of broccoli quality requires the invitation of broccoli quality identification experts to conduct on-site identification in order to standardize the standards for subsequent target detection model algorithm training and verification set production.

7. The method for constructing a multidimensional image database according to claim 6, characterized in that: In the step 3, the method for acquiring image data of dimension 4 is divided into two conditions: different light intensity and different solar altitude angle. There are three acquisition times under different light intensities, namely, 1 is cloudy; 2 is early morning, before the sun rises; 3 is evening, after the sun sets. These three time periods are selected to exclude the influence of the solar altitude angle and only consider the influence of the brightness of the natural environment. When acquiring aerial images of the broccoli field, the control board Jetson Xavier NX is used to connect the light intensity sensor BH1750 and set it to continuous H resolution mode. The data of natural light intensity during the shooting process and the corresponding Beijing time are recorded throughout the process. Subsequently, the timestamp is used to complete the correspondence between light and image data, so as to complete the grouping of images in different light intensity intervals and quantitatively study the influence of light intensity on the accuracy of flower head detection.

8. The method for constructing a multidimensional image database according to claim 6, characterized in that: In the step 3, in the dimension 5 data acquisition method, after the data collection is completed, in order to obtain the light intensity of the flower ball average growth height plane with more reference value, it is necessary to add a light intensity calibration process. This calibration process is to keep the position of the internal light sensor unchanged, install the central profile for installing the central light intensity sensor at the point where the central vertical line of the frame intersects the flower ball growth plane, obtain and record the values ​​of the internal light intensity sensor and the central light intensity sensor by changing the brightness of the adjustable LED lamp, and then fit the data relationship between the two light intensity sensors to obtain the light intensity value of the internal light intensity sensor corresponding to the light intensity value of the central flower ball surface.

9. A construction device for implementing the construction method of the multidimensional image database as claimed in claim 1, characterized in that: The construction device includes a partitioning strategy module, a collection device selection and design module, and a broccoli head multi-dimensional image data construction module, and each module sequentially executes the processing of steps 1 to 3 of the construction method.