Crop phenotype data fusion method and device, electronic equipment and storage medium

Abstract

The invention provides a crop phenotype data fusion method and device, electronic equipment and a storage medium, and relates to the technical field of electrical digital data processing, and the method comprises the steps: inputting target data corresponding to each piece of three-dimensional first to-be-fused crop phenotype data in each piece of to-be-fused crop phenotype data into a weight adjustment model, and obtaining a weight value corresponding to each piece of first to-be-fused crop phenotype data output by the weight adjustment model, and performing data fusion on each piece of to-be-fused crop phenotype data based on the weight value corresponding to each piece of first to-be-fused crop phenotype data and internal and external parameter data of a crop phenotype sensor for collecting each piece of to-be-fused crop phenotype data, the target data comprises environmental data when the crop phenotype data is collected, relative position information between the crop phenotype sensor and the crop, and a target parameter value of the crop phenotype data. According to the method, the dynamic weight is given to the crop phenotype data to be fused, so that the fusion effect, relevance, fusion efficiency and timeliness of data fusion can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a crop phenotype data fusion method, device, electronic equipment and storage medium. Background Art

[0002] Plant phenotype refers to the externally observable form and characteristics of individual plants, including the size, shape, structure, flowers, leaves, stems, roots, etc. Plant phenotype is the result of the interaction between the plant genome and the environment, and can reflect the plant's adaptation to the environment and its growth and development.

[0003] Using a crop phenotyping platform equipped with a variety of phenotyping sensors, crop phenotyping data of different dimensions can be obtained. Common phenotyping sensors mainly include lidar sensors, multispectral image sensors, visible light image sensors, infrared thermal imaging sensors, and depth image sensors.

[0004] Data fusion of crop phenotypic data of different dimensions is of great significance in improving data accuracy and completeness, reducing data redundancy and uncertainty, enhancing decision support capabilities, promoting agricultural informatization and intelligence, and promoting multidisciplinary cross-cutting and integration. However, when traditional crop phenotypic data fusion methods in related technologies are used to fuse crop phenotypic data of different dimensions and / or different periods, there are problems such as low fusion efficiency and poor fusion effect.

[0005] Therefore, how to improve the fusion efficiency and fusion effect of crop phenotypic data fusion is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0006] The present invention provides a crop phenotypic data fusion method, device, electronic device and storage medium, which are used to solve the defects of low fusion efficiency and poor fusion effect when fusing crop phenotypic data of different dimensions in the prior art, and to improve the fusion efficiency and fusion effect of crop phenotypic data fusion.

[0007] The present invention provides a crop phenotype data fusion method, comprising the following steps.

[0008] Acquire each crop phenotypic data to be fused, wherein each crop phenotypic data to be fused includes crop phenotypic data of a target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; Acquire internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, wherein the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes environmental data when collecting crop phenotypic data, relative position information between the crop phenotypic sensor collecting crop phenotypic data and the crop, and target parameter value of the crop phenotypic data, and the target parameter includes signal-to-noise ratio and / or characteristic entropy; Inputting target data corresponding to each of the first crop phenotypic data to be fused into a weight adjustment model, and obtaining a weight value corresponding to each of the first crop phenotypic data to be fused output by the weight adjustment model, wherein the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each of the sample crop phenotypic data, and the sample crop phenotypic data includes crop phenotypic data of sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of the sample crops at multiple growth stages; Based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, data fusion is performed on each of the crop phenotypic data to be fused to obtain fused data corresponding to each of the crop phenotypic data to be fused.

[0009] According to a crop phenotypic data fusion method provided by the present invention, the weight values ​​corresponding to the sample crop phenotypic data are obtained based on the following steps: Based on the target parameter value of the sample crop phenotypic data, a first weight score corresponding to the sample crop phenotypic data is obtained; based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop, a second weight score corresponding to the sample crop phenotypic data is obtained; based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a third weight score corresponding to the sample crop phenotypic data is obtained; based on the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a fourth weight score corresponding to the sample crop phenotypic data is obtained; Based on the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data, the weight value corresponding to the sample crop phenotypic data is calculated.

[0010] According to a crop phenotypic data fusion method provided by the present invention, based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, data fusion is performed on each of the crop phenotypic data to be fused to obtain fused data corresponding to each of the crop phenotypic data to be fused, including: Performing denoising processing on each of the crop phenotypic data to be fused, and obtaining each of the crop phenotypic data to be fused after denoising processing; Based on the internal and external reference data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, initially aligning the denoised crop phenotypic data to be fused to obtain the crop phenotypic data to be fused after the initial alignment; Based on the weight value corresponding to each of the first crop phenotypic data to be fused, data fusion is performed on each of the first crop phenotypic data to be fused after initial alignment to obtain first fused data; Each second crop phenotypic data to be fused after initial alignment is fused with the first fused data to obtain the fused data, wherein the second crop phenotypic data to be fused is the crop phenotypic data in each crop phenotypic data to be fused except the first crop phenotypic data to be fused.

[0011] According to a crop phenotypic data fusion method provided by the present invention, obtaining a first weight score corresponding to the sample crop phenotypic data based on a target parameter value of the sample crop phenotypic data includes: Determine the target parameter value of the sample crop phenotypic data as an independent variable, determine the first weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data and a positive correlation function, wherein the positive correlation function is used to describe the positive correlation between the independent variable and the dependent variable; The step of obtaining a second weight score corresponding to the sample crop phenotypic data based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop comprises: Based on the relative position information between the sample crop phenotype sensor and the sample crop, the distance between the sample crop phenotype sensor and the sample crop and the vertical distance between the sample crop and a target reference line are acquired, where the target reference line is the central axis of the field of view of the sample crop phenotype sensor; Determine the distance as an independent variable, determine the first sub-weight score corresponding to the sample crop phenotypic data as a dependent variable, calculate the first sub-weight score corresponding to the sample crop phenotypic data based on the distance and a positive correlation function, determine the vertical distance as an independent variable, determine the second sub-weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the second sub-weight score corresponding to the sample crop phenotypic data based on the distance and a negative correlation function; Based on the first sub-weight score and the second sub-weight score corresponding to the sample crop phenotypic data, a second weight score corresponding to the sample crop phenotypic data is calculated; The obtaining, based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a third weight score corresponding to the sample crop phenotypic data comprises: Acquire a first matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor; Determine the first matching degree as an independent variable, determine the third weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the third weight score corresponding to the sample crop phenotypic data based on the first matching degree and a positive correlation function; The acquiring, based on the growth stage of the sample crop and the type of the sample crop phenotypic sensor when the sample crop phenotypic data is collected, a fourth weight score corresponding to the sample crop phenotypic data, comprises: Acquiring a second matching degree between the growth stage of the sample crop and the type of the sample crop phenotypic sensor when collecting the sample crop phenotypic data; The second matching degree is determined as an independent variable, and the fourth weight score corresponding to the sample crop phenotypic data is determined as a dependent variable. Based on the second matching degree and a positive correlation function, the fourth weight score corresponding to the sample crop phenotypic data is calculated.

[0012] According to a crop phenotypic data fusion method provided by the present invention, the weight value corresponding to the sample crop phenotypic data is calculated based on the first weight score, the second weight score, the third weight score and the fourth weight score of the sample crop phenotypic data, including: Obtaining a temporal consistency evaluation value of the sample crop phenotypic data; Calculating the product of the first weight score, the second weight score, the third weight score and the fourth weight score of the sample crop phenotypic data as an intermediate result; The quotient of the intermediate result and the time consistency evaluation value of the sample crop phenotypic data is calculated as the weight value corresponding to the sample crop phenotypic data.

[0013] According to a crop phenotypic data fusion method provided by the present invention, based on the weight value corresponding to each of the first crop phenotypic data to be fused, the data of each of the first crop phenotypic data to be fused after initial alignment is fused to obtain first fused data, including: Based on the weight value corresponding to each of the first crop phenotypic data to be fused, using an iterative closest point algorithm to fuse the first crop phenotypic data to be fused after initial alignment, to obtain the first fused data; The step of fusing each second crop phenotypic data to be fused after the initial alignment with the first fused data to obtain fused data corresponding to each crop phenotypic data to be fused, wherein the second crop phenotypic data to be fused is the crop phenotypic data in each crop phenotypic data to be fused except the first crop phenotypic data to be fused, including: Performing feature extraction on each of the second crop phenotypic data to be fused after the initial alignment, and acquiring feature information corresponding to each of the second crop phenotypic data to be fused after the initial alignment; The characteristic information corresponding to each of the second crop phenotypic data to be fused after initial alignment is mapped into the first fused data to obtain the fused data.

[0014] The present invention also provides a crop phenotypic data fusion device, comprising the following modules: A first data acquisition module is used to acquire phenotypic data of each crop to be fused, wherein each of the phenotypic data of the crop to be fused includes crop phenotypic data of the target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; A second data acquisition module is used to acquire internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, wherein the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes environmental data when collecting crop phenotypic data, relative position information between the crop phenotypic sensor collecting crop phenotypic data and the crop, and target parameter values ​​of the crop phenotypic data, and the target parameters include signal-to-noise ratio and / or characteristic entropy; a dynamic weight determination module, for inputting target data corresponding to each of the first crop phenotypic data to be fused into a weight adjustment model, and obtaining a weight value corresponding to each of the first crop phenotypic data to be fused output by the weight adjustment model, wherein the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each of the sample crop phenotypic data, and the sample crop phenotypic data include crop phenotypic data of sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of the sample crops at multiple growth stages; A multi-source data fusion module is used to fuse the crop phenotypic data to be fused based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, so as to obtain the fused data corresponding to each of the crop phenotypic data to be fused.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the crop phenotypic data fusion method as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the crop phenotypic data fusion method as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the crop phenotypic data fusion method as described above is implemented.

[0018] The crop phenotypic data fusion method, device, electronic device and storage medium provided by the present invention input the target data corresponding to each first crop phenotypic data to be fused into a weight adjustment model, obtain the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model, then perform data fusion on each crop phenotypic data to be fused based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, and obtain the fused data corresponding to each crop phenotypic data to be fused, which can comprehensively consider the data quality of the sample crop phenotypic data, the sample crop phenotypic sensor The relative position relationship between the sensor and the sample crop when collecting sample crop phenotypic data, the influence of environmental factors when the sample crop phenotypic sensor collects sample crop phenotypic data, and the growth stage of the sample crop on the sample crop phenotypic data can be more accurately obtained, so as to more efficiently and accurately obtain the weight value corresponding to each crop phenotypic data to be fused through deep learning, which can improve the fusion efficiency and fusion effect of crop phenotypic data fusion, provide a more accurate data basis for high-precision three-dimensional modeling, crop feature extraction and other application scenarios of precision agriculture, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of the process of the crop phenotype data fusion method provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of the crop phenotype data fusion device provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0025] In the description of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, in the description of the present application, "and / or" represents at least one of the connected objects, and the character " / " generally represents that the front and back associated objects are in an "or" relationship.

[0026] It should be noted that when performing data fusion on crop phenotypic data of different dimensions and different periods based on the traditional crop phenotypic data fusion method in the relevant technology, fixed weights are usually assigned to the crop phenotypic data of different dimensions, and then the crop phenotypic data of different dimensions are fused based on the weight of the crop phenotypic data of each dimension.

[0027] However, when fusing crop performance data of different dimensions and / or different periods, the above-mentioned traditional phenotypic data fusion methods ignore the impact of factors such as crop type, crop growth stage, environmental changes and data source quality on data fusion, resulting in poor fusion effect of data fusion of crop phenotypic data of different dimensions based on traditional crop phenotypic data fusion methods in related technologies, and unable to fully utilize the advantages of multi-dimensional crop phenotypic data.

[0028] The above-mentioned traditional phenotypic data fusion method has a long processing flow for data fusion of crop phenotypic data of different dimensions and / or different periods, resulting in low fusion efficiency of data fusion of crop phenotypic data of different dimensions based on the above-mentioned traditional crop phenotypic data fusion method, making the above-mentioned traditional crop phenotypic data fusion method unable to meet the needs of high-timeliness data analysis, limiting the application of the above-mentioned traditional crop phenotypic data fusion method in scenarios requiring rapid response and real-time decision-making.

[0029] When the above-mentioned traditional phenotypic data fusion methods fuse crop phenotypic data of different dimensions and / or different periods, they lack intelligent adjustment strategies to explore the potential correlation and complementarity between crop phenotypic data of different dimensions and / or different periods, resulting in that the fused crop phenotypic data are difficult to achieve effective feature enhancement and precise crop characterization, which in turn affects the application effect of the above-mentioned traditional crop phenotypic data fusion methods in the fields of precision agriculture and environmental monitoring.

[0030] The above-mentioned traditional phenotypic data fusion methods are also difficult to cope with dynamic changes in crop growth environment, sensor status, etc., and have poor adaptability, which leads to the above-mentioned traditional phenotypic data fusion methods being flexibly adjusted according to actual conditions, limiting the application of the above-mentioned traditional crop phenotypic data fusion methods in complex agricultural environments and other dynamically changing scenarios.

[0031] In this regard, the present invention provides a crop phenotypic data fusion method, which analyzes the quality and characteristics of different data sources and dynamically adjusts their weights in data fusion, thereby improving the relevance, utilization efficiency and timeliness of data fusion, solving the problems of fixed weights, low timeliness and low data utilization in related technologies, and providing a more flexible, efficient and intelligent multi-source data fusion solution.

[0032] Combine the following Figure 1The crop phenotypic data fusion method of the present invention is described.

[0033] Figure 1 FIG. 1 is a flow chart of the crop phenotype data fusion method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101, obtaining each crop phenotypic data to be fused, wherein each crop phenotypic data to be fused includes crop phenotypic data of the target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages.

[0034] It should be noted that the executor of the embodiment of the present invention is a crop phenotype data fusion device. The crop phenotype data fusion device can be configured in electronic devices such as computers and servers.

[0035] Specifically, each crop phenotypic data to be fused is a fusion object of the crop phenotypic data fusion method provided by the present invention.

[0036] It should be noted that each crop phenotypic data to be fused includes crop phenotypic data of the target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages.

[0037] It can be understood that each crop phenotypic data to be fused may include crop phenotypic data of the target crop at different growth stages collected by the same crop phenotypic sensor; each crop phenotypic data to be fused may also include crop phenotypic data of the target crop at the same growth stage collected by different crop phenotypic sensors.

[0038] It is understandable that the target crop in the embodiment of the present invention can be determined based on actual needs. The target crop in the embodiment of the present invention is not specifically limited.

[0039] It should be noted that different types of crop phenotyping sensors in the embodiments of the present invention may include but are not limited to lidar sensors, multispectral image sensors, visible light image sensors, infrared thermal imaging sensors, depth image sensors, etc.

[0040] It can be understood that the crop phenotyping data collected by crop phenotyping sensors such as lidar sensors and depth image sensors are three-dimensional, and the crop phenotyping data collected by crop phenotyping sensors such as multispectral image sensors, visible light image sensors and infrared thermal imaging sensors are two-dimensional.

[0041] In the embodiment of the present invention, the phenotypic data of each crop to be fused can be obtained in a variety of ways. For example, in the embodiment of the present invention, the phenotypic data of each crop to be fused can be obtained based on the input of the user; or, in the embodiment of the present invention, the phenotypic data of each crop to be fused can be received from other electronic devices. In the embodiment of the present invention, the specific method of obtaining the phenotypic data of each crop to be fused is not limited.

[0042] Step 102: obtaining the internal and external reference data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and the target data corresponding to the first crop phenotypic data to be fused, wherein the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes the environmental data when collecting the crop phenotypic data, the relative position information between the crop phenotypic sensor collecting the crop phenotypic data and the crop, and the target parameter value of the crop phenotypic data, and the target parameter includes the signal-to-noise ratio and / or characteristic entropy.

[0043] It should be noted that, in the embodiments of the present invention, when the crop phenotyping sensors collect crop phenotyping data, they will record the internal and external parameter data of the crop phenotyping sensors when collecting data. Therefore, in the embodiments of the present invention, the internal and external parameter data of each crop phenotyping sensor when collecting each crop phenotyping data to be fused can be obtained by means of data query.

[0044] It should be noted that, in the embodiment of the present invention, the three-dimensional crop phenotypic data to be fused collected by crop phenotypic sensors such as lidar sensors and depth image sensors in each crop phenotypic data to be fused is determined as the first crop phenotypic data to be fused; the two-dimensional crop phenotypic data to be fused collected by crop phenotypic sensors such as multispectral image sensors, visible light image sensors and infrared thermal imaging sensors in each crop phenotypic data to be fused is determined as the second crop phenotypic data to be fused.

[0045] It should be noted that each crop phenotypic data to be fused in the embodiment of the present invention only includes one or more first crop phenotypic data to be fused and / or one or more second crop phenotypic data to be fused.

[0046] It should be noted that, in the embodiments of the present invention, when the crop phenotyping sensors collect crop phenotyping data, they all use environmental perception sensors to collect environmental data when the crop phenotyping sensors collect crop phenotyping data. Therefore, in the embodiments of the present invention, the environmental data of each crop phenotyping sensor when collecting each crop phenotyping data to be fused can be obtained by means of data query, as the target data corresponding to each crop phenotyping data to be fused.

[0047] After obtaining the internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused, the relative position information between each crop phenotypic sensor and the target crop when collecting each crop phenotypic data to be fused can be calculated based on the internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused, as the target data corresponding to each crop phenotypic data to be fused.

[0048] After obtaining each crop phenotypic data to be fused, the signal-to-noise ratio and / or characteristic entropy of each crop phenotypic data to be fused can be calculated by numerical calculation as the target data corresponding to each crop phenotypic data to be fused.

[0049] Step 103: input the target data corresponding to each first crop phenotypic data to be fused into a weight adjustment model, and obtain the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model. The weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each sample crop phenotypic data. The sample crop phenotypic data includes the crop phenotypic data of the sample crops collected by different types of crop phenotypic sensors and the crop phenotypic data of the sample crops at multiple growth stages.

[0050] It should be noted that in the embodiments of the present invention, the sample data set can be obtained in a variety of ways, for example, based on user input, the sample data set can be obtained; or, sample data sent by other electronic devices can be received. The specific way of obtaining the sample data set is not limited in the embodiments of the present invention.

[0051] Optionally, in an embodiment of the present invention, a crop phenotypic data group with better fusion effect after data fusion can be selected from the crop phenotypic data groups that have been subjected to data fusion, and then each crop phenotypic data in the crop phenotypic data group with better fusion effect after data fusion can be determined as each sample crop phenotypic data, and then each sample crop phenotypic data and the weight value corresponding to each sample crop phenotypic data can be determined as a sample data group.

[0052] After obtaining the sample data group, the target data corresponding to each sample crop phenotypic data can be used as a training sample, and the weight value corresponding to each sample crop phenotypic data can be used as a sample label to train the weight adjustment model to obtain a trained weight adjustment model.

[0053] It can be understood that there are multiple sample data groups in the embodiment of the present invention.

[0054] It can be understood that the sample crop table data in the embodiment of the present invention is three-dimensional.

[0055] After obtaining the trained weight adjustment model, the target data corresponding to each first crop phenotypic data to be fused is input into the trained weight adjustment model, and the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model can be obtained.

[0056] It should be noted that in order to further improve the adaptability of the weight adjustment model, a self-learning mechanism is introduced in the embodiment of the present invention. By retrospectively analyzing the historical collected data, the weight adjustment model can automatically optimize its weight adjustment strategy and fusion algorithm. The self-learning mechanism includes the following core modules: The historical data retrospective module analyzes the results of historical multi-source data, extracts scene features with better fusion effects, and feeds them back to the weight adjustment model for model optimization.

[0057] After each new data collection is completed, the weight adjustment model will perform incremental learning based on the latest data and continuously adjust the fusion parameters so that the model can still handle new sensor configurations or environmental changes efficiently.

[0058] Through this self-learning mechanism, the crop phenotypic data fusion method provided by the present invention can continuously optimize and improve the data fusion effect when facing complex scenes, dynamically changing sensor configurations or environmental conditions.

[0059] As an optional embodiment, the weight value corresponding to the sample crop phenotypic data is obtained based on the following steps: based on the target parameter value of the sample crop phenotypic data, obtaining a first weight score corresponding to the sample crop phenotypic data; based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop, obtaining a second weight score corresponding to the sample crop phenotypic data; based on the environmental data when the sample crop phenotypic data was collected and the type of the sample crop phenotypic sensor, obtaining a third weight score corresponding to the sample crop phenotypic data; based on the growth stage of the sample crop when the sample crop phenotypic data was collected and the type of the sample crop phenotypic sensor, obtaining a fourth weight score corresponding to the sample crop phenotypic data.

[0060] Specifically, in an embodiment of the present invention, numerical calculation, mathematical statistics, deep learning and the like can be used to obtain a first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data; obtain a second weight score corresponding to the sample crop phenotypic data based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop; obtain a third weight score corresponding to the sample crop phenotypic data based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor; and obtain a fourth weight score corresponding to the sample crop phenotypic data based on the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor.

[0061] As an optional embodiment, based on the target parameter value of the sample crop phenotypic data, obtaining a first weight score corresponding to the sample crop phenotypic data includes: determining the target parameter value of the sample crop phenotypic data as an independent variable, determining the first weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculating the first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data and a positive correlation function, wherein the positive correlation function is used to describe the positive correlation between the independent variable and the dependent variable.

[0062] It should be noted that a positive correlation means that if the independent variable changes from large to small, the dependent variable also changes from large to small, and if the independent variable changes from small to large, the dependent variable also changes from small to large. The positive correlation function in the embodiment of the present invention may include but is not limited to a linear positive correlation function, a quadratic function, an exponential function, and a logarithmic function. The positive correlation function in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions. The positive correlation function is not specifically limited in the embodiment of the present invention.

[0063] It can be understood that the signal-to-noise ratio and characteristic entropy are important indicators for measuring data quality. The higher the signal-to-noise ratio of the data, the higher the quality of the data; the higher the characteristic entropy of the data, the higher the information contained in the data.

[0064] Therefore, in the embodiment of the present invention, the target parameter value of the sample crop phenotypic data is determined as the independent variable, and the first weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Then, based on the target parameter value of the sample crop phenotypic data and the positive correlation function, the first weight score corresponding to the sample crop phenotypic data can be calculated to obtain the first weight score corresponding to the sample crop phenotypic data that has a positive correlation with the target parameter value of the sample crop phenotypic data.

[0065] Based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop, a second weight score corresponding to the sample crop phenotypic data is obtained, including: based on the relative position information between the sample crop phenotypic sensor and the sample crop, a distance between the sample crop phenotypic sensor and the sample crop and a vertical distance between the sample crop and a target reference line are obtained, where the target reference line is a central axis of the field of view of the sample crop phenotypic sensor.

[0066] The distance is determined as the independent variable, and the first sub-weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Based on the distance and the positive correlation function, the first sub-weight score corresponding to the sample crop phenotypic data is calculated. The vertical distance is determined as the independent variable, and the second sub-weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Based on the distance and the negative correlation function, the second sub-weight score corresponding to the sample crop phenotypic data is calculated.

[0067] Based on the first sub-weight score and the second sub-weight score corresponding to the sample crop phenotypic data, a second weight score corresponding to the sample crop phenotypic data is calculated.

[0068] It is understandable that when using crop phenotyping sensors to collect crop phenotyping data, the crop phenotyping sensors are usually set at different locations. For example, visible light image sensors can be set on drones to obtain wide-area views, while lidar sensors are usually set on the ground to obtain detailed point cloud dredging. Therefore, the relative position relationship between the crop phenotyping sensor and the crop needs to be considered in the data fusion process.

[0069] For three-dimensional sample crop phenotypic data, the closer the distance between the sample crop phenotypic sensor and the sample crop, and the closer the sample crop is to the center of the field of view of the sample crop phenotypic sensor, the more crop features the acquired sample crop phenotypic data contains.

[0070] Therefore, in the embodiment of the present invention, the distance between the sample crop phenotypic sensor and the sample crop is determined as the independent variable, and the first sub-weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Then, based on the distance between the sample crop phenotypic sensor and the sample crop and the positive correlation function, the first sub-weight score corresponding to the sample crop phenotypic data can be calculated to obtain the first sub-weight score corresponding to the sample crop phenotypic data that has a positive correlation with the distance between the sample crop phenotypic sensor and the sample crop.

[0071] It should be noted that in the embodiment of the present invention, the vertical distance between the sample crop and the central axis of the field of view of the sample crop phenotyping sensor is used to describe the position of the sample crop in the field of view of the sample crop phenotyping sensor. The smaller the vertical distance between the sample crop and the central axis of the field of view of the sample crop phenotyping sensor, the closer the crop is to the center of the field of view of the crop phenotyping sensor.

[0072] Accordingly, in the embodiment of the present invention, the vertical distance between the sample crop and the central axis of the field of view of the sample crop phenotyping sensor is determined as the independent variable, and the second sub-weight score corresponding to the sample crop phenotyping data is determined as the dependent variable. Then, based on the vertical distance between the sample crop and the central axis of the field of view of the sample crop phenotyping sensor and the negative correlation function, the second sub-weight score corresponding to the sample crop phenotyping data can be calculated to obtain the second sub-weight score corresponding to the sample crop phenotyping data that has a negative correlation with the vertical distance between the sample crop and the central axis of the field of view of the sample crop phenotyping sensor.

[0073] It should be noted that a negative correlation means that if the independent variable changes from large to small, the dependent variable changes from small to large, and if the independent variable changes from small to large, the dependent variable changes from large to small. The negative correlation function in the embodiment of the present invention may include but is not limited to a linear negative correlation function, a quadratic function, an exponential function, and a logarithmic function. The positive correlation function in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions. The positive correlation function is not specifically limited in the embodiment of the present invention.

[0074] After obtaining the first sub-weight score and the second sub-weight score corresponding to the sample crop phenotypic data, the average of the first sub-weight score and the second sub-weight score corresponding to the sample crop phenotypic data can be calculated by numerical calculation as the second weight score corresponding to the sample crop phenotypic data.

[0075] Based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a third weight score corresponding to the sample crop phenotypic data is obtained, including: obtaining a first matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor.

[0076] The first matching degree is determined as the independent variable, and the third weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Based on the first matching degree and the positive correlation function, the third weight score corresponding to the sample crop phenotypic data is calculated.

[0077] It is understandable that the confidence of crop phenotyping data collected by different types of crop phenotyping sensors in different environments is not the same. For example, when the ambient temperature is high, the confidence of crop phenotyping data collected by infrared thermal imaging sensors is higher; when the light intensity is high, the confidence of crop phenotyping data collected by visible light image sensors is higher; when the light intensity is low, the confidence of crop phenotyping data collected by visible light image sensors is lower, and the confidence of crop phenotyping data collected by multispectral image sensors and lidar sensors is higher; when the wind speed is high, the confidence of crop phenotyping data collected by various types of crop phenotyping sensors is lower.

[0078] Therefore, in the embodiment of the present invention, the first matching degree between the environmental data when collecting the sample crop phenotypic data and the type of the sample crop phenotypic sensor is determined as the independent variable, and the third weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Then, based on the above-mentioned first matching degree and the positive correlation function, the third weight score corresponding to the sample crop phenotypic data can be calculated to obtain the third weight score corresponding to the sample crop phenotypic data that has a positive correlation with the above-mentioned first matching degree.

[0079] It should be noted that, based on the sensitive environmental data type corresponding to the type of the sample crop phenotypic sensor, the value of the environmental data of the sensitive environmental data type in the environmental data when the sample crop phenotypic data is collected is determined, and then the value of the environmental data of the sensitive environmental data type in the environmental data when the sample crop phenotypic data is collected can be determined to determine the first matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor. The sensitive environmental data type corresponding to the type of the sample crop phenotypic sensor can be predefined based on prior knowledge and / or actual conditions.

[0080] For example, based on prior knowledge and / or it can be determined that the sensitive environmental data type corresponding to the infrared thermal imaging sensor is the ambient temperature, therefore, when the sample crop phenotypic data is collected by the infrared thermal imaging sensor, the first original matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor can be determined based on the temperature range of the ambient temperature when the infrared thermal imaging sensor collects the sample crop phenotypic data, and the second original matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor can be determined based on the wind speed when the infrared thermal imaging sensor collects the sample crop phenotypic data, and then the average of the first original matching degree and the second original matching degree can be determined as the first matching degree between the infrared thermal imaging sensor and the ambient temperature when the infrared thermal imaging sensor collects the sample crop phenotypic data. The corresponding relationship between different temperature ranges and different matching degrees can be determined based on prior knowledge and / or actual conditions.

[0081] For another example, based on prior knowledge and / or it can be determined that the sensitive environmental data type corresponding to the visible light image sensor, spectral image sensor and lidar sensor is light intensity, therefore, when the sample crop phenotypic data is collected by the visible light image sensor, spectral image sensor or lidar sensor, the first original matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor can be determined based on the light intensity interval in which the light intensity when the sample crop phenotypic data is collected, and the second original matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor can be determined based on the wind speed when the sample crop phenotypic data is collected, and then the average of the first original matching degree and the second original matching degree can be determined as the first matching degree between the sample crop phenotypic sensor and the environmental temperature when the sample crop phenotypic sensor collects the sample crop phenotypic data. Among them, the corresponding relationship between different light intensity intervals and different matching degrees can be determined based on prior knowledge and / or actual conditions.

[0082] Based on the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a fourth weight score corresponding to the sample crop phenotypic data is obtained, including: obtaining a second matching degree between the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor.

[0083] The second matching degree is determined as the independent variable, and the fourth weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Based on the second matching degree and the positive correlation function, the fourth weight score corresponding to the sample crop phenotypic data is calculated.

[0084] It should be noted that at different growth stages of crops, the contribution of crop phenotypic data collected by different types of crop phenotypic sensors to the analysis of crop growth status is not the same. For example, in the early growth stage of crops, the color information of leaves is more important for the analysis of the normal state of crops. Therefore, the crop phenotypic data collected by visible light image sensors contributes more to the analysis of crop growth status, while the crop phenotypic data collected by depth image sensors and lidar sensors contribute less to the analysis of crop growth status. In the mid- and late growth stages of crops, the plant height and volume of crops change significantly, and the crop phenotypic data collected by depth image sensors and lidar sensors contribute more to the analysis of crop growth status.

[0085] Therefore, in the embodiment of the present invention, the second matching degree between the growth stage of the sample crop and the type of the sample crop phenotypic sensor when collecting the sample crop phenotypic data is determined as the independent variable, and the fourth weight score corresponding to the sample crop phenotypic data is determined as the dependent variable. Then, based on the above second matching degree and the positive correlation function, the fourth weight score corresponding to the sample crop phenotypic data can be calculated to obtain the fourth weight score corresponding to the sample crop phenotypic data that has a positive correlation with the above second matching degree.

[0086] It should be noted that the second matching degree may be determined based on the sensitive growth stage corresponding to the type of the sample crop phenotypic sensor, wherein the sensitive growth stage corresponding to the type of the sample crop phenotypic sensor may be determined based on prior knowledge and / or actual conditions.

[0087] For example, in the case where the sample crop phenotypic data is collected by a visible light image sensor, if the growth stage of the sample crop when the sample crop phenotypic data is collected is an early growth stage, it can be determined that the second matching degree between the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor is 1; if the growth stage of the sample crop when the sample crop phenotypic data is not an early growth stage, it can be determined that the second matching degree between the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor is 0.

[0088] For example, in the case where the sample crop phenotypic data is collected by a depth image sensor, if the growth stage of the sample crop when the sample crop phenotypic data is collected is a mid-growth stage, it can be determined that the second matching degree between the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor is 1; if the growth stage of the sample crop when the sample crop phenotypic data is collected is an early growth stage, it can be determined that the second matching degree between the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor is 0.

[0089] Based on the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data, the weight value corresponding to the sample crop phenotypic data is calculated.

[0090] Specifically, after obtaining the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data, the weight value corresponding to the sample crop phenotypic data can be calculated by numerical calculation.

[0091] As an optional embodiment, based on the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data, the weight value corresponding to the sample crop phenotypic data is calculated, including: obtaining the time consistency evaluation value of the sample crop phenotypic data.

[0092] Specifically, since the collection times of the phenotypic data of each fusion object are not exactly the same, time asynchrony may occur during data fusion. Therefore, in the embodiment of the present invention, a time consistency evaluation value of the sample crop phenotypic data is introduced during the model training stage to calculate the weight value corresponding to the sample crop phenotypic data.

[0093] It should be noted that the time consistency evaluation value of the sample crop phenotypic data in the embodiment of the present invention can be used as a metric to measure whether the sample crop phenotypic data remains consistent between different time points.

[0094] The temporal consistency evaluation value of the sample crop phenotypic data can be calculated by the following steps: Calculate the time difference between the time when the sample crop phenotypic data is collected and the reference time The reference time may be predefined based on actual conditions.

[0095] Based on the above time difference , the temporal consistency evaluation value of the sample crop phenotypic data is calculated by the following formula:

[0096] in, Indicates The temporal consistency evaluation value of the sample crop phenotyping data; Indicates collection of The time difference between the moment of sample crop phenotyping data and the reference moment; Represents the time attenuation factor, which determines the influence of the time difference on Ti. The value can be set according to the actual application.

[0097] The product of the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data is calculated as an intermediate result.

[0098] The quotient of the intermediate result and the time consistency evaluation value of the sample crop phenotypic data is calculated as the weight value corresponding to the sample crop phenotypic data.

[0099] Specifically, the weight value corresponding to the sample crop phenotypic data can be calculated by the following formula:

[0100] in, Indicates The weight value corresponding to the sample crop phenotypic data; Indicates The first weight score corresponding to the sample crop phenotypic data; Indicates The second weight score corresponding to the sample crop phenotypic data; Indicates The third weight score corresponding to the sample crop phenotypic data; Indicates The fourth weight score corresponding to the sample crop phenotypic data.

[0101] The embodiment of the present invention obtains a first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data, obtains a second weight score corresponding to the sample crop phenotypic data based on the relative position information between the sample crop phenotypic sensor for collecting the sample crop phenotypic data and the sample crop, obtains a third weight score corresponding to the sample crop phenotypic data based on the environmental data when collecting the sample crop phenotypic data and the type of the sample crop phenotypic sensor, and obtains a fourth weight score corresponding to the sample crop phenotypic data based on the growth stage of the sample crop when collecting the sample crop phenotypic data and the type of the sample crop phenotypic sensor. The embodiment of the present invention can comprehensively consider the data quality of the sample crop phenotypic data, the relative position relationship between the sample crop phenotypic sensor and the sample crop when collecting the sample crop phenotypic data, the environmental factors when the sample crop phenotypic sensor collects the sample crop phenotypic data, and the influence of the growth stage of the sample crop on the sample crop phenotypic data, so as to more accurately obtain the weight value corresponding to the sample crop phenotypic data, and provide a more accurate data basis for the weight adjustment model.

[0102] Step 104: Based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, data fusion is performed on each crop phenotypic data to be fused to obtain fused data corresponding to each crop phenotypic data to be fused.

[0103] Specifically, after obtaining the weight value corresponding to each first crop phenotypic data to be fused, the phenotypic data to be fused can be fused based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, through numerical calculation, mathematical statistics, deep learning technology and other methods, to obtain the fused data corresponding to each crop phenotypic data to be fused.

[0104] As an optional embodiment, based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, data fusion is performed on each crop phenotypic data to be fused to obtain fused data corresponding to each crop phenotypic data to be fused, including: denoising each crop phenotypic data to be fused to obtain each crop phenotypic data to be fused after denoising.

[0105] Specifically, in the embodiment of the present invention, each crop phenotypic data to be fused can be denoised by wavelet denoising, Kalman filtering, etc., to obtain each crop phenotypic data to be fused after denoising.

[0106] Based on the internal and external reference data of the crop phenotypic sensor that collects the phenotypic data of each crop to be fused, the phenotypic data of each crop to be fused after denoising are initially aligned to obtain the phenotypic data of each crop to be fused after initial alignment.

[0107] Specifically, after obtaining each crop phenotypic data to be fused after denoising, the phenotypic data of each crop to be fused after denoising can be initially aligned using a synchronous multi-view geometry correction method based on the internal and external parameter data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, so as to achieve alignment of the phenotypic data of each crop to be fused after denoising in the time and space dimensions, and obtain the phenotypic data of each crop to be fused after initial alignment.

[0108] The specific alignment method includes: using the Levenberg-Marquardt optimization algorithm and the external parameter self-calibration algorithm to align the phenotypic data of each crop to be fused in time and space after denoising.

[0109] Based on the weight value corresponding to each first crop phenotypic data to be fused, data fusion is performed on each first crop phenotypic data to be fused after initial alignment to obtain first fused data.

[0110] Specifically, after obtaining the phenotypic data of each crop to be fused after the initial alignment, the phenotypic data of each crop to be fused after the initial alignment can be fused through numerical calculation based on the weight value corresponding to each first phenotypic data of the crop to be fused to obtain the first fused data.

[0111] As an optional embodiment, based on the weight value corresponding to each first crop phenotypic data to be fused, data fusion is performed on each first crop phenotypic data to be fused after initial alignment to obtain first fused data, including: based on the weight value corresponding to each first crop phenotypic data to be fused, data fusion is performed on each first crop phenotypic data to be fused after initial alignment using an iterative nearest point algorithm to obtain first fused data.

[0112] It should be noted that the Iterative Closest Point (ICP) algorithm is an iterative calculation method, which is mainly used for accurate stitching of depth images and point cloud matching (rigid registration) in computer vision. The ICP algorithm achieves accurate stitching or registration by continuously iteratively minimizing the distance between the corresponding points of the source data and the target data.

[0113] In the embodiment of the present invention, based on the weight value corresponding to each first crop phenotypic data to be fused, when the iterative closest point algorithm is used to perform data fusion on the first crop phenotypic data to be fused after initial alignment, the first crop phenotypic data to be fused is calculated according to the weight value corresponding to each first crop phenotypic data to be fused. The first crop phenotypic data to be fused is the distance The minimum error between the first crop phenotypic data to be fused and the closest first crop phenotypic data to be fused , the specific calculation formula is as follows:

[0114] in, represents the total number of phenotypic data of each first crop to be fused; Indicates The weight value corresponding to the first crop phenotypic data to be fused; Indicates The first crop phenotypic data to be integrated; Indicates the distance The first crop phenotypic data to be fused is the most recent first crop phenotypic data to be fused.

[0115] In each iterative calculation, the position and posture (rotation matrix and displacement vector) of the first crop phenotypic data to be fused are adjusted to minimize the weighted error function and ensure the accurate alignment of each first crop phenotypic data to be fused.

[0116] Through the above algorithm, the data fusion accuracy can be maximized in each first crop phenotypic data to be fused, ensuring that the first fused data obtained by data fusion has higher spatial consistency and feature expression ability.

[0117] The embodiment of the present invention not only obtains spatial alignment of each crop phenotypic data to be fused through least squares optimization, but also uses an iterative nearest point algorithm to fuse the first crop phenotypic data to be fused after initial alignment based on the weight value corresponding to each first crop phenotypic data to be fused to obtain first fused data, which can improve the fusion accuracy of the first crop phenotypic data to be fused, thereby further improving the data fusion accuracy and data fusion effect of the data fusion method provided by the present invention.

[0118] Each second crop phenotypic data to be fused after the initial alignment is fused with the first fused data to obtain fused data, and the second crop phenotypic data to be fused is the crop phenotypic data among the crop phenotypic data to be fused except the first crop phenotypic data to be fused.

[0119] Specifically, after obtaining the first fused data, each second crop phenotypic data to be fused after initial alignment can be fused with the first fused data through deep learning technology, numerical calculation, mathematical statistics, etc., to obtain fused data of each crop phenotypic data to be fused.

[0120] As an optional embodiment, each second crop phenotypic data to be fused after the initial alignment is fused with the first fused data to obtain fused data corresponding to each crop phenotypic data to be fused, and the second crop phenotypic data to be fused is the crop phenotypic data among the crop phenotypic data to be fused except the first crop phenotypic data to be fused, including: performing feature extraction on each second crop phenotypic data to be fused after the initial alignment, and obtaining feature information corresponding to each second crop phenotypic data to be fused after the initial alignment.

[0121] Specifically, in the embodiment of the present invention, a multi-scale convolution kernel of a convolutional neural network (CNN) can be used to extract feature information of different scales in each second crop phenotypic data to be fused. By extracting features at different scales, effective fusion of high-resolution and low-resolution data can be ensured, and the feature expression of fused data can be enhanced.

[0122] The characteristic information corresponding to each second crop phenotypic data to be fused after the initial alignment is mapped into the first fused data to obtain fused data.

[0123] Specifically, in the embodiment of the present invention, a deep learning network based on spatiotemporal consistency is adopted to map the feature information corresponding to each second crop phenotypic data to be fused after the initial alignment into the first fused data, thereby achieving a deep fusion of the feature information corresponding to each second crop phenotypic data to be fused with the first fused data.

[0124] The deep learning network based on spatiotemporal consistency includes a spatiotemporal consistency calibration module and a multi-level feature enhancement module.

[0125] The spatiotemporal consistency calibration module processes each second crop phenotypic data to be fused by using a recurrent neural network (RNN) to ensure the consistency of each second crop phenotypic data to be fused in the time dimension.

[0126] The data mapping module maps the feature information corresponding to each second crop phenotypic data to be fused after initial alignment into the first fused data through the spatial position correspondence between the first fused data and the feature information corresponding to each second crop phenotypic data to be fused, so as to obtain the original fused data.

[0127] The multi-level feature enhancement module uses a deep convolutional network to perform feature enhancement processing on the original fused data, extracts more detailed crop features, and further increases the weight of key features through the attention mechanism to obtain fused data.

[0128] It should be noted that after obtaining the fusion data of each crop phenotypic data to be fused, the fusion data can be stored in a standardized adaptive data structure, and the structure of any data point in the fusion data can include: XYZRGBS1S2S3S4T1T2WsEm1Em2TimeType1….

[0129] Among them, XYZ represents the three-dimensional spatial position of the data point. RGB represents the color information obtained by the visible light image sensor. S1-S4 represent the multispectral reflectance of different bands. T1 represents the thermal imaging temperature data, and T2 represents the ambient temperature. Ws represents the ambient wind speed. Em1-Em2 identify the sensor exposure time. Time is the acquisition timestamp, and Type1 represents the semantic annotation (such as plant type).

[0130] The embodiment of the present invention inputs the target data corresponding to each first crop phenotypic data to be fused into a weight adjustment model, obtains the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model, and then performs data fusion on each crop phenotypic data to be fused based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, so as to obtain the fusion data corresponding to each crop phenotypic data to be fused. The embodiment of the present invention can comprehensively consider the data quality of the sample crop phenotypic data, the relative position relationship between the sample crop phenotypic sensor and the sample crop when collecting the sample crop phenotypic data, the environmental factors when the sample crop phenotypic sensor collects the sample crop phenotypic data, and the influence of the growth stage of the sample crop on the sample crop phenotypic data, so as to more accurately obtain the weight value corresponding to the sample crop phenotypic data, and then more efficiently and accurately obtain the weight value corresponding to each crop phenotypic data to be fused by deep learning, so as to improve the fusion efficiency and fusion effect of crop phenotypic data fusion, and provide a more accurate data basis for high-precision three-dimensional modeling, crop feature extraction and other application scenarios of precision agriculture, and has broad application prospects.

[0131] The crop phenotyping data fusion method provided by the present invention uses an adaptive weight adjustment method to change the contribution of different data sources in real time according to the actual situation, thereby improving the flexibility and accuracy of data fusion. Combined with machine learning algorithms and self-learning mechanisms, the fusion strategy can be quickly adjusted, greatly reducing data processing time and adapting to the time-sensitive crop phenotyping application requirements. Through the fusion of multi-dimensional data, especially the synchronous processing of spatiotemporal data, the overall relevance and analysis accuracy of the data can be significantly improved.

[0132] Figure 2 Schematic diagram of the structure of the crop phenotype data fusion device provided by the present invention. Figure 2The crop phenotype data fusion device provided by the present invention is described. The crop phenotype data fusion device described below and the crop phenotype data fusion method provided by the present invention described above can be referred to each other. Figure 2 As shown, the device includes: a first data acquisition module 201, a second data acquisition module 202, a dynamic weight determination module 203 and a multi-source data fusion module 204.

[0133] The first data acquisition module 201 is used to acquire the phenotypic data of each crop to be fused, wherein each phenotypic data of each crop to be fused includes the crop phenotypic data of the target crop collected by different types of crop phenotypic sensors and / or the crop phenotypic data of the target crop at multiple growth stages.

[0134] The second data acquisition module 202 is used to obtain the internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and the target data corresponding to the first crop phenotypic data to be fused. The first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused. The target data includes the environmental data when collecting the crop phenotypic data, the relative position information between the crop phenotypic sensor collecting the crop phenotypic data and the crop, and the target parameter value of the crop phenotypic data. The target parameters include the signal-to-noise ratio and / or characteristic entropy.

[0135] The dynamic weight determination module 203 is used to input the target data corresponding to each first crop phenotypic data to be fused into the weight adjustment model, and obtain the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model. The weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each sample crop phenotypic data. The sample crop phenotypic data includes the crop phenotypic data of the sample crops collected by different types of crop phenotypic sensors and the crop phenotypic data of the sample crops at multiple growth stages.

[0136] The multi-source data fusion module 204 is used to fuse the crop phenotypic data to be fused based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, so as to obtain the fused data corresponding to each crop phenotypic data to be fused.

[0137] Specifically, the first data acquisition module 201 , the second data acquisition module 202 , the dynamic weight determination module 203 and the multi-source data fusion module 204 are electrically connected.

[0138] The crop phenotypic data fusion device in the embodiment of the present invention inputs the target data corresponding to each first crop phenotypic data to be fused into the weight adjustment model, obtains the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model, and then performs data fusion on each crop phenotypic data to be fused based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, so as to obtain the fusion data corresponding to each crop phenotypic data to be fused. The device can comprehensively consider the data quality of the sample crop phenotypic data, the relative position relationship between the sample crop phenotypic sensor and the sample crop when collecting the sample crop phenotypic data, the environmental factors when the sample crop phenotypic sensor collects the sample crop phenotypic data, and the influence of the growth stage of the sample crop on the sample crop phenotypic data, so as to more accurately obtain the weight value corresponding to the sample crop phenotypic data, and then more efficiently and accurately obtain the weight value corresponding to each crop phenotypic data to be fused by deep learning, so as to improve the fusion efficiency and fusion effect of crop phenotypic data fusion, and provide a more accurate data basis for high-precision three-dimensional modeling, crop feature extraction and other application scenarios of precision agriculture, and has broad application prospects.

[0139] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the crop phenotypic data fusion method, which includes: obtaining each crop phenotypic data to be fused, each crop phenotypic data to be fused includes crop phenotypic data of a target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of a target crop at multiple growth stages; obtaining the internal and external reference data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and the target data corresponding to the first crop phenotypic data to be fused, the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, the target data includes the environmental data when collecting the crop phenotypic data, the relative position information between the crop phenotypic sensor collecting the crop phenotypic data and the crop, and the target parameter value of the crop phenotypic data, the target parameter includes the signal-to-noise ratio and / or characteristic entropy; inputting the target data corresponding to each first crop phenotypic data to be fused into a weight adjustment model, obtaining the weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model, wherein the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each sample crop phenotypic data, and the sample crop phenotypic data includes the crop phenotypic data of the sample crop collected by different types of crop phenotypic sensors and the crop phenotypic data of the sample crop at multiple growth stages; based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, data fusion is performed on each crop phenotypic data to be fused to obtain fused data corresponding to each crop phenotypic data to be fused.

[0140] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the crop phenotypic data fusion method provided by the above methods, which method includes: obtaining each crop phenotypic data to be fused, each crop phenotypic data to be fused includes crop phenotypic data of a target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; obtaining internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes environmental data when collecting crop phenotypic data, and the relationship between the crop phenotypic sensor collecting crop phenotypic data and the crop. The target parameters include signal-to-noise ratio and / or characteristic entropy; the target data corresponding to each first crop phenotypic data to be fused is input into a weight adjustment model, and a weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model is obtained, the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each sample crop phenotypic data, and the sample crop phenotypic data include crop phenotypic data of sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of sample crops at multiple growth stages; based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, the crop phenotypic data to be fused are fused to obtain fused data corresponding to each crop phenotypic data to be fused.

[0142] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the crop phenotypic data fusion method provided by the above-mentioned methods, the method comprising: obtaining each crop phenotypic data to be fused, each crop phenotypic data to be fused comprising crop phenotypic data of a target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; obtaining internal and external reference data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, the first crop phenotypic data to be fused being three-dimensional crop phenotypic data to be fused, the target data comprising environmental data when collecting crop phenotypic data, relative position information between the crop phenotypic sensor collecting crop phenotypic data and the crop, and crop phenotypic data. The target parameter value of each first crop phenotypic data to be fused is input into a weight adjustment model to obtain a weight value corresponding to each first crop phenotypic data to be fused output by the weight adjustment model, the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each sample crop phenotypic data, the sample crop phenotypic data including crop phenotypic data of the sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of the sample crops at multiple growth stages; based on the weight value corresponding to each first crop phenotypic data to be fused and the internal and external reference data of the crop phenotypic sensor that collects each crop phenotypic data to be fused, the crop phenotypic data to be fused are fused to obtain fused data corresponding to each crop phenotypic data to be fused.

[0143] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A crop phenotypic data fusion method, characterized in that: include: Acquire each crop phenotypic data to be fused, wherein each crop phenotypic data to be fused includes crop phenotypic data of a target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; Acquire internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, wherein the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes environmental data when collecting crop phenotypic data, relative position information between the crop phenotypic sensor collecting crop phenotypic data and the crop, and target parameter value of the crop phenotypic data, and the target parameter includes signal-to-noise ratio and / or characteristic entropy; Inputting target data corresponding to each of the first crop phenotypic data to be fused into a weight adjustment model, and obtaining a weight value corresponding to each of the first crop phenotypic data to be fused output by the weight adjustment model, wherein the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each of the sample crop phenotypic data, and the sample crop phenotypic data includes crop phenotypic data of sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of the sample crops at multiple growth stages; Based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, data fusion is performed on each of the crop phenotypic data to be fused to obtain fused data corresponding to each of the crop phenotypic data to be fused.

2. The crop phenotypic data fusion method according to claim 1, characterized in that: The weight values ​​corresponding to the sample crop phenotypic data are obtained based on the following steps: Based on the target parameter value of the sample crop phenotypic data, a first weight score corresponding to the sample crop phenotypic data is obtained; based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop, a second weight score corresponding to the sample crop phenotypic data is obtained; based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a third weight score corresponding to the sample crop phenotypic data is obtained; based on the growth stage of the sample crop when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a fourth weight score corresponding to the sample crop phenotypic data is obtained; Based on the first weight score, the second weight score, the third weight score and the fourth weight score corresponding to the sample crop phenotypic data, the weight value corresponding to the sample crop phenotypic data is calculated.

3. The crop phenotypic data fusion method according to claim 1, characterized in that: The method of fusing each of the crop phenotypic data to be fused based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused to obtain fused data corresponding to each of the crop phenotypic data to be fused includes: Performing denoising processing on each of the crop phenotypic data to be fused, and obtaining each of the crop phenotypic data to be fused after denoising processing; Based on the internal and external reference data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, initially aligning the denoised crop phenotypic data to be fused to obtain the crop phenotypic data to be fused after the initial alignment; Based on the weight value corresponding to each of the first crop phenotypic data to be fused, data fusion is performed on each of the first crop phenotypic data to be fused after initial alignment to obtain first fused data; Each second crop phenotypic data to be fused after initial alignment is fused with the first fused data to obtain the fused data, wherein the second crop phenotypic data to be fused is the crop phenotypic data in each crop phenotypic data to be fused except the first crop phenotypic data to be fused.

4. The crop phenotypic data fusion method according to claim 2, characterized in that: The step of obtaining a first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data includes: Determine the target parameter value of the sample crop phenotypic data as an independent variable, determine the first weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the first weight score corresponding to the sample crop phenotypic data based on the target parameter value of the sample crop phenotypic data and a positive correlation function, wherein the positive correlation function is used to describe the positive correlation between the independent variable and the dependent variable; The step of obtaining a second weight score corresponding to the sample crop phenotypic data based on the relative position information between the sample crop phenotypic sensor that collects the sample crop phenotypic data and the sample crop comprises: Based on the relative position information between the sample crop phenotype sensor and the sample crop, the distance between the sample crop phenotype sensor and the sample crop and the vertical distance between the sample crop and a target reference line are acquired, where the target reference line is the central axis of the field of view of the sample crop phenotype sensor; Determine the distance as an independent variable, determine the first sub-weight score corresponding to the sample crop phenotypic data as a dependent variable, calculate the first sub-weight score corresponding to the sample crop phenotypic data based on the distance and a positive correlation function, determine the vertical distance as an independent variable, determine the second sub-weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the second sub-weight score corresponding to the sample crop phenotypic data based on the distance and a negative correlation function; Based on the first sub-weight score and the second sub-weight score corresponding to the sample crop phenotypic data, a second weight score corresponding to the sample crop phenotypic data is calculated; The obtaining, based on the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor, a third weight score corresponding to the sample crop phenotypic data comprises: Acquire a first matching degree between the environmental data when the sample crop phenotypic data is collected and the type of the sample crop phenotypic sensor; Determine the first matching degree as an independent variable, determine the third weight score corresponding to the sample crop phenotypic data as a dependent variable, and calculate the third weight score corresponding to the sample crop phenotypic data based on the first matching degree and a positive correlation function; The acquiring, based on the growth stage of the sample crop and the type of the sample crop phenotypic sensor when the sample crop phenotypic data is collected, a fourth weight score corresponding to the sample crop phenotypic data, comprises: Acquiring a second matching degree between the growth stage of the sample crop and the type of the sample crop phenotypic sensor when collecting the sample crop phenotypic data; The second matching degree is determined as an independent variable, and the fourth weight score corresponding to the sample crop phenotypic data is determined as a dependent variable. Based on the second matching degree and a positive correlation function, the fourth weight score corresponding to the sample crop phenotypic data is calculated.

5. The crop phenotypic data fusion method according to claim 4, characterized in that: The first weight score, the second weight score, the third weight score and the fourth weight score based on the sample crop phenotypic data are used to calculate the weight value corresponding to the sample crop phenotypic data, including: Obtaining a temporal consistency evaluation value of the sample crop phenotypic data; Calculating the product of the first weight score, the second weight score, the third weight score and the fourth weight score of the sample crop phenotypic data as an intermediate result; The quotient of the intermediate result and the time consistency evaluation value of the sample crop phenotypic data is calculated as the weight value corresponding to the sample crop phenotypic data.

6. The crop phenotypic data fusion method according to claim 3, characterized in that: The step of fusing the first crop phenotypic data to be fused after initial alignment based on the weight value corresponding to each of the first crop phenotypic data to be fused to obtain first fused data includes: Based on the weight value corresponding to each of the first crop phenotypic data to be fused, an iterative closest point algorithm is used to perform data fusion on the first crop phenotypic data to be fused after initial alignment to obtain the first fused data.

7. The crop phenotypic data fusion method according to claim 6, characterized in that: The step of fusing each second crop phenotypic data to be fused after the initial alignment with the first fused data to obtain fused data corresponding to each crop phenotypic data to be fused, wherein the second crop phenotypic data to be fused is the crop phenotypic data in each crop phenotypic data to be fused except the first crop phenotypic data to be fused, including: Performing feature extraction on each of the second crop phenotypic data to be fused after the initial alignment, and acquiring feature information corresponding to each of the second crop phenotypic data to be fused after the initial alignment; The characteristic information corresponding to each of the second crop phenotypic data to be fused after initial alignment is mapped into the first fused data to obtain the fused data.

8. A crop phenotypic data fusion device, characterized in that: include: A first data acquisition module is used to acquire phenotypic data of each crop to be fused, wherein each of the phenotypic data of the crop to be fused includes crop phenotypic data of the target crop collected by different types of crop phenotypic sensors and / or crop phenotypic data of the target crop at multiple growth stages; A second data acquisition module is used to acquire internal and external parameter data of each crop phenotypic sensor when collecting each crop phenotypic data to be fused and target data corresponding to the first crop phenotypic data to be fused, wherein the first crop phenotypic data to be fused is three-dimensional crop phenotypic data to be fused, and the target data includes environmental data when collecting crop phenotypic data, relative position information between the crop phenotypic sensor collecting crop phenotypic data and the crop, and target parameter values ​​of the crop phenotypic data, and the target parameters include signal-to-noise ratio and / or characteristic entropy; a dynamic weight determination module, for inputting target data corresponding to each of the first crop phenotypic data to be fused into a weight adjustment model, and obtaining a weight value corresponding to each of the first crop phenotypic data to be fused output by the weight adjustment model, wherein the weight adjustment model is obtained after training based on the target data corresponding to each sample crop phenotypic data in the sample data group and the weight value corresponding to each of the sample crop phenotypic data, and the sample crop phenotypic data include crop phenotypic data of sample crops collected by different types of crop phenotypic sensors and crop phenotypic data of the sample crops at multiple growth stages; A multi-source data fusion module is used to fuse the crop phenotypic data to be fused based on the weight value corresponding to each of the first crop phenotypic data to be fused and the internal and external parameter data of the crop phenotypic sensor that collects each of the crop phenotypic data to be fused, so as to obtain the fused data corresponding to each of the crop phenotypic data to be fused.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the crop phenotypic data fusion method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the crop phenotypic data fusion method according to any one of claims 1 to 7 is implemented.

Images