Multi-modal data fusion crop yield prediction method, device and medium

Abstract

The invention discloses a multi-modal data fusion crop yield prediction method and device and a medium, and relates to the field of crop yield assessment, and the method comprises the steps: collecting an original image of a target farmland; preprocessing the original image of the target farmland to obtain preprocessed images of a plurality of quadrat areas; based on the preprocessed image of each quadrat region, extracting crop parameters of each quadrat region; determining feature information of each quadrat region based on the preprocessed image and crop parameters of each quadrat region; respectively inputting the feature information of each quadrat region into a multi-modal data fusion prediction model to obtain a crop yield prediction value of each quadrat region; and summing the crop yield predicted values of all the quadrat areas to obtain a crop yield predicted value of the target farmland. According to the invention, manual intervention is reduced, and the crop yield prediction efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the technical field of crop yield assessment, and in particular to a crop yield prediction method, device and medium using multimodal data fusion. Background Art

[0002] Crop yield is a core indicator of agricultural production and is directly related to food security, economic development and ecological balance. High crop yield can not only meet the needs of the growing population, but also increase farmers' income and promote the sustainable development of the rural economy. In addition, reasonable crop yield assessment and prediction can help optimize resource allocation, reduce waste, and promote agricultural technology innovation, thereby achieving ecological environmental protection and efficient agricultural development.

[0003] In order to obtain the true value of crop yield, the traditional method is to sample and dry the crops in the farmland, use a moisture meter to measure the moisture of the dried crops, and then weigh the crops in a dry state to obtain their net weight. Finally, the crop yield per acre of farmland is determined based on the measured net weight and moisture content. Although the traditional method can determine the true value of crop yield, it requires a lot of manpower time and energy, and the entire process is inefficient.

[0004] With the advancement of science and technology, a crop yield prediction method that relies on feature selection and traditional machine learning model modeling has gradually emerged. Although this method has low requirements for computing resources and storage space, it still needs to rely on manual selection and design of features, and this process often requires certain professional knowledge. Otherwise, it may not be able to effectively capture key feature information, thereby affecting the accuracy of crop parameter prediction. Summary of the invention

[0005] The purpose of this application is to provide a crop yield prediction method, device and medium based on multimodal data fusion, which reduces human intervention and improves the efficiency and accuracy of crop yield prediction.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a crop yield prediction method based on multimodal data fusion, the crop yield prediction method based on multimodal data fusion comprising: collecting original images of target farmland; the original images comprising: original visible light images, original multispectral images and original thermal infrared images; preprocessing the original images of the target farmland to obtain preprocessed images of multiple sample areas; the multiple sample areas are obtained by segmenting the target farmland; the preprocessed images comprising: preprocessed visible light images, preprocessed multispectral images and preprocessed thermal infrared images; based on the preprocessed images of each sample area, extracting crop parameters of each sample area; the crop parameters The data include: vegetation index, texture characteristics and canopy coverage; based on the preprocessed images and crop parameters of each sample area, the characteristic information of each sample area is determined; the characteristic information includes image features and text features; the characteristic information of each sample area is input into the multimodal data fusion prediction model respectively to obtain the crop yield prediction value of each sample area; the multimodal data fusion prediction model is constructed based on the improved ResNet model; the improved ResNet model expands the input interface and customizes the data loading class; the crop yield prediction values ​​of all sample areas are summed to obtain the crop yield prediction value of the target farmland.

[0008] In a second aspect, the present application also provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the crop yield prediction method based on multimodal data fusion described in the first aspect.

[0009] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the crop yield prediction method based on multimodal data fusion described in the first aspect.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0011] This application preprocesses the original images of the target farmland and extracts crop parameters, and finally determines the image features and text features of each sample area. Compared with a single feature, this combination of multimodal features can better enhance the characteristic information of crops in the sample area. In order to process image features and text features at the same time, this application also improves the ResNet model, expands the input interface of the ResNet model and customizes the data loading class, so that the multimodal data fusion prediction model built based on the improved ResNet model can better enhance the fusion of image features and text features, and can more effectively capture the relationship between multimodal features and crop yield prediction values, while also providing greater flexibility for cross-modal learning. Therefore, this application improves the efficiency and accuracy of crop yield prediction while reducing human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 A flowchart of a crop yield prediction method based on multimodal data fusion provided in an embodiment of the present application.

[0014] Figure 2 A schematic diagram of the structure of a multimodal data fusion prediction model provided in an embodiment of the present application.

[0015] Figure 3 A schematic diagram of the structure of the residual module provided in an embodiment of the present application.

[0016] Figure 4 A scatter plot of crop yield during the jointing period provided in an embodiment of the present application.

[0017] Figure 5 A scatter plot of crop yield during flowering period provided in an embodiment of the present application.

[0018] Figure 6 A scatter plot of crop yield during the grain filling period provided in an embodiment of the present application.

[0019] Figure 7 A scatter plot of crop yield at maturity provided in an embodiment of the present application.

[0020] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The purpose of this application is to provide a crop yield prediction method, device and medium based on multimodal data fusion, which reduces human intervention and improves the efficiency and accuracy of crop yield prediction.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Example 1 like Figure 1 As shown, this embodiment provides a crop yield prediction method based on multimodal data fusion, and the crop yield prediction method based on multimodal data fusion is specifically as follows.

[0025] Step S1: collecting original images of the target farmland; the original images include: original visible light images, original multispectral images and original thermal infrared images.

[0026] In this embodiment, a field planted with winter wheat is selected as the target farmland, which is divided into 54 rectangular sample areas, and the size of each sample area is 2m×20m.

[0027] A DJI Phantom 4 drone was used, equipped with one visible light sensor and five monochrome sensors, namely blue, green, red, red-edge and near-infrared. The drone's flight altitude was set to 20m, and the heading and lateral overlap rates were set to 80%. Before each flight, radiation calibration plates with three reflectances of 25%, 50% and 75% were photographed for radiation calibration. The original visible light images and original multispectral images of the target farmland were collected during the jointing, flowering, filling and maturity stages of winter wheat.

[0028] A Mavic 3 Thermal drone was used, equipped with a thermal imaging sensor and a 4 / 3 CMOS sensor. The flight altitude was set to 20m, and the heading and lateral overlap rates were set to 80%. A temperature gun was used to measure the ambient temperature and ground temperature before each flight (mainly to facilitate later temperature correction). Original thermal infrared images of the target farmland were collected during the jointing, flowering, filling and maturity stages of winter wheat.

[0029] Step S2: preprocessing the original image of the target farmland to obtain preprocessed images of multiple sample areas; the preprocessed images include: preprocessed visible light images, preprocessed multispectral images and preprocessed thermal infrared images.

[0030] In this embodiment, step S2 is specifically as follows.

[0031] After the original visible light images of the target farmland are stitched, they are cropped according to the area of ​​each sample area to obtain the preprocessed visible light images of each sample area.

[0032] After image stitching and radiation correction, the original multispectral images of the target farmland are cropped according to the area of ​​each sample area to obtain the preprocessed multispectral images of each sample area.

[0033] After the original thermal infrared images of the target farmland are stitched and temperature corrected, they are cropped according to the area of ​​each sample area to obtain the preprocessed thermal infrared images of each sample area.

[0034] Step S3: extracting crop parameters of each sample area based on the preprocessed images of each sample area; the crop parameters include: vegetation index, texture characteristics and canopy coverage.

[0035] In this embodiment, step S3 is specifically as follows.

[0036] The emissivity of the pre-processed multispectral images of each sample area is extracted and the vegetation index is constructed. After recursive feature elimination of the constructed vegetation index, the vegetation index of each sample area is obtained.

[0037] The texture features of the pre-processed visible light images of each sample area are extracted to obtain the texture features of each sample area.

[0038] The canopy coverage of each sample area was obtained by ratioing the number of crop pixels in the preprocessed visible light image to the number of soil background pixels.

[0039] Step S4: based on the preprocessed images and crop parameters of each sample area, determining the characteristic information of each sample area; the characteristic information includes: image features and text features.

[0040] In this embodiment, step S4 is specifically as follows.

[0041] The preprocessed visible light image, preprocessed multispectral image and preprocessed thermal infrared image of each sample area are horizontally flipped and randomly rotated to obtain the image features of each sample area.

[0042] The vegetation index, texture characteristics and canopy coverage of each sample area were standardized and normalized to obtain the text characteristics of each sample area.

[0043] Step S5: Input the characteristic information of each sample area into the multimodal data fusion prediction model to obtain the crop yield prediction value of each sample area.

[0044] In this embodiment, the multimodal data fusion prediction model is constructed based on the improved ResNet model, which extends the input interface and customizes the data loading class. Among them, after the improved ResNet model extends the input interface, it is convenient to accept features from multimodal data sources. After the improved ResNet model customizes the data loading class, the class accepts multiple path parameters in the initialization, supports different types of image inputs, and designs a special loading, resizing, normalization and conversion to tensor process for each type of image, while the conventional ResNet model usually only processes RGB images; this class sets independent mean and standard deviation for different types of images for normalization, improves the processing accuracy of specific data types, and this personalized normalization processing method is more refined than the conventional ResNet model, which can bring better performance in multimodal data processing; this class can return a tuple containing multiple data: RGB image tensor, multispectral image tensor, thermal infrared image tensor, text feature tensor and crop yield label, which can adapt to multimodal input requirements, while the conventional ResNet model only processes single-modal RGB image input. The improved ResNet model constructs convolution channels for three different input image data: visible light images, multispectral images, and thermal infrared images. This branch design improves the multimodal processing capability of the residual block, supports the simultaneous input of multiple image data sources, and can adjust the network structure for different data modes. The improved ResNet model allows the simultaneous processing of image features and text features, and enhances information fusion by merging image features with text features in the forward propagation. At the same time, it can adjust the number of channels of image feature input and the dimension of text feature input according to actual applications.

[0045] Furthermore, the multimodal data fusion prediction model includes a first convolution module, a maximum pooling layer, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a global average pooling layer, a flattening layer, a first fully connected layer and a second fully connected layer, which are connected in sequence; the first convolution module is composed of a convolution layer, a batch normalization layer and an activation layer; the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are all composed of a convolution layer and a residual module.

[0046] like Figure 2As shown in the figure, the image features of 224×224×9 and the text features of 18×1 are input into the first convolution module. The convolution kernel of the convolution layer in the first convolution module is 7×7, the activation layer is the ReLU activation function, the output size of the first convolution module is 64×112×112, the output size of the maximum pooling layer is 64×56×56, the convolution kernel of the convolution layer in the second convolution module is 3×3, the number of convolution kernels is 256, the output size of the second convolution module is 64×56×56, there are 2 residual modules, the convolution kernel of the convolution layer in the third convolution module is 3×3, the number of convolution kernels is 512, the output size of the third convolution module is 128×28×28, and the residual module There are 2, the convolution kernel of the convolution layer in the fourth convolution module is 3×3, the number of convolution kernels is 1024, the output size of the fourth convolution module is 256×14×14, there are 2 residual modules, the convolution kernel of the convolution layer in the fifth convolution module is 3×3, the number of convolution kernels is 2048, the output size of the second convolution module is 512×7×7, there are 2 residual modules, the convolution kernel in the global average pooling layer is 3×3, the number of convolution kernels is 2048, the output size of the global average pooling layer is 512×1×1, the output size of the flat layer is 512, the output size of the first fully connected layer is 128, and the output size of the second fully connected layer is 1. Finally, the crop yield prediction value is obtained.

[0047] like Figure 3 As shown in the figure, in the residual module, (1) the input data first enters the convolution layer and the activation layer: In the main path, after the input data passes through the first convolution layer, batch normalization and activation function, some preliminary features will be extracted. (2) The jump connection retains the original features: In this process, the jump connection will pass the input directly to the output without being processed by the convolution layer, which means that the features of the input data will not be "filtered" or "transformed", but will be retained as unprocessed features in the output of the residual module. (3) The main path features are added to the jump connection features: After the main path completes all convolution, normalization and activation operations, the resulting feature map will be added to the features of the jump connection. This addition operation is the "residual" part of the residual module. The final output is the superposition of the two parts of the features, including the features processed by the convolution and the unprocessed original features. In addition, the skip connection allows the gradient to be directly back-propagated from the output to the input, avoiding the problem of gradient vanishing in deep networks. Since the skip connection can effectively utilize shallow features, the network can be designed to be deeper without worrying about performance degradation, and the training speed is faster. The skip connection ensures that the input information is not completely lost. Even if the convolution operation of the main path loses some information, the skip connection will still retain the original feature information, which is very helpful to improve the expressiveness of the model.

[0048] As a preferred implementation, the process of determining the multimodal data fusion prediction model includes: Step 100: Acquire original images of different farmlands.

[0049] Step 200: Based on original images of different farmlands, determine image features and text features of each test area in the different farmlands.

[0050] The process of determining the image features and text features of each test area in different farmlands is the same as the above steps S1 to S4, and will not be repeated here.

[0051] Step 300: determine the true value of the crop yield of each test area in different farmlands, and standardize and normalize the true value of the crop yield of each test area to obtain the calibrated value of the crop yield of each test area.

[0052] Among them, crops in each test area are harvested at the maturity stage of different farmlands, and the moisture content of the crops after drying is used to weigh the dried crops and convert them into yield per mu, so as to determine the true value of crop yield in each test area, and then obtain the calibrated value of crop yield after standardization and normalization.

[0053] Step 400: construct a training data set based on the image features, text features and crop yield calibration values ​​of all test areas.

[0054] Step 500: Based on the training data set, the image features and text features of each test area are used as input, and the crop yield calibration value of each test area is used as output, and the improved ResNet model is trained to obtain a multimodal data fusion prediction model.

[0055] Among them, when training the improved ResNet model, the main thing is to continuously calculate its corresponding root mean square error (RMSE) and R 2 .

[0056] .

[0057] .

[0058] In the formula, is the crop yield calibration value, is the average value of the crop yield calibration value, is the crop yield prediction value, n is the number of test areas.

[0059] In different growth periods of crops, the crop yield prediction value, crop yield calibration value, RMSE, R 2 See also Figures 4 to 7 , when RMSE reaches the minimum and R 2When it is close to 1, the improved ResNet model is trained and a multimodal data fusion prediction model is obtained.

[0060] Step S6: sum the predicted crop yield values ​​of all sample areas to obtain the predicted crop yield value of the target farmland.

[0061] In order to verify the effectiveness of the above-mentioned multimodal data fusion crop yield prediction method, this embodiment trained two different network architecture models: a single-input single-output model (corresponding to the conventional ResNet model) and a multi-input single-output model (corresponding to the improved ResNet model) for comparison. In order to ensure the consistency of the training data, this embodiment uses the image features and text features of each test area in the jointing stage, flowering stage, filling stage, and maturity stage as the common data source for the two models. The final training results of the two models are shown in Table 1 below.

[0062] Table 1 Comparison of crop yield prediction performance of different models

[0063] For crop yield, the multi-input single-output model has a higher yield at the jointing stage (R 2 =0.768), flowering period (R 2 =0.785), grouting period (R 2 =0.761) and maturity (R 2 =0.761) performed better than the input prediction accuracy of single text features by 0.3849, 0.3650, 0.3463 and 0.3696 respectively, and the input prediction accuracy of single image features by 0.0822, 0.0406, 0.0358 and 0.0191 respectively. For the multi-input single-output model, it has a higher accuracy in predicting crop yield. At a flight altitude of 20m, the R 2 and RMSE are 0.7569 (0.1145), 0.9171 (0.0702), 0.8747 (0.0814), and 0.8775 (0.0792) respectively, which not only significantly improves the accuracy of the model, but also can flexibly adjust the input dimension and number of channels to optimize the model efficiency.

[0064] In summary, the above-mentioned multimodal data fusion crop yield prediction method uses the image features of multimodal remote sensing data and the text features extracted from the image as input, and uses the multimodal data fusion prediction model to perform end-to-end crop yield prediction, which effectively solves the time-consuming, inefficient and difficult development problems faced by manual measurement of crop yield and traditional machine learning models in the prediction process, and has the significant advantages of high precision, high efficiency and easy application development. In addition, the multimodal data fusion crop yield prediction method has a wide range of applicability and can be extended to different crops, providing innovative ideas for crop yield prediction and promoting the accuracy and efficiency of agricultural production.

[0065] Example 2 This embodiment provides a computer device, which may be a server or a terminal. Its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the original image of the target farmland. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned multimodal data fusion crop yield prediction method is implemented.

[0066] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0069] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0070] All actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0071] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0072] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A crop yield prediction method based on multimodal data fusion, characterized in that: The crop yield prediction method using multimodal data fusion includes: Collecting original images of the target farmland; the original images include: original visible light images, original multispectral images and original thermal infrared images; Preprocessing the original image of the target farmland to obtain preprocessed images of multiple sample areas; the multiple sample areas are obtained by segmenting the target farmland; the preprocessed images include: preprocessed visible light images, preprocessed multispectral images and preprocessed thermal infrared images; Extracting crop parameters of each sample area based on preprocessed images of each sample area; the crop parameters include: vegetation index, texture characteristics and canopy coverage; Determine feature information of each sample area based on preprocessed images and crop parameters of each sample area; the feature information includes: image features and text features; The characteristic information of each sample area is respectively input into the multimodal data fusion prediction model to obtain the crop yield prediction value of each sample area; the multimodal data fusion prediction model is constructed based on the improved ResNet model; the improved ResNet model expands the input interface and customizes the data loading class; The predicted crop yield values ​​of all sample areas are summed up to obtain the predicted crop yield value of the target farmland.

2. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: The original image of the target farmland is preprocessed to obtain preprocessed images of multiple sample areas, including: After the original visible light images of the target farmland are stitched, they are cropped according to the area of ​​each sample area to obtain the preprocessed visible light images of each sample area; After image stitching and radiation correction are performed on the original multispectral images of the target farmland, they are cropped according to the area of ​​each sample area to obtain the preprocessed multispectral images of each sample area; After the original thermal infrared images of the target farmland are stitched and temperature corrected, they are cropped according to the area of ​​each sample area to obtain the preprocessed thermal infrared images of each sample area.

3. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: Based on the preprocessed images of each sample area, the crop parameters of each sample area are extracted, including: The emissivity of the pre-processed multispectral images of each sample area is extracted and the vegetation index is constructed. After recursive feature elimination of the constructed vegetation index, the vegetation index of each sample area is obtained. Extracting texture features of pre-processed visible light images of each sample area to obtain texture features of each sample area; The canopy coverage of each sample area was obtained by ratioing the number of crop pixels in the preprocessed visible light image to the number of soil background pixels.

4. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: Based on the preprocessed images and crop parameters of each sample area, the characteristic information of each sample area is determined, including: The preprocessed visible light image, preprocessed multispectral image and preprocessed thermal infrared image of each sample area are horizontally flipped and randomly rotated to obtain the image features of each sample area; The vegetation index, texture characteristics and canopy coverage of each sample area were standardized and normalized to obtain the text characteristics of each sample area.

5. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: The improved ResNet model allows for simultaneous processing of image features and text features, and enhances information fusion by merging image features and text features in forward propagation; the improved ResNet model adjusts the number of channels of image feature input and the dimension of text feature input according to actual applications.

6. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: The multimodal data fusion prediction model includes a first convolution module, a maximum pooling layer, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a global average pooling layer, a flat layer, a first fully connected layer and a second fully connected layer connected in sequence; the first convolution module is composed of a convolution layer, a batch normalization layer and an activation layer; the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are all composed of a convolution layer and a residual module.

7. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: The process of determining the multimodal data fusion prediction model includes: Obtain original images of different farmlands; Based on the original images of different farmlands, the image features and text features of each test area in the different farmlands are determined; Determine the true value of crop yield in each test area in different farmlands, and standardize and normalize the true value of crop yield in each test area to obtain the calibrated value of crop yield in each test area; Construct a training dataset based on image features, text features, and crop yield calibration values ​​of all test areas; Based on the training data set, the image features and text features of each test area are used as input, and the crop yield calibration value of each test area is used as output, and the improved ResNet model is trained to obtain a multimodal data fusion prediction model.

8. The crop yield prediction method based on multimodal data fusion according to claim 1, characterized in that: Collect the original images of the target farmland, including: Use DJI Phantom 4 drone to collect raw visible light images and raw multispectral images of target farmland; The Mavic 3 Thermal UAV was used to collect raw thermal infrared images of the target farmland.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the crop yield prediction method based on multimodal data fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the crop yield prediction method using multimodal data fusion described in any one of claims 1 to 8 is implemented.

Images