Crop yield prediction method and device based on multi-modal information fusion

Abstract

The application provides a crop yield prediction method and device based on multi-modal information fusion and a computer storage medium, multi-modal crop data including daily growth data and historical annual yield data of crops are acquired; a crop yield prediction model is established and the crop yield prediction model is used for crop yield prediction, the model comprises a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module and a crop yield prediction module, on one hand, the application introduces crop daily growth picture time series data to perform fine-grained modeling on the growth trend of crops; on the other hand, the daily growth data and the historical annual yield of crops are modeled through a hierarchical time series modeling method, and the potential correlation between the historical annual yields of different crops in a region is considered, so that the crop yield prediction effect is further improved. The method has the characteristics of easy implementation and good effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agricultural information processing, in particular to a crop yield prediction method and device based on multi-modal information fusion and a computer storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and agricultural informatization technology, artificial intelligence technology has been gradually applied to various aspects of agricultural production and management, and has achieved good results. The purpose of crop yield prediction is to predict yield in advance during the growth period of crops, which has important value and significance for national planning of economic development, especially for formulating agricultural policies (including purchase, storage, processing and trade, etc.).

[0003] Currently, there are some methods for predicting the yield of different crops, which can be roughly divided into three categories: (1) statistical analysis-based methods, such as autoregressive integrated moving average model (ARIMA), grey prediction model, stepwise regression model, etc. This kind of model has relatively simple structure and considers fewer factors, and the model accuracy needs to be improved; (2) traditional machine learning methods, such as multiple linear regression (MLR), back propagation neural network (BPNN), random forest (RF), support vector machine (SVM), artificial neural network (ANN) and K-nearest neighbor (KNN), etc. This kind of model can establish the nonlinear relationship between soil, weather and spatial factors and yield, and has achieved good accuracy, but still has room for improvement; (3) deep learning methods, such as convolutional neural network (CNN) and long short-term memory (LSTM), etc. CNN is used to encode image data, and LSTM is used to model time series data. In the problem of crop yield prediction, the performance of deep learning methods is significantly better than that of traditional machine learning methods. However, the current crop yield prediction methods based on deep learning mainly use structured data such as soil and weather, and only a few use remote sensing image data, which slightly lacks modeling capability in tracking crop growth trends. SUMMARY

[0004] The present application aims at the above problems, and provides a crop yield prediction method and device based on multi-modal information fusion and a computer storage medium, which introduces crop daily growth picture time series data to model crop growth trend in a fine-grained manner, and models crop daily growth data and historical annual yield through a hierarchical time series modeling manner to consider potential correlation between different crop historical annual yields in a region, so as to further improve crop yield prediction effect, and the method has the characteristics of easy implementation and good effect.

[0005] In a first aspect of the present application, a crop yield prediction method based on multi-modal information fusion is provided, and the method comprises the following steps:

[0006] Crop multi-modal data including crop daily growth data and historical annual yield data are obtained;

[0007] A crop yield prediction model is established, and the crop yield prediction model is used for crop yield prediction, and the model comprises a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module, and a crop yield prediction module, wherein:

[0008] The representation module is used for representing the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation;

[0009] The first modeling module is used for modeling the multi-modal daily growth data representation for consecutive multiple days;

[0010] The first fusion module is used for fusing the output of the first modeling module to obtain an overall representation of multi-modal crop daily growth time series data;

[0011] The second fusion module is used for fusing the overall representation of multi-modal crop daily growth time series data and the historical annual yield data representation to obtain a fused overall annual data representation;

[0012] The second modeling module is used for modeling the overall annual data representation;

[0013] The crop yield prediction module is used for crop yield prediction according to the output of the second modeling module to obtain a target crop yield prediction result.

[0014] In a further technical solution of the present application, the first modeling module / the second modeling module is any one of LSTM and Transformer.

[0015] In a further technical solution of the present application, the second fusion module is a self-attention network.

[0016] A further technical solution of the present application is that the prediction method used by the crop yield prediction module is linear regression, and the loss function used is L1 loss or L2 loss.

[0017] A further technical solution of the present application is that the representation module represents the inputted crop multi-modal data, specifically including:

[0018] The obtained crop daily growth data is encoded by ResNet or DenseNet and the encoding results are average-pooled;

[0019] The crop fruit is detected by a YOLO series model, and the fruit detection results are numerically quantified;

[0020] The average-pooled output results and the numerically quantified output results are spliced to obtain the crop multi-modal daily growth data representation.

[0021] In a second aspect, the present application provides a crop yield prediction device based on multi-modal information fusion, comprising:

[0022] A crop multi-modal data acquisition unit is configured to acquire crop multi-modal data including crop daily growth data and historical annual yield data;

[0023] A model establishment and prediction unit is configured to establish a crop yield prediction model and use the crop yield prediction model to predict crop yield, the model comprising a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module, and a crop yield prediction module, wherein:

[0024] The representation module is configured to represent the inputted crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation;

[0025] The first modeling module is configured to model the multi-modal daily growth data representation for consecutive days;

[0026] The first fusion module is configured to fuse the output of the first modeling module after modeling to obtain an overall representation of multi-modal crop daily growth time series data;

[0027] The second fusion module is configured to fuse the overall representation of multi-modal crop daily growth time series data and the yield data representation of the historical annual yield to obtain a fused overall annual data representation;

[0028] The second modeling module is configured to model the overall annual data representation;

[0029] The crop yield prediction module is configured to predict crop yield according to the output of the second modeling module, and obtain a target crop yield prediction result.

[0030] A further technical solution of the present application is that the first modeling module / the second modeling module is any one of LSTM and Transformer.

[0031] A further technical solution of the present application is that the prediction method used by the crop yield prediction module is linear regression, and the loss function used is L1 loss or L2 loss.

[0032] In a third aspect of the present application, a crop yield prediction device based on multi-modal information fusion comprises a processor and a memory, wherein the memory stores computer executable programs, and when the computer executable programs are executed by the processor, the above-mentioned crop yield prediction method based on multi-modal information fusion is executed.

[0033] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores instructions, and when the instructions are executed by a processor, the processor executes the above-mentioned crop yield prediction method based on multi-modal information fusion.

[0034] The present application provides a crop yield prediction method, device and computer storage medium based on multi-modal information fusion, which acquires crop multi-modal data including crop daily growth data and historical annual yield data, establishes a crop yield prediction model and uses the crop yield prediction model to predict crop yield, the model comprises a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module and a crop yield prediction module, and the present application has the following beneficial effects:

[0035] (1) The existing crop yield prediction method does not consider the fusion of crop daily growth image data and structured data such as soil and weather, the present application is a supplement to the data used by the current crop yield prediction method based on multi-modal data fusion, and provides a crop yield prediction method and device based on multi-modal data fusion;

[0036] (2) The first modeling module is configured to model the multi-modal daily growth data representation for consecutive days, which can more finely depict the crop growth trend and better improve the accuracy of the model compared with the past only considering historical annual data;

[0037] (3) The second fusion module is used for fusing the overall representation of the multi-modal crop daily growth time series data and the yield data representation of the historical year, and in the specific implementation process, the crop growth trend representation, the historical yield representation of other crops in the region and the historical yield representation of the target crop are fused. In the annual data fusion process, the attention mechanism can be used to model the correlation between the crop growth trend and the historical yield of different crops, further improving the model progress. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of a crop yield prediction method based on multi-modal information fusion in an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of establishing a crop yield prediction model in an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a crop yield prediction device based on multi-modal information fusion in an embodiment of the present application;

[0041] Figure 4 is the architecture of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0043] Before discussing the example embodiments in more detail, it is important to note that some of the example embodiments are described as processes or methods depicted as flow diagrams. Although the processes are described in a particular sequential order, many of the steps can be performed in parallel, concurrently or in any order. In addition, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but the processes can also have additional steps not included in the figure, which can also be performed after the processes are terminated. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0044] The embodiments of the present application are directed to a crop yield prediction method, device and computer storage medium based on multi-modal information fusion, which provides the following embodiments:

[0045] Based on the embodiment 1 of the present application

[0046] Reference Figure 1 is a flowchart of a crop yield prediction method based on multi-modal information fusion, the method comprising the following steps:

[0047] S110, obtain crop multi-modal data including daily growth data of crops and historical annual yield data of crops;

[0048] In the implementation process, the crop multi-modal data includes crop growth image, soil data, weather data, crop area, crop historical yield, etc.; the crop growth image can be obtained through a certain number of fixed position cameras; the soil data includes clay / silt / sand percentage, effective water content, pH value, organic matter content, soil saturated hydraulic conductivity, temperature, etc., which can be obtained through a soil moisture monitor; the weather data includes day length, precipitation, solar radiation, maximum temperature, minimum temperature, average temperature, etc., which can be obtained through a meteorological data collection instrument; the crop area and the crop historical yield can be historical record data, which can be directly obtained. The embodiments collect the crop growth image, the soil data and the weather data in units of days, and collect the crop area and the crop historical yield in units of years. Both the daily time series data such as the crop growth image, the soil data and the weather data and the historical yield of different crops in the region are considered; the crop yield influencing factors are modeled from different granularities and levels, which can effectively predict the crop yield.

[0049] S120, referring to Figure 2 , a crop yield prediction model is established and the crop yield prediction model is used for crop yield prediction, the model includes a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module and a crop yield prediction module, wherein:

[0050] The representation module is used for representing the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation;

[0051] e=f(x)

[0052] The first modeling module is used for modeling the multi-modal daily growth data representation of consecutive multiple days;

[0053] h=Encoder1(e1,e2,…,en)

[0054] Preferably, the first modeling module can be any one of LSTM and Transformer.

[0055] The first fusion module is used for fusing the output of the first modeling module after modeling to obtain an overall representation of multi-modal crop daily growth time series data;

[0056] In the implementation process, the network of pooling and attention is used for fusion to obtain the overall representation of the multi-modal crop daily growth time series data.

[0057] h 1 =Fusion1(h)

[0058] The second fusion module is configured to fuse the overall representation of the multi-modal crop daily growth time series data and the yield data d of the historical year to obtain the overall representation of the fused annual data.

[0059] h 2 =Fusion2([h 1 ,d])

[0060] Preferably, the second fusion module is a self-attention network.

[0061] The second modeling module is configured to model the overall representation of the annual data.

[0062] h 3 =Encoder2(h 2 )

[0063] Preferably, the second modeling module is any one of LSTM and Transformer.

[0064] The crop yield prediction module is configured to predict the crop yield according to the output of the second modeling module to obtain a target crop yield prediction result.

[0065] m=g θ (h 3 )

[0066]

[0067] Preferably, the prediction method used by the crop yield prediction module is linear regression, and the loss function used is L1 loss or L2 loss.

[0068] In the implementation process, the representation module quantifies the obtained daily soil data and weather data, and the historical annual yield data to obtain the corresponding representation, and the representation module represents the input multi-modal crop data, specifically including:

[0069] The obtained crop daily growth data X 1 is encoded by ResNet or DenseNet, and the encoded result is average-pooled;

[0070]

[0071] The crop fruit X is detected by a YOLO series model 2 The fruit detection result is detected and quantified.

[0072]

[0073] The average-pooled output result and the quantified output result are spliced to obtain the multi-modal daily growth data representation of the crop.

[0074]

[0075] Based on the embodiment 2 of the present application

[0076] The crop yield prediction device 300 provided by the embodiment 2 of the present application can execute the crop yield prediction method based on multi-modal information fusion provided by the embodiment 1 of the present application, has the corresponding function modules and beneficial effects of the execution method, and can be realized by software and / or hardware (integrated circuit), and can be generally integrated in a server or a terminal device. Figure 3 FIG. 1 is a structural schematic diagram of the crop yield prediction device 300 based on multi-modal information fusion in the embodiment 2 of the present application. Referring to FIG. 1, Figure 3 The crop yield prediction device 300 based on multi-modal information fusion in the embodiment of the present application can specifically include:

[0077] The crop multi-modal data acquisition unit 310 is configured to acquire crop multi-modal data including daily growth data and historical annual yield data of the crop;

[0078] The model establishment and prediction unit 320 is configured to establish a crop yield prediction model and perform crop yield prediction by using the crop yield prediction model, and the model includes a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module, and a crop yield prediction module, wherein:

[0079] The representation module is configured to represent the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation;

[0080] The first modeling module is configured to model the multi-modal daily growth data representation for consecutive multiple days;

[0081] The first fusion module is configured to fuse the output after modeling of the first modeling module to obtain an overall representation of multi-modal crop daily growth time series data;

[0082] The second fusion module is used to fuse the overall representation of the daily growth time series data of the multimodal crops and the historical annual yield data representation to obtain the overall representation of the fused annual data.

[0083] The second modeling module is used to model the overall representation of the annual data;

[0084] The crop yield prediction module is used to predict crop yield based on the output of the second modeling module, and obtain the target crop yield prediction result.

[0085] Furthermore, the first modeling module / the second modeling module is either LSTM or Transformer.

[0086] Furthermore, the crop yield prediction module uses linear regression as its prediction method and employs either L1 loss or L2 loss as its loss function.

[0087] In addition to the units described above, the crop yield prediction device 300 based on multimodal information fusion may also include other components. However, since these components are not related to the content of this disclosure, their illustrations and descriptions are omitted here.

[0088] The specific working process of the crop yield prediction device 300 based on multimodal information fusion is described in Example 1 of the crop yield prediction method based on multimodal information fusion, and will not be repeated here.

[0089] Based on Embodiment 3 of the present invention

[0090] The system according to embodiments of the present invention can also be used by means of Figure 4 The architecture of the computing device shown is used to implement this. Figure 4 The architecture of the computing device is shown. For example... Figure 4 As shown, the computer system 410 includes a system bus 430, one or more CPUs 440, input / output 420, and memory 450. The memory 450 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the method of Embodiment 1. Figure 4 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 4one or more components in the server 400. The memory 450, as a kind of computer readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the crop yield prediction method based on multi-modal information fusion in the embodiments of the present application (for example, the crop multi-modal data acquisition unit 310 and the model establishing and prediction unit 320 in the crop yield prediction device 300 based on multi-modal information fusion). The one or more CPUs 440 perform various functional applications and data processing of the system of the present application by running the software programs, instructions and modules stored in the memory 450, that is, implement the crop yield prediction method based on multi-modal information fusion as described above, which includes the following steps:

[0091] acquiring crop multi-modal data including daily growth data and historical annual yield data of crops;

[0092] establishing a crop yield prediction model and using the crop yield prediction model to predict the yield of crops, the model including a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module and a crop yield prediction module, wherein:

[0093] The representation module is configured to represent the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation;

[0094] The first modeling module is configured to model the multi-modal daily growth data representation for consecutive multiple days;

[0095] The first fusion module is configured to fuse the output of the first modeling module after modeling to obtain an overall representation of multi-modal crop daily growth time series data;

[0096] The second fusion module is configured to fuse the overall representation of multi-modal crop daily growth time series data and the yield data representation of the historical annual yield to obtain a fused overall annual data representation;

[0097] The second modeling module is configured to model the overall annual data representation;

[0098] The crop yield prediction module is configured to predict the yield of crops according to the output of the second modeling module after modeling to obtain a target crop yield prediction result.

[0099] Of course, the processor of the server provided in the embodiments of the present application is not limited to performing the method operations as described above, but can also perform the related operations in the crop yield prediction method based on multi-modal information fusion provided in any embodiments of the present application.

[0100] The memory 450 can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required for at least one function, and the data storage area can store data created according to a use of the terminal, etc. Furthermore, the memory 450 can include a high-speed random access memory and also include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-volatile solid state storage device. In some examples, the memory 450 can further include a memory disposed remotely from the one or more CPUs 440, which can be connected to the device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0101] The input / output 420 can be used to receive inputted digital or character information, and to generate key signal inputs related to a user setting and a function control of the device. The input / output 420 can further include a display device such as a display screen.

[0102] Based on Embodiment 4 of the present application

[0103] The embodiments of the present application can also be implemented as a computer readable storage medium. The computer readable storage medium according to Embodiment 4 has stored thereon a computer program. When the computer program is executed by a processor, the method for predicting crop yield based on multi-modal information fusion according to Embodiment 1 of the present application described with reference to the above drawings can be performed.

[0104] Of course, the storage medium provided by the embodiments of the present application includes computer executable instructions, which are not limited to the method operations described above, but can also perform related operations in the method for predicting crop yield based on multi-modal information fusion provided by any embodiments of the present application.

[0105] The computer readable storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0106] Computer readable signal media can include a propagated data signal with computer readable program code embodied therein. For example, a propagated signal can be an electromagnetic signal, an optical signal, and / or the like. Such a propagated signal can take a variety of forms, including, but not limited to, electro-magnetic signals, optical signals, and / or the like. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device.

[0107] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0108] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In an embodiment, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry, includes the circuitry required to produce and transmit identifying information to a remote computer.

[0109] The crop yield prediction method and device based on multi-modal information fusion and the computer storage medium provided by the above embodiments comprehensively have the beneficial effects that the existing crop yield prediction methods do not consider fusing crop daily growth image data and structured data such as soil and weather, the work of the present application based on multi-modal data fusion is a supplement to the data used by the current crop yield prediction method, and a crop yield prediction method and device based on multi-modal data fusion are provided; the first modeling module is used for modeling the multi-modal daily growth data representation of consecutive days, compared with the past only considering historical annual data, modeling the multi-modal daily growth data representation of consecutive days can more finely depict the crop growth trend and better improve the accuracy of the model; the second fusion module is used for fusing the overall representation of the multi-modal crop daily growth time series data and the yield data representation of the historical year, and in the specific implementation process, the crop growth trend representation, the historical yield representation of other crops in the region and the historical yield representation of the target crop are fused, and in the annual data fusion process, the use of the attention mechanism can model the correlation between the crop growth trend and the historical yield of different crops, and further improve the model progress.

[0110] It should be noted that the above are only the preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1.A crop yield prediction method based on multi-modal information fusion, characterized in that, The method comprises the following steps: obtaining crop multi-modal data including crop daily growth data and historical annual yield data; establishing a crop yield prediction model and using the crop yield prediction model to predict crop yield, the model comprising a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module, and a crop yield prediction module, wherein: the representation module is used to represent the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation; the first modeling module is used to model the multi-modal daily growth data representation for consecutive days; the first fusion module is used to fuse the output of the first modeling module after modeling to obtain an overall representation of multi-modal crop daily growth time series data; the second fusion module is used to fuse the overall representation of multi-modal crop daily growth time series data and the yield data representation of the historical year to obtain a fused overall annual data representation; the second modeling module is used to model the overall annual data representation; the crop yield prediction module is used to predict crop yield according to the output of the second modeling module after modeling to obtain a target crop yield prediction result; the representation module represents the input crop multi-modal data, specifically comprising: encoding the obtained crop daily growth data through ResNet or DenseNet and performing average pooling on the encoded results; detecting crop fruits through a YOLO series model and quantifying the fruit detection results numerically; splicing the output results after average pooling and the output results after numerical quantification to obtain crop multi-modal daily growth data representation. 2.The crop yield prediction method based on multi-modal information fusion according to claim 1, characterized in that, The first modeling module or the second modeling module is any one of LSTM and Transformer. 3.The crop yield prediction method based on multi-modal information fusion according to claim 1, characterized in that, The second fusion module is a self-attention network. 4.The crop yield prediction method based on multi-modal information fusion according to claim 1, characterized in that, The prediction method used by the crop yield prediction module is linear regression, and the loss function used is L1 loss or L2 loss. 5.A crop yield prediction device based on multi-modal information fusion, characterized in that, comprises: a crop multi-modal data acquisition unit for acquiring crop multi-modal data including crop daily growth data and historical annual yield data; a model establishment and prediction unit for establishing a crop yield prediction model and using the crop yield prediction model to predict crop yield, the model comprising a representation module, a first modeling module, a first fusion module, a second fusion module, a second modeling module, and a crop yield prediction module, wherein: the representation module is used to represent the input crop multi-modal data to obtain multi-modal daily growth data representation and historical annual yield data representation; the first modeling module is used to model the multi-modal daily growth data representation for consecutive days; the first fusion module is used to fuse the output of the first modeling module after modeling to obtain an overall representation of multi-modal crop daily growth time series data; The second fusion module is configured to fuse the overall representation of the multi-modal crop daily growth time series data and the yield data representation of the historical year to obtain a fused overall representation of the annual data; The second modeling module is configured to model the overall representation of the annual data; The crop yield prediction module is configured to predict crop yield according to the output of the second modeling module to obtain a target crop yield prediction result; The representation module represents the input multi-modal crop data, specifically including: The obtained crop daily growth data is encoded by ResNet or DenseNet, and the encoded results are average-pooled; YOLO series model is used for crop fruit detection, and the fruit detection results are numerically quantified; The average-pooled output results and the numerically quantified output results are spliced to obtain the multi-modal crop daily growth data representation. 6.The crop yield prediction device based on multi-modal information fusion according to claim 5, wherein, The first modeling module or the second modeling module is any one of LSTM and Transformer. 7.The crop yield prediction device based on multi-modal information fusion according to claim 5, wherein, The prediction method used by the crop yield prediction module is linear regression, and the loss function used is L1 loss or L2 loss. 8.A crop yield prediction device based on multi-modal information fusion, characterized by, comprise: a processor; and a memory, wherein the memory stores computer executable programs, and when the computer executable programs are executed by the processor, the multi-modal information fusion based crop yield prediction method in any one of claims 1-4 is executed. 9.A computer readable storage medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to perform the multi-modal information fusion based crop yield prediction method in any one of claims 1-4.

Images