Rape development period identification method and device based on multi-modal big data

Abstract

The embodiment of the invention provides a rape development period identification method and device based on multi-modal big data, and is applied to the technical field of crop identification. The method comprises the following steps: acquiring an image data set and a modal data set of rape; preprocessing the image data set and the modal data set; inputting the preprocessed image data set into a preset image feature extraction model to output image features, and inputting the preprocessed modal data set into a preset text extraction model to output semantic features; inputting the image features and the semantic features into a preset multi-modal feature fusion model to output fusion features; and inputting the fusion features into a preset developmental period identification model to output the rape developmental period. In this way, the recognition accuracy of the developmental periods such as the squaring period of the rape and the like with unobvious image features can be effectively improved, and accurate recognition of the whole developmental period of the rape is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of crop identification technology, and specifically to a method and device for identifying the development period of rapeseed based on multimodal big data. Background Art

[0002] Rapeseed generally buds before bolting, but for some varieties or under certain cultivation conditions, rapeseed bolts before buds, or buds and bolts occur simultaneously. Currently, most of the recognition of rapeseed development period uses deep learning classification algorithms, which train a single modality recognition model based on the image features of rapeseed development period to identify the crop development period. However, the above technology is mainly based on image feature data, and the recognition accuracy is low for the development period when the image features of rapeseed bud period are not obvious. Summary of the invention

[0003] The present invention provides a method and device for identifying the development period of rapeseed based on multimodal big data.

[0004] According to a first aspect of the present disclosure, a method for identifying the developmental stage of rapeseed based on multimodal big data is provided. The method comprises:

[0005] Obtain rapeseed image dataset and modality dataset;

[0006] Preprocessing the image dataset and the modality dataset;

[0007] The preprocessed image dataset is input into a preset image feature extraction model to output image features, and the preprocessed modal dataset is input into a preset text extraction model to output semantic features;

[0008] Inputting the image features and the semantic features into a preset multimodal feature fusion model to output fusion features;

[0009] The fusion feature is input into a preset developmental stage recognition model to output the rapeseed developmental stage.

[0010] According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein preprocessing the image data set includes:

[0011] The image data set is cleaned based on a preset resolution, a preset size, and a preset definition.

[0012] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein preprocessing the modality data set includes:

[0013] Based on limit value checking and variability checking, data cleaning is performed on the modal data set.

[0014] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the preset image feature extraction model is used to extract image features related to the rapeseed development period in an image data set, and the image features include color, contour, height, shape and texture.

[0015] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the preset text extraction model is used to extract semantic features related to the rapeseed development period in the modal dataset, and the semantic features include plant height, accumulated temperature, hyperspectral spectrum and soil pH.

[0016] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the preset multimodal feature fusion model includes at least two fully connected layers and at least two vector concatenation layers.

[0017] According to the aspects and any possible implementations described above, there is further provided an implementation, wherein the rapeseed development period includes sowing, emergence, five true leaves, budding, bolting, flowering, green maturity and maturity.

[0018] According to a second aspect of the present disclosure, a device for identifying the growth period of rapeseed based on multimodal big data is provided. The device comprises:

[0019] An acquisition module, used to acquire an image dataset and a modality dataset of rapeseed;

[0020] A preprocessing module, used for preprocessing the image dataset and the modality dataset;

[0021] An extraction module, used to input the preprocessed image data set into a preset image feature extraction model to output image features, and input the preprocessed modal data set into a preset text extraction model to output semantic features;

[0022] A fusion module, used for inputting the image features and the semantic features into a preset multimodal feature fusion model to output fusion features;

[0023] The recognition module is used to input the fusion feature into a preset development period recognition model to output the rapeseed development period.

[0024] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0025] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0026] The embodiment of the present application provides a method and device for identifying the development period of rapeseed based on multimodal big data, which can obtain an image data set and a modal data set of rapeseed; then preprocess the image data set and the modal data set; then input the preprocessed image data set into a preset image feature extraction model to output image features, and input the preprocessed modal data set into a preset text extraction model to output semantic features; then input the image features and semantic features into a preset multimodal feature fusion model to output fusion features; then input the fusion features into a preset development period recognition model to output the development period of rapeseed; based on this, multimodal data feature fusion recognition technology is used to fuse spectral data, crop growth data, etc., to extract spectral features and growth features related to the development period of rapeseed, thereby effectively improving the recognition accuracy of development periods such as the budding period where image features are not obvious, and achieving accurate recognition of the entire development period of rapeseed.

[0027] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0029] Figure 1 A flowchart of a method for identifying rapeseed development stages based on multimodal big data according to an embodiment of the present disclosure is shown;

[0030] Figure 2 A schematic diagram of a multimodal big data acquisition device according to an embodiment of the present disclosure is shown;

[0031] Figure 3 A schematic diagram of a multimodal deep learning algorithm model according to an embodiment of the present disclosure is shown;

[0032] Figure 4 A block diagram of a rapeseed development period identification device based on multimodal big data according to an embodiment of the present disclosure is shown;

[0033] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0036] In the present disclosure, multimodal data feature fusion recognition technology is used to fuse spectral data, crop growth data, etc., to extract spectral features and growth features related to the rapeseed development period, thereby effectively improving the recognition accuracy of rapeseed development periods such as the budding period where image features are not obvious, and achieving accurate recognition of rapeseed throughout the entire development period.

[0037] Figure 1 A flowchart of a method 100 for identifying rapeseed development stages based on multimodal big data according to an embodiment of the present disclosure is shown.

[0038] In block 110 , an image dataset and a modality dataset of rapeseed are acquired.

[0039] At block 120 , the image dataset and the modality dataset are preprocessed.

[0040] In box 130, the preprocessed image dataset is input into a preset image feature extraction model to output image features, and the preprocessed modality dataset is input into a preset text extraction model to output semantic features.

[0041] In block 140 , the image features and semantic features are input into a preset multimodal feature fusion model to output fused features.

[0042] In block 150 , the fused features are input into a preset developmental stage recognition model to output the rapeseed developmental stage.

[0043] In block 110 , an image dataset and a modality dataset of rapeseed may be collected by a multimodal big data collection device.

[0044] like Figure 2As shown in the figure, the multimodal big data collection equipment includes standard equipment and optional equipment. Among them, the standard equipment includes crop spectrometer, crop growth (upper) spherical camera, canopy height observation instrument, canopy temperature observation instrument, total radiation collector, two-dimensional ultrasonic wind and optical rainfall observation instrument, regional profile soil moisture sensor, soil temperature and humidity profile sensor, 3-layer air temperature and humidity sensor, crop growth (lower) gun-type camera, tipping bucket rainfall observation instrument, 2-layer air temperature and humidity sensor and 1-layer air temperature sensor. Optional equipment includes soil pH, soil organic matter, soil conductivity and soil nutrient content sensors, evaporation observation instrument, soil freeze-thaw sensor and carbon dioxide collector.

[0045] In some embodiments, the image data set of rapeseed can be collected by a spherical camera of crop growth (upper) at camera position 1 and a gun-type camera of crop growth (lower) at camera position 2 in a multimodal big data collection device.

[0046] In some embodiments, the modal data set can be collected through the canopy height observation instrument, canopy temperature observation instrument, crop spectrometer and soil pH sensor in the multimodal big data collection equipment. Specifically, the plant height data is collected through the canopy height observation instrument, the accumulated temperature data is collected through the canopy temperature observation instrument, the hyperspectral data is collected through the crop spectrometer, and the soil pH data is collected through the soil pH sensor.

[0047] In block 120 , data preprocessing may be performed on the image dataset and the modality dataset according to their respective data features.

[0048] In some embodiments, preprocessing the image dataset includes:

[0049] The image data set is cleaned based on the preset resolution, preset size and preset clarity.

[0050] In some embodiments, the preset resolution, preset size, and preset definition can be set according to actual needs of the user.

[0051] In some embodiments, the image data set is cleaned based on a preset resolution, preset size, and preset clarity to eliminate images with low resolution, small size, low clarity, damage, or not yet entered the developmental stage, and select high-quality image data.

[0052] In some embodiments, the basis for cleaning the image data set can refer to Table 1.

[0053] Table 1: Image data cleaning basis

[0054]

[0055]

[0056] In some embodiments, preprocessing the modality dataset includes:

[0057] Data cleaning is performed on the modal data set based on limit value checking and variability checking.

[0058] In some embodiments, for modal data such as plant height, accumulated temperature, hyperspectral spectrum, soil pH, etc., limit value checks and variability checks can be used to perform quality control to eliminate outliers and retain valid samples.

[0059] In some embodiments, the basis for cleaning plant height data can refer to Table 2.

[0060] Table 2: Plant height data quality control basis

[0061]

[0062] In some embodiments, the basis for cleaning the hyperspectral data may refer to that shown in Table 3.

[0063] Table 3: Hyperspectral data quality control basis

[0064]

[0065] In some embodiments, the basis for cleaning soil pH data can refer to Table 4.

[0066] Table 4: Soil pH data quality control basis

[0067]

[0068] In some embodiments, for the accumulated temperature data, data below 0°C may be excluded, and data above 0°C may be used to accumulate and calculate the effective accumulated temperature of rapeseed at each development stage.

[0069] In some embodiments, preprocessing the image data set and the modal data set may also include: selecting images and modal features based on spatiotemporal consistency, and screening out and pairing image data and modal data such as plant height, accumulated temperature, and spectral features at the same time and space.

[0070] In box 130, the preset image feature extraction model and the preset text extraction model can be trained according to the actual needs of the user.

[0071] In some embodiments, the preset image feature extraction model is used to extract image features related to the development period of rapeseed in an image dataset, and the image features include color, contour, height, shape and texture.

[0072] In some embodiments, the preset text extraction model is used to extract semantic features related to the rapeseed development period in the modal dataset, and the semantic features include plant height, accumulated temperature, hyperspectral spectrum and soil pH.

[0073] In box 140, the preset multimodal feature fusion model can be trained according to the actual needs of the user.

[0074] In some embodiments, the preset multimodal feature fusion model includes at least two fully connected layers and at least two vector concatenation layers.

[0075] In box 150, the preset developmental stage recognition model can be trained according to the actual needs of the user.

[0076] In some embodiments, the rapeseed development stages include sowing, emergence, five true leaves, budding, bolting, flowering, green maturity and maturity.

[0077] like Figure 3 As shown, the preset image feature extraction model, the preset text extraction model, the preset multimodal feature fusion model and the preset developmental period recognition model can be jointly constructed into a multimodal deep learning algorithm model to identify the developmental period of rapeseed.

[0078] Specifically, in the multimodal deep learning algorithm model, the image dataset is input into the image feature extraction module after quality control processing to output image features. The hyperspectral dataset, plant height dataset, and accumulated temperature dataset are input into the text feature extraction module after quality control processing to output semantic features. The image features and semantic features are input into the multimodal feature fusion module to output fusion features. The fusion features are input into the classifier to output the rapeseed development period.

[0079] In some embodiments, the preset image feature extraction model can select a neural network model based on the Transformer architecture, such as a neural network model (Visual Transformer, ViT) used to process computer vision tasks.

[0080] In some embodiments, ViT can be used to extract rapeseed development stage features in an image, such as color, contour, height, shape, texture features, etc.

[0081] Specifically, ViT can include 1 convolution (Conv) layer, 1 class encoding (class-embedding) layer, 1 position encoding (positional-embedding) layer, 2 normalization (LayerNorm) layers, 1 Transformer layer, and 1 parameter learning (parameter) layer. First, the input image, that is, each image in the preprocessed image dataset, is scaled to a width and height of 224 pixels. ViT divides the input image into 49 feature blocks (patches) with a width and height of 16 pixels each through a convolutional layer. The category encoding layer uses random numbers to perform initial category encoding on the 49 feature blocks. The position encoding layer uses random numbers to perform initial position encoding on the 49 feature blocks. At the same time, a starting position encoding is added in front of the first feature block to output 50 encoded feature blocks. A normalization layer is used to normalize all feature blocks, and the normalized features are input into the multi-head attention mechanism Transformer layer to extract features. Then, a normalization layer and a parameter learning layer are used to obtain 512-dimensional key developmental features including color, contour, height, shape, texture, etc.

[0082] In some embodiments, the preset text extraction model may select a text encoder, such as a text-encoder.

[0083] In some embodiments, a text-encoder can be used to extract semantic features related to the rapeseed development stage, such as plant height, accumulated temperature, hyperspectral spectrum, and soil pH.

[0084] Specifically, the text-encoder includes 4 token-embedding layers, 4 positional-embedding layers, 4 Transformer layers, and 4 fully connected layers. The plant height data, accumulated temperature data, hyperspectral data, and soil pH data of rapeseed at different development stages after quality control are input into the text-encoder respectively. Each type of data uses 1 token-embedding layer, 1 positional-embedding layer, 1 Transformer layer, and 1 fully connected layer to obtain 512-dimensional developmental semantic features.

[0085] In some embodiments, the preset multimodal feature fusion model can select a multiplication-addition convolutional neural network model (Multi Add).

[0086] In some embodiments, Multi Add includes two fully connected layers and two multidimensional vector concatenation layers to fuse multimodal features; the 512-dimensional features extracted by ViT output 256-dimensional features through a fully connected layer, the text-encoder concatenates them into 2048-dimensional features through a multidimensional vector concatenation layer, and then outputs 256-dimensional features through a fully connected layer, and the two 256-dimensional features are concatenated into 512-dimensional features through a multidimensional vector concatenation layer.

[0087] In some embodiments, the preset developmental stage recognition model can use the fused features to train a developmental stage identifier, which includes a fully connected layer and a softmax classification layer. The 512-dimensional features output by Multi Add are used to output 128-dimensional features through a fully connected layer, and the 128-dimensional features are used to output the developmental stage code of rapeseed through a softmax classification layer, and the specific developmental stage is determined based on the code.

[0088] In some embodiments, the training process of the multimodal deep learning algorithm model may include:

[0089] The image data, accumulated temperature data, hyperspectral data, and soil pH data of rapeseed during the development period are collected through multimodal big data acquisition equipment. After quality control, a high-quality data set is obtained. At the same time, the rapeseed development period represented by the above data can be annotated in combination with the relevant background of agricultural meteorological observation. The eight development periods of sowing, emergence, five true leaves, budding, bolting, flowering, green maturity, and maturity are annotated, and the development period is coded from 0 to 7. Then, the data set is divided into training set, validation set, and test set in a ratio of 8:1:1. Iterative training is performed on the training set, with 1000 iterations. After each training of 100 times, the recognition accuracy is evaluated on the validation set, and the model parameters with the highest recognition accuracy are saved. After the training is completed, the recognition accuracy of the model is evaluated on the test set. If the average recognition accuracy is greater than 90%, the business requirements are met. If not, the hyperparameters of the model (learning rate, batch size, and number of iterations) are tuned on the test data set to achieve a recognition accuracy of more than 90%.

[0090] According to the embodiments of the present disclosure, the following technical effects are achieved:

[0091] The method can obtain an image dataset and a modal dataset of rapeseed; preprocess the image dataset and the modal dataset; input the preprocessed image dataset into a preset image feature extraction model to output image features, and input the preprocessed modal dataset into a preset text extraction model to output semantic features; input the image features and semantic features into a preset multimodal feature fusion model to output fusion features; input the fusion features into a preset developmental period recognition model to output the developmental period of rapeseed; based on this, use multimodal data feature fusion recognition technology to fuse spectral data, crop growth data, etc., to extract spectral features and growth features related to the developmental period of rapeseed, thereby effectively improving the recognition accuracy of developmental periods such as the budding period when image features are not obvious, and achieving accurate recognition of the entire developmental period of rapeseed.

[0092] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0093] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0094] Figure 4 FIG. 4 is a block diagram of a rapeseed growth period identification device 400 based on multimodal big data according to an embodiment of the present disclosure. Figure 4 As shown, the device 400 includes:

[0095] An acquisition module 410 is used to acquire an image dataset and a modality dataset of rapeseed;

[0096] A preprocessing module 420, configured to preprocess the image dataset and the modality dataset;

[0097] An extraction module 430 is used to input the preprocessed image data set into a preset image feature extraction model to output image features, and input the preprocessed modal data set into a preset text extraction model to output semantic features;

[0098] A fusion module 440 is used to input image features and semantic features into a preset multimodal feature fusion model to output fusion features;

[0099] The recognition module 450 is used to input the fusion features into a preset development period recognition model to output the rapeseed development period.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0101] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0103] Figure 5 A block diagram of an exemplary electronic device 500 capable of implementing an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] The electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM 502 or a computer program loaded from a storage unit 508 into a RAM 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O interface 505 is also connected to the bus 504.

[0105] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508.

[0107] In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method 100 in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for identifying rapeseed growth stages based on multimodal big data, characterized in that: include: Obtain rapeseed image dataset and modality dataset; Preprocessing the image dataset and the modality dataset; The preprocessed image dataset is input into a preset image feature extraction model to output image features, and the preprocessed modal dataset is input into a preset text extraction model to output semantic features; Inputting the image features and the semantic features into a preset multimodal feature fusion model to output fusion features; The fusion feature is input into a preset developmental stage recognition model to output the rapeseed developmental stage.

2. The method according to claim 1, characterized in that Preprocessing the image data set includes: The image data set is cleaned based on a preset resolution, a preset size, and a preset definition.

3. The method according to claim 1, characterized in that: Preprocessing the modal data set includes: Based on limit value checking and variability checking, data cleaning is performed on the modal data set.

4. The method according to claim 1, characterized in that: The preset image feature extraction model is used to extract image features related to the rapeseed development period in an image data set, and the image features include color, contour, height, shape and texture.

5. The method according to claim 1, characterized in that The preset text extraction model is used to extract semantic features related to the rapeseed development period in the modal data set, and the semantic features include plant height, accumulated temperature, hyperspectral spectrum and soil pH.

6. The method according to claim 1, characterized in that The preset multimodal feature fusion model includes at least two fully connected layers and at least two vector concatenation layers.

7. The method according to any one of claims 1 to 6, characterized in that: The rapeseed development period includes sowing, emergence, five true leaves, budding, bolting, flowering, green maturity and maturity.

8. A rapeseed development period identification device based on multimodal big data, characterized in that: include: An acquisition module, used to acquire an image dataset and a modality dataset of rapeseed; A preprocessing module, used for preprocessing the image dataset and the modality dataset; An extraction module, used to input the preprocessed image data set into a preset image feature extraction model to output image features, and input the preprocessed modal data set into a preset text extraction model to output semantic features; A fusion module, used for inputting the image features and the semantic features into a preset multimodal feature fusion model to output fusion features; The recognition module is used to input the fusion feature into a preset development period recognition model to output the rapeseed development period.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images