Crop phenotype data storage and retrieval method and device, equipment and storage medium

Abstract

The invention relates to the field of data storage, and provides a crop phenotype data storage and retrieval method and device, equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data comprising crop phenotype data and metadata information; storing the structured data and the metadata information to a distributed database, and storing the unstructured data to a distributed file system; synchronously storing a retrieval field of the structured data, a retrieval field of the small file data and metadata information of the large file data into a distributed search analysis engine; and under the condition that the data matched with the user retrieval condition does not exist in the memory database, determining a retrieval result based on the user retrieval condition and the distributed search analysis engine. According to the method, the multi-modal crop phenotype data are classified and stored according to different data types, so that the storage efficiency and the retrieval rate of the multi-modal crop phenotype data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a crop phenotypic data storage and retrieval method, device, equipment and storage medium. Background Art

[0002] Crop phenotypic data is large in volume, diverse in types, with varying storage requirements and access characteristics. Existing centralized storage and retrieval methods suffer from data storage and retrieval issues such as low storage efficiency, slow retrieval speeds, and difficulty ensuring data consistency when handling large-scale and diverse crop phenotypic data. Therefore, achieving efficient storage and fast retrieval of crop phenotypic data has become a pressing technical challenge. Summary of the Invention

[0003] The present invention provides a crop phenotypic data storage and retrieval method, device, equipment and storage medium, which are used to solve the problems of low storage efficiency and slow retrieval speed in existing data storage and retrieval methods.

[0004] The present invention provides a crop phenotypic data storage and retrieval method, which is applied to a hybrid architecture storage and retrieval system; the hybrid architecture storage and retrieval system includes a distributed database, a distributed search analysis engine, a distributed file system, and an in-memory database; the method includes: Acquiring multimodal data to be stored; the multimodal data including crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data including structured data and unstructured data; the unstructured data including small file data and large file data; Storing the structured data and the metadata information in the distributed database, and storing the unstructured data in the distributed file system; Synchronously storing the search fields of the structured data, the search fields of the small file data, and the metadata information of the large file data in the distributed search analysis engine; When there is no data matching the user's search conditions in the memory database, a search result is determined based on the user's search conditions and the distributed search analysis engine; the memory database is used to cache metadata information of the structured data and the unstructured data.

[0005] According to a crop phenotypic data storage and retrieval method provided by the present invention, the metadata information of the structured data includes a crop identifier and other crop information; and storing the structured data and the metadata information in the distributed database includes: Filling a first row key and a first column family into a first data table of the distributed database; the first row key is determined based on the crop identifier; and the first column family is determined based on the structured data and the other crop information.

[0006] According to a crop phenotypic data storage and retrieval method provided by the present invention, storing the unstructured data in the distributed file system includes: Traverse the unstructured data to be stored and obtain the file size of the currently uploaded file; If the file size is greater than a first threshold, compress the currently uploaded file and store it in the distributed file system; if the file size is less than the first threshold and greater than a second threshold, place the currently uploaded file in an empty merge queue; if the file size is less than or equal to the second threshold, place the currently uploaded file in a temporary queue; Selecting the largest file from the temporary queue and determining the maximum remaining space in each of the merged queues; If the largest file is larger than the maximum remaining space, the largest file is placed in a merge queue converted from the standby queue; if the largest file is smaller than or equal to the maximum remaining space, the largest file is placed in a merge queue corresponding to the maximum remaining space; When the files in the merge queue meet a preset condition, the files in the merge queue are merged and stored in the distributed file system.

[0007] According to a crop phenotypic data storage and retrieval method provided by the present invention, the crop phenotypic data storage and retrieval method further comprises: Establish a mapping relationship between the file before the merge and the file after the merge; Filling a second row key and a second column family into a second data table of the distributed database; the second row key is determined based on the pre-merger file; and the second column family is determined based on the metadata information of the unstructured data and the mapping relationship.

[0008] According to a crop phenotypic data storage and retrieval method provided by the present invention, the retrieval fields of the structured data, the retrieval fields of the small file data, and the metadata information of the large file data are synchronously stored in the distributed search analysis engine, and then include: When the structured data is searched based on the user's search conditions and the search results exist in the memory database, the search results are loaded locally; When the structured data is searched based on the user search condition and no search result exists in the memory database, the target row key corresponding to the user search condition is determined by the distributed search analysis engine, the search result is obtained from the distributed database based on the target row key, the search result is loaded locally, and the memory database is updated based on the loaded result; In the case of retrieving the unstructured data based on the user's search conditions, metadata results are retrieved from the memory database, search data is acquired from the distributed file system based on the metadata results, and the search data is loaded locally.

[0009] According to a crop phenotypic data storage and retrieval method provided by the present invention, the crop phenotypic data storage and retrieval method further comprises: Determining access information for each cache key in the memory database; the access information includes access frequency, last access time, and current access time; Adjust the dynamic weight value of each cache key based on the access information of each cache key; The cached data in the memory database is dynamically adjusted based on the adjusted weight value.

[0010] The present invention also provides a crop phenotypic data storage and retrieval device, comprising the following modules: a multimodal data acquisition module, configured to acquire multimodal data to be stored; the multimodal data comprising crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data comprising structured data and unstructured data; the unstructured data comprising small file data and large file data; A first storage module, configured to store the structured data and the metadata information in a distributed database, and store the unstructured data in a distributed file system; A second storage module is used to synchronously store the search field of the structured data, the search field of the small file data, and the metadata information of the large file data in a distributed search analysis engine; A retrieval module is used to determine the retrieval results based on the user retrieval conditions and the distributed search analysis engine when there is no data matching the user retrieval conditions in the memory database; the memory database is used to cache the metadata information of the structured data and the unstructured data.

[0011] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-described crop phenotypic data storage and retrieval methods is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described crop phenotypic data storage and retrieval methods.

[0013] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described crop phenotypic data storage and retrieval methods.

[0014] The crop phenotypic data storage and retrieval method, apparatus, device, and storage medium provided by this invention utilize a hybrid architecture storage and retrieval system to categorize and store multimodal crop phenotypic data. Structured and unstructured crop phenotypic data are stored in separate blocks. During user searches, results are retrieved from these blocks based on the user's search criteria. By categorizing and storing multimodal crop phenotypic data according to different data types, the storage efficiency and retrieval rate of this data are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is one of the flow charts of the crop phenotypic data storage and retrieval method provided by the present invention.

[0017] Figure 2 This is the second flow chart of the crop phenotypic data storage and retrieval method provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of the crop phenotypic data storage and retrieval device provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following combination Figure 1-Figure 4 The present invention describes the crop phenotypic data storage and retrieval method, device, equipment and storage medium.

[0022] Figure 1 This is one of the flow charts of the crop phenotypic data storage and retrieval method provided by the present invention, such as Figure 1 As shown, the method is applied to a hybrid architecture storage and retrieval system; the hybrid architecture storage and retrieval system includes a distributed database, a distributed search analysis engine, a distributed file system, and an in-memory database; the crop phenotypic data storage and retrieval method provided by the present invention includes the following: Step 100: Acquire multimodal data to be stored; the multimodal data includes crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data includes structured data and unstructured data; the unstructured data includes small file data and large file data; Specifically, before storing multimodal crop phenotypic data, the data is preprocessed. The preprocessing process is as follows: the multimodal crop phenotypic data is cleaned, classified, denoised, and formatted in a standardized manner. Multimodal crop phenotypic data includes structured crop phenotypic data (e.g., sowing date, maturity date, plant height, and yield) and unstructured crop phenotypic data (e.g., images, point clouds, spectra, and thermal imaging). The multimodal data to be stored includes the aforementioned multimodal crop phenotypic data and its corresponding metadata (e.g., crop name, plant ID, and variety). The multimodal data to be stored also includes environmental data (e.g., maximum temperature, minimum temperature, sunshine duration, and precipitation). Unstructured crop phenotypic data can also be divided into small file data and large file data.

[0023] Step 200: storing the structured data and the metadata information in the distributed database, and storing the unstructured data in the distributed file system; Specifically, the crop phenotypic data storage and retrieval method provided by the present invention is applied to a hybrid architecture storage and retrieval system including a distributed database, a distributed search analysis engine, a distributed file system and an in-memory database.

[0024] The structured crop phenotypic data and metadata are stored in a distributed database (e.g., Hbase, a distributed, column-oriented open-source database). Unstructured crop phenotypic data are stored differentially based on file size. Unstructured crop phenotypic data files larger than a certain threshold are compressed and uploaded directly to a distributed file system, such as the Hadoop Distributed File System (HDFS), the core distributed file system of the Apache Hadoop (Distributed System Infrastructure) ecosystem. Unstructured crop phenotypic data files smaller than a certain threshold are merged into small files before being uploaded to the distributed file system.

[0025] Step 300: Synchronously store the search fields of the structured data, the search fields of the small file data, and the metadata information of the large file data in the distributed search analysis engine; Specifically, a distributed search and analysis engine (e.g., Elasticsearch, a distributed search and analysis engine that supports real-time data processing, full-text retrieval, and multidimensional aggregation analysis) in a hybrid architecture storage and retrieval system is used to store retrieval fields (and their corresponding row keys) of structured crop phenotypic data, retrieval fields (and their corresponding row keys) of small file data, and metadata information of large file data to achieve diversified retrieval of multimodal crop phenotypic data.

[0026] Using a real-time synchronization tool between the distributed database and the distributed search and analysis engine, search fields for structured crop phenotypic data (such as crop name, variety, sowing period, and plant ID) can be synchronized from the distributed database to the distributed search and analysis engine. The row key of the distributed database table serves as the unique identifier for the distributed search and analysis engine document (i.e., synchronizing the row key of the HBase table to Elasticsearch). This not only ensures data consistency and uniqueness but also enables multi-condition combination queries based on crop phenotypic data, thereby improving the efficiency and flexibility of data retrieval and analysis.

[0027] Step 400: If there is no data matching the user's search criteria in the memory database, determine the search results based on the user's search criteria and the distributed search analysis engine; the memory database is used to cache the structured data and metadata information of the unstructured data.

[0028] Specifically, an in-memory database (e.g., Redis) can store structured crop phenotypic data itself, as well as metadata for unstructured data. Based on the user's search criteria, if a cache hit is found in the in-memory database, the data is directly returned; if not, a diversified search is performed.

[0029] Distributed search and analysis engine diversified search: Based on the user's diversified search criteria, a list of row keys that meet the criteria is returned. Distributed database batch retrieval: Based on the row key, the complete data is loaded from the distributed database table (crop_structuring) and returned. Finally, the loaded data is stored in the in-memory database. Other search methods are described in detail below.

[0030] This embodiment builds a hybrid storage and retrieval system to categorize and store multimodal crop phenotypic data. Structured and unstructured crop phenotypic data are stored in different blocks. When a user searches, results are retrieved from different blocks based on the user's search criteria. By categorizing and storing multimodal crop phenotypic data according to different data types, the storage efficiency and retrieval speed of multimodal crop phenotypic data are improved.

[0031] In one embodiment, the metadata information of the structured data includes a crop identifier and other crop information. The crop phenotypic data storage and retrieval method provided by the embodiment of the present invention may further include: Step 210: Fill a first row key and a first column family into a first data table of the distributed database; the first row key is determined based on the crop identifier; the first column family is determined based on the structured data and the other crop information.

[0032] Specifically, the distributed database includes a structured data table (crop_structuring) and an unstructured data table (crop_unstructuring). The structured data table (i.e., the first data table in this embodiment) has the first row key consisting of the hashed crop ID (i.e., the crop identifier in this embodiment), and the first column family stores structured crop phenotypic data (e.g., sowing date, maturity date, plant height, and yield). The unstructured data table is described in detail below.

[0033] This embodiment implements classified storage of structured crop phenotypic data through a structured data table.

[0034] In one embodiment, the crop phenotypic data storage and retrieval method provided by the embodiment of the present invention may further include: Step 220: traverse the unstructured data to be stored and obtain the file size of the currently uploaded file; Step 230: If the file size is greater than a first threshold, compress the currently uploaded file and store it in the distributed file system; if the file size is less than the first threshold and greater than a second threshold, place the currently uploaded file in an empty merge queue; if the file size is less than or equal to the second threshold, place the currently uploaded file in a temporary queue; Step 240: Select the largest file from the temporary queue and determine the maximum remaining space in each of the merged queues; Step 250: If the largest file is larger than the maximum remaining space, place the largest file in a merge queue converted from the standby queue; if the largest file is smaller than or equal to the maximum remaining space, place the largest file in a merge queue corresponding to the maximum remaining space. Step 260: When the files in the merge queue meet a preset condition, merge the files in the merge queue and store them in the distributed file system.

[0035] Specifically, the storage process of small file data in unstructured crop phenotypic data is as follows.

[0036] Traverse the unstructured crop phenotypic data file set stored in the distributed file system to be uploaded in sequence. If the size of the current uploaded file in the file set is larger than the data block threshold (for example, 128MB), ( ) times (that is, the file size of the currently uploaded file is greater than the first threshold), the currently uploaded file is directly stored in the distributed file system, and the remaining files in the file set are continued to be traversed.

[0037] Based on the total file size of the file set to be uploaded and the file merging threshold, determine the number of merge queues A, the number of standby queues B, and the number of temporary queues C. B and C are much smaller than A. Then, the corresponding merge queues, standby queues, and temporary queues are initialized in sequence as needed.

[0038] If the size of the currently uploaded file is larger than the minimum file threshold (e.g., 1MB) (i.e., the second threshold) and smaller than the first threshold, the currently uploaded file is added to the empty merge queue and the remaining files in the file set are traversed. If the size of the currently uploaded file is not larger than the second threshold, the currently uploaded file is placed in the temporary queue.

[0039] The largest file in the temporary queue (i.e., the largest file in this embodiment) is selected and compared with the maximum remaining space in the current merge queue (i.e., the maximum remaining space in this embodiment). If the largest file is larger than the maximum remaining space, indicating that the remaining space of all current merge queues cannot accommodate the file, the largest file is added to the empty standby queue, which is converted into a new merge queue, and the new merge operation continues. If the largest file is less than or equal to the maximum remaining space, the largest file is added to the merge queue corresponding to the maximum remaining space.

[0040] Calculate the total volume occupied by the files in the current merge queue. If the total volume reaches the data block threshold, ( ) times (that is, the files in the merge queue meet the preset conditions), all the files in the merge queue are packaged and stored in the data blocks of the distributed file system cluster nodes, and then the merge queue is destroyed, otherwise a new file merge operation is continued.

[0041] This embodiment uses the large file storage area and the small file merging area in the distributed file system to classify and store unstructured crop phenotypic data, thereby improving the flexibility of crop phenotypic data storage.

[0042] In one embodiment, the crop phenotypic data storage and retrieval method provided by the embodiment of the present invention may further include: Step 270: Establish a mapping relationship between the pre-merge file and the post-merge file. Step 280: Fill a second row key and a second column family into a second data table of the distributed database; the second row key is determined based on the pre-merge file; and the second column family is determined based on the metadata information of the unstructured data and the mapping relationship.

[0043] Specifically, a mapping relationship is established from small files (i.e., files before merging) to merged files (i.e., files after merging). This mapping relationship includes file name, file location, offset, and file size. An Elastic Hash Table (EHT) is used to construct n buckets, each storing index information for k small files. The file index information in each bucket is sorted lexicographically according to the hash value of the file name. A Minimal Perfect Hashing with Chaining (MWHC) hash function is then used to store the location information for each small file index. The MWHC hash function is written at the beginning of each index file.

[0044] The unstructured data table (HBase table named crop_unstructuring, the second data table in this embodiment) can store small file index information and metadata information. The index data (e.g., file name hash, file location, file size, and offset) stored after the small files are merged is used to locate the small file data; the metadata (e.g., crop name, plant ID, variety, and modality identifier) ​​is used to retrieve the small file data. The modality identifier is primarily used to efficiently retrieve small file data of the same modality type.

[0045] The second row key of the second data table can be the small file name. The second column family of the unstructured data table includes Metadata_info and Small_info. The Metadata_info column family contains information such as the crop name, plant ID, variety, and modality identifier. The Metadata_info column family is used to facilitate multi-condition combination retrieval of small file data. The Small_info column family contains the file name (hash), file location (HDFS path), offset, and file size. The Small_info column family is primarily used to determine the specific location of small files within the merged file.

[0046] This embodiment implements classified storage of unstructured crop phenotypic data and its metadata information through an unstructured data table.

[0047] In one embodiment, the crop phenotypic data storage and retrieval method provided by the embodiment of the present invention may further include: Step 500: When the structured data is searched based on the user's search conditions and the search results exist in the memory database, the search results are loaded locally; Step 600: When the structured data is searched based on the user's search criteria and no search results exist in the in-memory database, the distributed search analysis engine is used to determine a target row key corresponding to the user's search criteria, the search results are obtained from the distributed database based on the target row key, the search results are loaded locally, and the in-memory database is updated based on the loaded results. Step 700: When searching the unstructured data based on user search conditions, metadata results are retrieved from the memory database, search data is obtained from the distributed file system based on the metadata results, and the search data is loaded locally.

[0048] Specifically, the user's search conditions are analyzed, and based on the analysis results, when retrieving structured data, first, the data is retrieved from the memory database. If a hit is found, it is directly loaded locally. If a hit is not found, the row key corresponding to the specific search condition (the target row key in this embodiment) is obtained through the distributed search analysis engine. Then, the data is obtained from the distributed database through the target row key, and the obtained data is loaded locally. At the same time, the memory database is updated according to the obtained data.

[0049] When searching for large file data (such as point cloud data and spectral data), metadata information is first retrieved from the in-memory database. If a match is found, the system directly interacts with the DataNode (a node in the distributed file system) based on the metadata information to retrieve the data and load it locally. If a match is not found, metadata information is obtained from the distributed search and analysis engine, and then interacts with the DataNode again. After the data is retrieved, it is loaded locally and the metadata information for this data is loaded into the in-memory database.

[0050] When searching for small file data (such as images and thermal imaging data), the corresponding metadata is retrieved from the in-memory database based on specific search criteria. If a match is found, the data is directly loaded locally via the DataNode. If a match is not found, the distributed search and analysis engine uses a diversified search to return the corresponding row key. Metadata is then loaded from the unstructured data table (crop_unstructuring) using the row key. The NameNode (a node in the distributed file system) is then contacted to obtain the location of the merged file blocks. Phenotypic data is then retrieved from the DataNode and loaded locally. Finally, the corresponding merged file block information and small file metadata are loaded into the in-memory database. The retrieved multimodal crop phenotypic data is then integrated and returned to the user. The small file metadata is primarily used to locate the merged file in which the small file is located and its location within the merged file. The interaction with the NameNode is primarily to obtain the merged file location (including the number of blocks and the DataNode on which the blocks are located).

[0051] This embodiment improves the retrieval efficiency of multimodal crop phenotypic data by performing diversified retrieval on a hybrid architecture storage and retrieval system with classified storage.

[0052] Figure 2 This is the second flow chart of the crop phenotypic data storage and retrieval method provided by the present invention. Figure 2 As shown, the method may further include: Step 10: Determine access information for each cache key in the memory database; the access information includes access frequency, last access time, and current access time; Step 20: Based on the access information of each cache key, adjust the dynamic weight value of each cache key; Step 30: Dynamically adjust the cached data in the memory database based on the adjusted weight value.

[0053] Specifically, the present invention further provides a least recently used (Least Frequently Used, LFU) cache data elimination strategy with an adaptive weight mechanism, which is applied to the in-memory database of the present invention.

[0054] Redis stores structured data and metadata about unstructured data. Each piece of data is stored as a key-value pair. The cache key is the unique index of the data stored in Redis, and the stored data is directly located by the key value of the cache key. For example, if the cache key is point_cloud_2024, the HDFS path for its key value can be / large_files / point2024.pcd.

[0055] A set of attributes is established for each cache key in the in-memory database. The attribute set includes a dynamic weight value and access information (including access frequency, last access time, and current access time). The dynamic weight value (the weight value of each cache key is continuously updated according to the attributes. The dynamic weight value is used to evaluate the popularity of each cache key, that is, the popularity of the phenotypic data corresponding to the cache key) and access information are continuously updated after each retrieval. The initial value of the access frequency can be set to 1, and 1 is added after each retrieval access. The dynamic weight value of the cache key is dynamically calculated according to the following weight formula 1. Cache keys with a small difference between the current access time and the last access time and a large number of accesses are high-frequency retrieval data, and the weight value increment is large; cache keys with a large difference between the current retrieval time and the last access time and a small number of accesses are low-frequency retrieval data, and the weight value increment is small.

[0056] For sporadic, high-frequency, discontinuous access data, its access characteristics are characterized by a high initial search frequency and a significant decrease in access frequency later, resulting in a significant gap between the current access time and the most recent historical access time. By introducing access frequency as a weight calculation factor, this type of sporadic, high-frequency, discontinuous access data accumulates a higher number of searches in the early, high-frequency access phase, preventing incremental deviations in its weight value.

[0057] ; (1) in, is the dynamic weight value; is the frequency of visits; is the current access time; The time of the last visit; It is a compound assignment operator, which means accumulation followed by assignment.

[0058] The execution process of cache data elimination is as follows: During the weighted sorting phase, when the in-memory database reaches a certain memory threshold, the elimination process is triggered. All cache keys are sorted in descending order by weight, generating an ordered key list. The sorting is based on the weight attribute maintained for each cache key, with higher weights ranking higher.

[0059] High-weight enhancement: Based on the weight value, the top-ranked cache data keys (i.e., hot data) are intercepted as the enhancement objects. By forcing the data to reload (i.e., rewriting the complete data), the access counter of the underlying storage system's memory database is updated. The cached data is eliminated according to the LFU algorithm, and the weight value of the cache key is reset.

[0060] This embodiment forms a hot data maintenance solution with adaptive characteristics through dynamic weight value adjustment.

[0061] The crop phenotypic data storage and retrieval device provided by the present invention is described below. The crop phenotypic data storage and retrieval device described below and the crop phenotypic data storage and retrieval method described above can be referenced to each other.

[0062] Please refer to Figure 3 The present invention also provides a crop phenotypic data storage and retrieval device, comprising: A multimodal data acquisition module 301 is configured to acquire multimodal data to be stored; the multimodal data includes crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data includes structured data and unstructured data; the unstructured data includes small file data and large file data; A first storage module 302 is configured to store the structured data and the metadata information in a distributed database, and store the unstructured data in a distributed file system; The second storage module 303 is used to synchronously store the search field of the structured data, the search field of the small file data, and the metadata information of the large file data in a distributed search analysis engine; The retrieval module 304 is used to determine the retrieval results based on the user retrieval conditions and the distributed search analysis engine when there is no data matching the user retrieval conditions in the memory database; the memory database is used to cache the metadata information of the structured data and the unstructured data.

[0063] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the crop phenotypic data storage and retrieval method, which includes: obtaining multimodal data to be stored; the multimodal data includes crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data includes structured data and unstructured data; the unstructured data includes small file data and large file data; storing the structured data and the metadata information in a distributed database, and storing the unstructured data in a distributed file system; synchronously storing the retrieval field of the structured data, the retrieval field of the small file data and the metadata information of the large file data in a distributed search analysis engine; when there is no data matching the user's retrieval conditions in the memory database, determining the retrieval results based on the user's retrieval conditions and the distributed search analysis engine; the memory database is used to cache the structured data and the metadata information of the unstructured data.

[0064] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0065] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the crop phenotypic data storage and retrieval method provided by the above methods, the method including: obtaining multimodal data to be stored; the multimodal data includes crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data includes structured data and unstructured data; the unstructured data includes small file data and large file data; storing the structured data and the metadata information in a distributed database, and storing the unstructured data in a distributed file system; synchronously storing the retrieval field of the structured data, the retrieval field of the small file data and the metadata information of the large file data in a distributed search analysis engine; when there is no data matching the user's retrieval conditions in the memory database, determining the retrieval results based on the user's retrieval conditions and the distributed search analysis engine; the memory database is used to cache the structured data and the metadata information of the unstructured data.

[0066] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the crop phenotypic data storage and retrieval method provided by the above-mentioned methods, the method comprising: obtaining multimodal data to be stored; the multimodal data comprises crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data comprises structured data and unstructured data; the unstructured data comprises small file data and large file data; storing the structured data and the metadata information in a distributed database, and storing the unstructured data in a distributed file system; synchronously storing the retrieval field of the structured data, the retrieval field of the small file data and the metadata information of the large file data in a distributed search analysis engine; when there is no data matching the user's retrieval conditions in the memory database, determining the retrieval results based on the user's retrieval conditions and the distributed search analysis engine; the memory database is used to cache the structured data and the metadata information of the unstructured data.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0068] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for storing and retrieving crop phenotypic data, characterized in that: Applied to a hybrid architecture storage and retrieval system; the hybrid architecture storage and retrieval system includes a distributed database, a distributed search and analysis engine, a distributed file system, and an in-memory database; the method includes: Acquiring multimodal data to be stored; the multimodal data including crop phenotypic data and metadata information of the crop phenotypic data; the crop phenotypic data including structured data and unstructured data; the unstructured data including small file data and large file data; Storing the structured data and the metadata information in the distributed database, and storing the unstructured data in the distributed file system; Synchronously storing the search fields of the structured data, the search fields of the small file data, and the metadata information of the large file data in the distributed search analysis engine; When there is no data matching the user's search conditions in the memory database, a search result is determined based on the user's search conditions and the distributed search analysis engine; the memory database is used to cache metadata information of the structured data and the unstructured data.

2. The crop phenotypic data storage and retrieval method according to claim 1, characterized in that: The metadata information of the structured data includes crop identification and other crop information; Storing the structured data and the metadata information in the distributed database includes: Filling a first row key and a first column family into a first data table of the distributed database; the first row key is determined based on the crop identifier; and the first column family is determined based on the structured data and the other crop information.

3. The crop phenotypic data storage and retrieval method according to claim 1, characterized in that: Storing the unstructured data in the distributed file system includes: Traverse the unstructured data to be stored and obtain the file size of the currently uploaded file; If the file size is greater than a first threshold, compress the currently uploaded file and store it in the distributed file system; if the file size is less than the first threshold and greater than a second threshold, place the currently uploaded file in an empty merge queue; if the file size is less than or equal to the second threshold, place the currently uploaded file in a temporary queue; Selecting the largest file from the temporary queue and determining the maximum remaining space in each of the merged queues; If the largest file is larger than the maximum remaining space, the largest file is placed in a merge queue converted from the standby queue; if the largest file is smaller than or equal to the maximum remaining space, the largest file is placed in a merge queue corresponding to the maximum remaining space; When the files in the merge queue meet a preset condition, the files in the merge queue are merged and stored in the distributed file system.

4. The crop phenotypic data storage and retrieval method according to claim 3, characterized in that: The crop phenotypic data storage and retrieval method further comprises: Establish a mapping relationship between the file before the merge and the file after the merge; Filling a second row key and a second column family into a second data table of the distributed database; the second row key is determined based on the pre-merger file; and the second column family is determined based on the metadata information of the unstructured data and the mapping relationship.

5. The crop phenotypic data storage and retrieval method according to claim 1, characterized in that: The step of synchronously storing the search field of the structured data, the search field of the small file data, and the metadata information of the large file data in the distributed search analysis engine includes: When the structured data is searched based on the user's search conditions and the search results exist in the memory database, the search results are loaded locally; When the structured data is searched based on the user search condition and no search result exists in the memory database, the target row key corresponding to the user search condition is determined by the distributed search analysis engine, the search result is obtained from the distributed database based on the target row key, the search result is loaded locally, and the memory database is updated based on the loaded result; In the case of retrieving the unstructured data based on the user's search conditions, metadata results are retrieved from the memory database, search data is acquired from the distributed file system based on the metadata results, and the search data is loaded locally.

6. The crop phenotypic data storage and retrieval method according to claim 1, characterized in that: The crop phenotypic data storage and retrieval method further comprises: Determining access information for each cache key in the memory database; the access information includes access frequency, last access time, and current access time; Adjust the dynamic weight value of each cache key based on the access information of each cache key; The cached data in the memory database is dynamically adjusted based on the adjusted weight value.

7. A crop phenotypic data storage and retrieval device, characterized in that: include: A multimodal data acquisition module, configured to acquire multimodal data to be stored; the multimodal data includes crop phenotypic data and metadata information of the crop phenotypic data; The crop phenotypic data includes structured data and unstructured data; the unstructured data includes small file data and large file data; A first storage module, configured to store the structured data and the metadata information in a distributed database, and store the unstructured data in a distributed file system; A second storage module is used to synchronously store the search field of the structured data, the search field of the small file data, and the metadata information of the large file data in a distributed search analysis engine; A retrieval module is used to determine the retrieval results based on the user retrieval conditions and the distributed search analysis engine when there is no data matching the user retrieval conditions in the memory database; the memory database is used to cache the metadata information of the structured data and the unstructured data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the crop phenotypic data storage and retrieval method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the crop phenotypic data storage and retrieval method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the crop phenotypic data storage and retrieval method according to any one of claims 1 to 6 is implemented.

Images