Method and System for Constructing an Agricultural Product Data Space Platform Based on a Large-Scale Corpus

Abstract

The present invention relates to the technical field of agricultural product data management, and a method and system for constructing an agricultural product data space platform based on a large-scale corpus, including: obtaining an agricultural product text set, segmenting the agricultural product text to obtain an agricultural product sentence set, calculating a comprehensive key value set of the agricultural product sentence set, obtaining key agricultural product texts according to the comprehensive key value set, performing overlapping segmentation on the key agricultural product texts to obtain an agricultural product text segment set, obtaining associated feature vectors based on the agricultural product text segment set, summarizing and combining the associated feature vectors to obtain a target agricultural product vector set, using the target agricultural product vector set to perform data structuring to obtain an agricultural product structure data set, and obtaining an agricultural product data space platform based on the agricultural product structure data set. The present invention can improve the efficiency of constructing an agricultural product data platform and the relevance of agricultural product data in the agricultural product data platform.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural product data management, and particularly to a method and system for constructing an agricultural product data space platform based on a large-scale corpus. Background Art

[0002] With the advancement of agricultural modernization and digitization, the construction of an agricultural product data platform has become a key link in improving agricultural production efficiency, optimizing supply chain management, and meeting consumer needs. By integrating scattered agricultural product information, the agricultural product data platform realizes the structured and intelligent processing of data, which can not only provide accurate market insights for agricultural producers but also help consumers obtain comprehensive product information.

[0003] The construction of traditional agricultural product data platforms usually relies on manual screening and simple rule matching. Although this method can achieve data integration to a certain extent, it is difficult to efficiently process complex text data in a large-scale corpus, resulting in low efficiency and the inability to deeply explore semantic associations in agricultural product texts, leading to information fragmentation and low relevance of agricultural product data in the constructed platform. Summary of the Invention

[0004] The present invention provides a method and system for constructing an agricultural product data space platform based on a large-scale corpus, and its main purpose is to improve the efficiency of constructing an agricultural product data platform and the relevance of agricultural product data in the agricultural product data platform.

[0005] To achieve the above object, a method for constructing an agricultural product data space platform based on a large-scale corpus provided by the present invention includes:

[0006] Receiving a data platform construction instruction, determining a corpus channel set based on the data platform construction instruction, and sequentially extracting corpus channels in the corpus channel set;

[0007] Obtaining an agricultural product text set in the corpus channel, and sequentially extracting agricultural product texts in the agricultural product text set, where the agricultural product text includes agricultural product introduction articles;

[0008] Segmenting the agricultural product text to obtain an agricultural product sentence set, performing multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, and calculating a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set;

[0009] According to a preset standard key value and the comprehensive key value set, screening out a key agricultural product sentence set in the agricultural product sentence set, merging the key agricultural product sentence sets to obtain a key agricultural product text, and performing overlapping segmentation on the key agricultural product text to obtain an agricultural product text segment set;

[0010] Input the set of agricultural product text fragments into a pre-acquired Longformer model to obtain agricultural product feature vectors, and input the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors;

[0011] Summarize the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merge the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set;

[0012] Identify multiple agricultural product names in the corpus channel set, and perform data structuring on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain an agricultural product structure data set;

[0013] Store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

[0014] Optionally, performing multi-dimensional key-value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key-value set, an information key-value set, and a title key-value set includes:

[0015] Perform paragraph division on the agricultural product sentence set to obtain multiple paragraph agricultural product sentence sets, where the paragraph agricultural product sentences in the same paragraph agricultural product sentence set come from the same paragraph;

[0016] Perform the following operations on the paragraph agricultural product sentence sets in the multiple paragraph agricultural product sentence sets:

[0017] Identify the agricultural product paragraph where the paragraph agricultural product sentence set is located, and count the paragraph serial number of the agricultural product paragraph in the agricultural product text;

[0018] Extract the paragraph agricultural product sentences in the paragraph agricultural product sentence set in sequence, and count the sentence serial number of the paragraph agricultural product sentence in the agricultural product paragraph, where the sentence serial number is the position of the paragraph agricultural product in the agricultural product paragraph;

[0019] Summarize the sentence serial numbers corresponding to each paragraph agricultural product sentence in the paragraph agricultural product sentence set to obtain a sentence serial number set, and identify the median sentence serial number in the sentence serial number set;

[0020] Calculate a sentence sorting value set based on the median sentence serial number and the sentence serial number set;

[0021] Summarize the paragraph serial numbers corresponding to each paragraph agricultural product sentence set in the multiple paragraph agricultural product sentence sets to obtain multiple paragraph serial numbers, identify the median paragraph serial number in the multiple paragraph serial numbers, and calculate multiple paragraph sorting values based on the median paragraph serial number and the multiple paragraph serial numbers;

[0022] Based on multiple paragraph sorting values and sentence sorting value sets, calculate the sorting key value of each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set;

[0023] Perform information content analysis on the agricultural product sentence set to obtain an information key value set, and perform title association analysis on the agricultural product sentence set to obtain a title key value set.

[0024] Optionally, the calculating the sentence sorting value set based on the median sentence number and the sentence number set includes:

[0025] Successively extract sentence numbers in the sentence number set, and calculate the sentence sorting value using the following formula:

[0026]

[0027] where, represents the sentence sorting value, represents the median sentence number, represents the sentence number, represents taking the absolute value, represents the number of sentence numbers in the sentence number set;

[0028] Summarize the sentence sorting values to obtain a sentence sorting value set.

[0029] Optionally, the performing information content analysis on the agricultural product sentence set to obtain an information key value set includes:

[0030] Successively extract agricultural product sentences in the agricultural product sentence set, perform word segmentation on the agricultural product sentences to obtain an original word segmentation group, and remove stop words in a preset stop word list from the original word segmentation group to obtain a target word segmentation group, where the target word segmentation group includes multiple target words;

[0031] Successively extract target words in the target word segmentation group, and respectively count the number of target words of the target word in the agricultural product text and the total number of words in the agricultural product text;

[0032] Identify multiple agricultural product text sets of the corpus channel set, and respectively count the number of target texts containing the target word and the total number of texts in the multiple agricultural product text sets, where the corpus channel set includes multiple corpus channels, and the corpus channels correspond to the agricultural product text sets one by one;

[0033] Respectively summarize the number of target words and the number of target texts corresponding to each target word in the target word segmentation group to obtain a target word number array and a target text number array;

[0034] Calculate the information key value according to the target word number array, the target text number array, the total number of words and the total number of texts;

[0035] Summarize the information key values of each agricultural product sentence in the agricultural product sentence set to obtain an information key value set.

[0036] Optionally, calculating the information key value according to the target word array, the target text array, the total number of words, and the total number of texts includes:

[0037] Calculate the information key value using the following formula:

[0038]

[0039] Wherein, represents the information key value, represents the number of target words in the target word segmentation group, represents the th target word count in the target word array, represents the total number of words, represents the natural logarithm, represents the total number of texts, represents the th target text count in the target text array.

[0040] Optionally, performing title association analysis on the agricultural product sentence set to obtain a title key value set, including:

[0041] Identify the text title of the agricultural product text, segment the text title to obtain a title phrase group, sequentially extract title words from the title phrase group, count the number of title texts containing the title word in multiple agricultural product text sets, and summarize the number of title texts to obtain a title text array;

[0042] Construct a title word vector of the title phrase group according to the title text array and the total number of texts, wherein the title word vector is expressed as:

[0043]

[0044] Wherein, represents the title word vector, represents the first vector element in the title word vector, represents the th vector element in the title word vector, represents the first title text count in the title text array, represents the th title text count in the title text array, represents the vector dimension of the title word vector;

[0045] Extract agricultural product sentences from the agricultural product sentence set in sequence, construct the single-sentence word vectors of the agricultural product sentences, calculate the cosine of the vectors between the single-sentence word vectors and the title word vectors, and record the cosine of the vectors as the title key value. Before calculating the cosine of the vectors, it is also necessary to convert the single-sentence word vectors and the title word vectors to the same dimension;

[0046] Summarize the title key values of each agricultural product sentence in the agricultural product sentence set to obtain a title key value set.

[0047] Optionally, the overlapping segmentation of the key agricultural product text to obtain an agricultural product text fragment set includes:

[0048] Extract the first text in the key agricultural product text according to the preset input character length to obtain an agricultural product text fragment, where the character length of the agricultural product text fragment is the same as the input character length;

[0049] Based on the preset text overlapping ratio, extract the last text in the agricultural product text fragment to obtain an agricultural product overlapping fragment, where the ratio of the character length of the agricultural product overlapping fragment to the input character length is the text overlapping ratio;

[0050] Delete the agricultural product text fragment from the key agricultural product text to obtain a trimmed agricultural product text;

[0051] If it is confirmed that the step of extracting the first text in the key agricultural product text is not the first time to perform the extraction of the first text, then supplement the agricultural product overlapping fragment to the trimmed agricultural product text to obtain a supplemented agricultural product text;

[0052] Take the supplemented agricultural product text as the key agricultural product text, and return to the step of extracting the first text in the key agricultural product text according to the preset input character length until the trimmed agricultural product text does not contain any characters;

[0053] Summarize the agricultural product text fragments to obtain an agricultural product text fragment set.

[0054] Optionally, the data structuring is performed on each agricultural product name in the target agricultural product vector set by using the target agricultural product vector set to obtain an agricultural product structure data set, including:

[0055] Classify the target agricultural product vector set based on multiple agricultural product names to obtain multiple same-name agricultural product vector sets;

[0056] Perform the following operations on each same-name agricultural product vector set in the multiple same-name agricultural product vector sets:

[0057] Extract the same-name agricultural product vectors in the same-name agricultural product vector set in sequence, and calculate the vector center distance of the same-name agricultural product vectors;

[0058] If it is confirmed that the vector center distance is not greater than the preset standard center distance, then the same-name agricultural product vector is recorded as a valid agricultural product vector, and the valid agricultural product vectors in the same-name agricultural product vector set are aggregated to obtain a valid agricultural product vector set;

[0059] Based on the valid agricultural product vector set, an average calculation of the vectors is performed to obtain a target feature vector, and the target feature vector is paired with the agricultural product name to obtain agricultural product structure data, where the agricultural product structure data is expressed as:

[0060]

[0061] Among them, represents the agricultural product structure data, represents the agricultural product name, represents the target feature vector;

[0062] Aggregate the agricultural product structure data of each agricultural product name among multiple agricultural product names to obtain an agricultural product structure data set.

[0063] Optionally, calculating the vector center distance of the same-name agricultural product vectors includes:

[0064] Calculate the vector center distance of the same-name agricultural product vectors using the following formula:

[0065]

[0066] Among them, represents the vector center distance, represents the same-name agricultural product vector, represents the number of same-name agricultural product vectors in the same-name agricultural product vector set, represents the th same-name agricultural product vector in the same-name agricultural product vector set, represents the modulus of the vector.

[0067] To achieve the above object, the present invention also provides a system for constructing an agricultural product data space platform based on a large-scale corpus, including:

[0068] An agricultural product text extraction module, configured to receive a data platform construction instruction, determine a corpus channel set based on the data platform construction instruction, sequentially extract corpus channels in the corpus channel set, obtain an agricultural product text set in the corpus channel, and sequentially extract agricultural product texts in the agricultural product text set, where the agricultural product text includes agricultural product introduction articles;

[0069] An agricultural product segment splitting module, which is used to split agricultural product texts to obtain a set of agricultural product sentences, perform multi-dimensional key value analysis on each agricultural product sentence in the set of agricultural product sentences to obtain a sorted key value set, an information key value set, and a title key value set, and calculate a comprehensive key value set of the set of agricultural product sentences based on the sorted key value set, the information key value set, and the title key value set. According to a preset standard key value and the comprehensive key value set, key agricultural product sentences are screened out from the set of agricultural product sentences, the key agricultural product sentences are merged to obtain key agricultural product texts, and the key agricultural product texts are subjected to overlapping segmentation to obtain a set of agricultural product text segments;

[0070] An agricultural product vector calculation module, which is used to input the set of agricultural product text segments into a pre-obtained Longformer model to obtain agricultural product feature vectors, and input the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors. The associated feature vectors corresponding to each agricultural product text in the set of agricultural product texts are summarized to obtain an associated feature vector set, and the associated feature vector sets corresponding to each corpus channel in the corpus channel set are merged to obtain a target agricultural product vector set;

[0071] An agricultural product platform construction module, which is used to identify multiple agricultural product names in the corpus channel set, perform data structuring on each agricultural product name in the multiple agricultural product names by using the target agricultural product vector set to obtain an agricultural product structure data set, and store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

[0072] To solve the above problems, the present invention also provides an electronic device, and the electronic device includes:

[0073] A memory that stores at least one instruction;

[0074] A processor that executes the instructions stored in the memory to implement the above-mentioned method for constructing an agricultural product data space platform based on a large-scale corpus.

[0075] To solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned method for constructing an agricultural product data space platform based on a large-scale corpus.

[0076] To solve the problems described in the background art, the present invention first determines a corpus channel set, which can ensure the diversity of agricultural product data sources, provide rich basic data support for subsequent agricultural product data processing, and thus improve the reliability and coverage of platform construction. Then, multi-dimensional key value analysis is performed on each agricultural product sentence in the agricultural product sentence set, and a comprehensive key value set is calculated based on the analysis results. Through text segmentation and multi-dimensional key value analysis, this step can accurately extract the core information in the agricultural product text, and the calculation of the comprehensive key value set further improves the degree of information structuring, significantly improving the efficiency of constructing the agricultural product data space platform. Then, the key agricultural product sentence set is screened out and the key agricultural product sentence set is merged to obtain the key agricultural product text, and the key agricultural product text is overlapped and segmented to obtain the agricultural product text segment set. Through screening and overlapping segmentation, this step can extract the most valuable agricultural product information segments while retaining the relevance of the context, providing high-quality input data for the subsequent generation of feature vectors. Further, by combining the Longformer model and the LSTM network, the feature vectors of the agricultural product text can be efficiently extracted and the associated feature vectors can be further generated, so that the associated feature vectors can better represent the agricultural product feature information contained in the agricultural product text. Then, multiple agricultural product names in the corpus channel set are confirmed, and data structuring is performed on each agricultural product name in the target agricultural product vector set to obtain the agricultural product structure data set. By performing data structuring on the agricultural product names through the target agricultural product vector set, complex text information can be converted into a structured data set, significantly improving the readability and usability of the data, and providing a standardized data basis for the final construction of the platform. Finally, the agricultural product structure data set is stored in the data space platform to obtain the agricultural product data space platform. By storing the structured data in the data space platform, this step can realize the efficient management and retrieval of agricultural product data, provide an intuitive and convenient agricultural product data access interface for users, and thus improve the usability and user experience of the platform. Therefore, the present invention can improve the efficiency of constructing the agricultural product data platform and the relevance of agricultural product data in the agricultural product data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 FIG. is a flowchart of a method for constructing an agricultural product data space platform based on a large-scale corpus according to an embodiment of the present invention;

[0078] Figure 2 FIG. is a functional module diagram of a system for constructing an agricultural product data space platform based on a large-scale corpus according to an embodiment of the present invention;

[0079] Figure 3 FIG. is a structural diagram of an electronic device for implementing the method for constructing an agricultural product data space platform based on a large-scale corpus according to an embodiment of the present invention.

[0080] Description of reference numerals:

[0081] 1. Electronic device; 10. Processor; 11. Memory; 12. Bus.

[0082] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0083] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0084] The embodiment of the present application provides a method for constructing an agricultural product data space platform based on a large-scale corpus. The execution subject of the method for constructing an agricultural product data space platform based on a large-scale corpus includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for constructing an agricultural product data space platform based on a large-scale corpus can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0085] Reference Figure 1 FIG. 1 is a flow chart of a method for constructing a large-scale corpus-based agricultural product data space platform according to an embodiment of the present invention. In this embodiment, the method for constructing a large-scale corpus-based agricultural product data space platform includes:

[0086] S1. Receive a data platform construction instruction, determine a corpus channel set based on the data platform construction instruction, and extract corpus channels in the corpus channel set in sequence.

[0087] It can be understood that the data platform construction instruction refers to an instruction initiated by humans to build an agricultural product data space platform, and the corpus channel refers to a channel through which humans can obtain articles related to agricultural products. The corpus channel is, for example, a website of a local government that introduces local specialty agricultural products.

[0088] S2. Obtain an agricultural product text set from a corpus channel, and sequentially extract agricultural product texts from the agricultural product text set, wherein the agricultural product texts include agricultural product introduction articles.

[0089] It is clear that the agricultural product text refers to a complete article in the corpus channel. For example, there is an introductory article about a certain agricultural product on a local website, and the introductory article is the agricultural product text.

[0090] S3. Segment the agricultural product text to obtain a set of agricultural product sentences. Perform multi-dimensional key value analysis on each agricultural product sentence in the set of agricultural product sentences to obtain a sorted key value set, an information key value set, and a title key value set. Based on the sorted key value set, the information key value set, and the title key value set, calculate the comprehensive key value set of the set of agricultural product sentences.

[0091] It can be understood that the set of agricultural product sentences refers to a set of multiple agricultural product sentences. Among them, an agricultural product sentence refers to a sentence obtained by segmenting the agricultural product text, and the segmentation of the agricultural product text means dividing the agricultural product text into multiple sentences according to the full stop delimiter. The sorted key value set, the information key value set, and the title key value set respectively include multiple sorted key values, multiple information key values, and multiple title key values, and each agricultural product sentence in the set of agricultural product sentences corresponds to a sorted key value, an information key value, and a title key value. The comprehensive key value set includes multiple comprehensive key values, and the comprehensive key value corresponds to an agricultural product sentence.

[0092] It should be explained that the sorted key value refers to a value indicating the importance of the position of the agricultural product sentence in the agricultural product text. The higher the sorted key value, the higher the importance of the position of the agricultural product sentence in the agricultural product text. Since the form of the agricultural product introduction article usually adopts the general - point - to - general form, that is, first summarize the characteristics of the agricultural product, then elaborate on the characteristics of the agricultural product paragraph by paragraph, and finally summarize the characteristics of the agricultural product. In this form of article, the paragraphs and sentences at the front or the back are more capable of expressing the characteristics of the agricultural product compared to the middle paragraphs and sentences. In order to effectively extract the sentences that can represent the characteristics of the agricultural product from this form of article, a sorted key value can be set for each agricultural product sentence. The higher the sorted key value, the higher the importance of the agricultural product sentence in terms of sorting.

[0093] Furthermore, the information key value refers to the degree of importance of the information contained in the agricultural product sentence. The higher the information key value, the higher the degree of importance of the information contained in the agricultural product sentence. The words contained in an agricultural product sentence can represent the information shown by the agricultural product sentence. When there are more stop words in the agricultural product sentence, it means that the information shown by the agricultural product sentence has a weak ability to summarize the characteristics of the agricultural product. Among them, stop words refer to words that cannot represent the characteristics of the agricultural product, such as modal particles, conjunctions, adverbs, etc. The title key value refers to a value indicating the degree of association between the agricultural product sentence and the title of the corresponding agricultural product text. The higher the title key value, the higher the degree of association between the agricultural product sentence and the title of the corresponding agricultural product text. Since the title in the agricultural product text has a high - level summary effect on the entire agricultural product text, therefore, the agricultural product sentences with a high degree of similarity to the agricultural product title will be able to better represent the characteristics of the agricultural product in this agricultural product text.

[0094] Importantly, the comprehensive key value refers to a comprehensive indicator obtained by combining the sorting key value, information key value, and title key value corresponding to an agricultural product sentence. The calculation formula for this comprehensive key value is as follows: , where represents the comprehensive key value, , and respectively represent the artificially set weight coefficients. Optionally, , and are respectively set to: 0.5, 0.3, and 0.2, , and respectively represent the sorting key value, information key value, and title key value.

[0095] Specifically, performing multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set includes:

[0096] Performing paragraph division on the agricultural product sentence set to obtain multiple paragraph agricultural product sentence sets, where the paragraph agricultural product sentences in the same paragraph agricultural product sentence set come from the same paragraph;

[0097] Performing the following operations on the paragraph agricultural product sentence sets in the multiple paragraph agricultural product sentence sets:

[0098] Identifying the agricultural product paragraph where the paragraph agricultural product sentence set is located and counting the paragraph serial number of the agricultural product paragraph in the agricultural product text;

[0099] Sequentially extracting the paragraph agricultural product sentences in the paragraph agricultural product sentence set and counting the sentence serial number of the paragraph agricultural product sentence in the agricultural product paragraph, where the sentence serial number is the position of the paragraph agricultural product in the agricultural product paragraph;

[0100] Summarizing the sentence serial numbers corresponding to each paragraph agricultural product sentence in the paragraph agricultural product sentence set to obtain a sentence serial number set and identifying the median sentence serial number in the sentence serial number set;

[0101] Calculating a sentence sorting value set based on the median sentence serial number and the sentence serial number set;

[0102] Summarizing the paragraph serial numbers corresponding to each paragraph agricultural product sentence set in the multiple paragraph agricultural product sentence sets to obtain multiple paragraph serial numbers, identifying the median paragraph serial number among the multiple paragraph serial numbers, and calculating multiple paragraph sorting values based on the median paragraph serial number and the multiple paragraph serial numbers;

[0103] Calculating the sorting key value of each agricultural product sentence in the agricultural product sentence set based on the multiple paragraph sorting values and the sentence sorting value set to obtain a sorting key value set;

[0104] Perform information content analysis on the agricultural product sentence set to obtain the information key value set, and perform title correlation analysis on the agricultural product sentence set to obtain the title key value set.

[0105] It is understandable that the paragraph agricultural product sentence set refers to the set of sentences from the same paragraph in the agricultural product text. The median sentence serial number refers to the median of each sentence serial number in the sentence serial number set. The sentence sorting value set includes multiple sentence sorting values, and the sentence sorting value refers to the importance of the paragraph agricultural product sentence in the corresponding paragraph. The detailed steps for calculating the sentence sorting value set will be given later. The paragraph serial number refers to the position of the agricultural product paragraph in the agricultural product text, and the median paragraph serial number refers to the median value of multiple paragraph serial numbers. The paragraph sorting value refers to the importance of the agricultural product paragraph in the entire agricultural product text. The method for calculating multiple paragraph sorting values is the same as that for calculating the sentence sorting value set, and the steps for calculating the sorting key value set can be referred to later.

[0106] Importantly, each agricultural product paragraph in the agricultural product text corresponds to a paragraph sorting value. Furthermore, each paragraph agricultural product sentence in this agricultural product paragraph corresponds to this paragraph sorting value. That is, each agricultural product sentence in the agricultural product text corresponds to a paragraph sorting value and a sentence sorting value. Combining the above description, the detailed steps for calculating the sorting key value of each agricultural product sentence in the agricultural product sentence set are as follows: sequentially extract agricultural product sentences in the agricultural product sentence set, and respectively identify the paragraph sorting value and sentence sorting value corresponding to the agricultural product sentence in the multiple paragraph sorting value and sentence sorting value sets, perform weighted summation on the paragraph sorting value and sentence sorting value to obtain the sorting key value, and summarize the sorting key values corresponding to each agricultural product sentence in the agricultural product sentence set to obtain the sorting key value set. Among them, the formula for weighted summation is as follows: , where and respectively represent the artificially set weight coefficients. Optionally, set and to 0.6 and 0.4 respectively. and respectively represent the sentence sorting value and the paragraph sorting value.

[0107] Specifically, the calculation of the sentence sorting value set based on the median sentence serial number and the sentence serial number set includes:

[0108] Sequentially extract sentence serial numbers in the sentence serial number set, and calculate the sentence sorting value using the following formula:

[0109]

[0110] where represents the sentence sorting value, represents the median sentence serial number, represents the sentence serial number. Indicates taking the absolute value, represents the number of statement numbers in the set of statement numbers;

[0111] Summarize the statement sorting values to obtain a set of statement sorting values.

[0112] It can be understood that the steps for calculating the sorting values of multiple paragraphs are the same as those for calculating the set of statement sorting values, and will not be elaborated here.

[0113] Specifically, the information content analysis of the agricultural product sentence set to obtain the set of information key values includes:

[0114] Successively extract agricultural product sentences from the agricultural product sentence set, perform word segmentation on the agricultural product sentences to obtain an original word segmentation group, and remove the stop words in the preset stop word list from the original word segmentation group to obtain a target word segmentation group, where the target word segmentation group includes multiple target words;

[0115] Successively extract target words from the target word segmentation group, and respectively count the number of target words of the target word in the agricultural product text and the total number of words in the agricultural product text;

[0116] Identify multiple agricultural product text sets in the corpus channel set, and respectively count the number of target texts containing the target word and the total number of texts in the multiple agricultural product text sets, where the corpus channel set includes multiple corpus channels, and the corpus channels correspond to the agricultural product text sets one by one;

[0117] Respectively summarize the number of target words and the number of target texts corresponding to each target word in the target word segmentation group to obtain a target word array and a target text array;

[0118] Calculate the information key value according to the target word array, the target text array, the total number of words and the total number of texts;

[0119] Summarize the information key values of each agricultural product sentence in the agricultural product sentence set to obtain a set of information key values.

[0120] It can be understood that the original word segmentation group refers to the combination of multiple original words obtained after performing word segmentation on the agricultural product sentence. Among them, performing word segmentation on the agricultural product can be achieved through a Chinese word segmentation tool, such as the jieba library, etc. The stop word list refers to the set of stop words obtained manually, and this stop word list can directly use existing word lists, such as: Baidu stop word list, etc. The target word segmentation group refers to the original word segmentation group after removal. The number of target words refers to the number of times the target word appears in the agricultural product text, and the total number of words refers to the total number of all words in the agricultural product text. The number of target texts refers to the number of agricultural product texts containing the target word in the multiple agricultural product text sets, and the total number of texts refers to the number of agricultural product texts in the multiple agricultural product text sets.

[0121] Specifically, calculating the information key value according to the target word array, target text array, total number of words, and total number of texts includes:

[0122] Calculating the information key value using the following formula:

[0123]

[0124] Wherein, represents the information key value, represents the number of target words in the target word segmentation group, represents the th target word count in the target word array, represents the total number of words, represents the natural logarithm, represents the total number of texts, represents the th target text count in the target text array.

[0125] It can be understood that in the above formula for calculating the information key value, the item is the term frequency item,

[0126] Specifically, performing title association analysis on the agricultural product sentence set to obtain the title key value set includes:

[0127] Identifying the text title of the agricultural product text, segmenting the text title to obtain a title word group, sequentially extracting title words from the title word group, counting the number of title texts containing the title word in multiple agricultural product text sets, and summarizing the number of title texts to obtain a title text array;

[0128] Constructing the title word vector of the title word group according to the title text array and the total number of texts, where the title word vector is expressed as:

[0129]

[0130] Wherein, represents the title word vector, represents the first vector element in the title word vector, represents the th vector element in the title word vector, represents the first title text count in the title text array, represents the The number of title texts, indicating the vector dimension of the title word vector;

[0131] Successively extract agricultural product sentences from the agricultural product sentence set, construct the single-sentence word vectors of the agricultural product sentences, calculate the vector cosine between the single-sentence word vectors and the title word vectors, and record the vector cosine as the title key value. Before calculating the vector cosine, the single-sentence word vectors and the title word vectors need to be converted to the same dimension;

[0132] Summarize the title key values of each agricultural product sentence in the agricultural product sentence set to obtain a title key value set.

[0133] It can be understood that the title phrase refers to the combination of words in the text title after word segmentation, and the number of title texts refers to the number of agricultural product texts containing title words. The title word vector refers to the vector composed of the inverse document frequencies of each title word in the title phrase. The construction method of the single-sentence word vector is the same as that of the title word vector, which will not be elaborated here. Before calculating the vector cosine, the single-sentence word vectors and the title word vectors need to be converted to the same dimension, which means that since the vector dimensions of the title word vector and the single-sentence word vector are not necessarily the same, when their vector dimensions are different, the vector with a smaller vector dimension needs to be supplemented with elements, that is, 0 elements are supplemented to the end of the vector until the vector has the same dimension as the other vector.

[0134] S4. According to the preset standard key value and the comprehensive key value set, screen out the key agricultural product sentence set from the agricultural product sentence set, merge the key agricultural product sentence set to obtain the key agricultural product text, and perform overlapping segmentation on the key agricultural product text to obtain the agricultural product text segment set.

[0135] It can be understood that the standard key value refers to a constant set by humans. Screening in the agricultural product sentence set means that the agricultural product sentences with a comprehensive key value greater than the standard key value are recorded as key agricultural product sentences. The key agricultural product sentence set refers to the set of multiple key agricultural products, and the key agricultural product sentence refers to the agricultural product sentence with a comprehensive key value greater than the standard key value. The merging means merging the multiple key agricultural product sentences in the key agricultural product sentence set into a complete text at the beginning and end. The agricultural product text segment refers to a part of the key agricultural product text obtained after overlapping segmentation.

[0136] Specifically, performing overlapping segmentation on the key agricultural product text to obtain the agricultural product text segment set includes:

[0137] According to the preset input character length, perform first and last text extraction on the key agricultural product text to obtain the agricultural product text segment, where the character length of the agricultural product text segment is the same as the input character length;

[0138] Perform last text extraction on the agricultural product text segment based on a preset text overlap ratio to obtain an agricultural product overlapping segment, where the ratio of the character length of the agricultural product overlapping segment to the input character length is the text overlap ratio;

[0139] Delete the agricultural product text segment from the key agricultural product text to obtain a text with the agricultural product text removed;

[0140] If it is confirmed that the step of performing first text extraction in the key agricultural product text is not the first time to perform first text extraction, then supplement the agricultural product overlapping segment to the text with the agricultural product text removed to obtain a supplemented agricultural product text;

[0141] Take the supplemented agricultural product text as the key agricultural product text, and return the step of performing first text extraction in the key agricultural product text according to the preset input character length until the text with the agricultural product text removed does not contain any characters;

[0142] Summarize the agricultural product text segments to obtain a set of agricultural product text segments.

[0143] It can be understood that the input character length refers to the character length set artificially, and this input character length should not exceed the maximum length of the text that can be input by the Longformer model. The first text extraction refers to extracting text from the beginning in the key agricultural product text. Correspondingly, the last text extraction refers to extracting text from the end in the agricultural product text segment. The text overlap ratio refers to the ratio of the length of the text extracted in the last text extraction step to the length of the agricultural product text segment set artificially.

[0144] Furthermore, in order to limit the character length that can be input by the subsequent Longformer model, it is necessary to split the key agricultural product. However, if it is directly split, there will be no overlap between the split text segments, resulting in the Longformer model being unable to capture the semantic association between different text segments. Therefore, it is necessary to perform overlapping splitting on the key agricultural product.

[0145] S5. Input the set of agricultural product text segments into a pre-obtained Longformer model to obtain agricultural product feature vectors, and input the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors.

[0146] It should be explained that the Longformer model is an improved Transformer model designed specifically for processing long texts. It can efficiently capture the context information in long texts and finally generate a high-dimensional feature vector containing the full text semantics of agricultural product texts. Among them, the agricultural product feature vector is this high-dimensional feature vector. The LSTM network (Long Short-Term Memory network) is a type of recurrent neural network that is good at modeling the temporal dependencies in sequential data. In this solution, this network can capture the dynamic associations within the agricultural product feature vector, thereby further extracting global semantic features. The associated feature vector refers to the output vector of the LSTM network.

[0147] Furthermore, the above-mentioned Longformer model can encode the agricultural product text fragment set into a high-dimensional feature vector, which can solve the problem of information dispersion caused by the too long agricultural product text. Then, the LSTM network is used to further capture the dynamic associations of the agricultural product feature vectors output by the Longformer model. For example, a certain agricultural product text details the changes in the sales data of a certain agricultural product: abnormal weather leads to a decrease in the output of a certain agricultural product, which in turn causes the supply chain to be interrupted. Then the Longformer model can identify the key information in this text, such as: "abnormal weather", "a decrease in the output of a certain agricultural product", and "supply chain interruption", and encode this information to generate an agricultural product feature vector. Then this vector is input into the LSTM network, enabling the LSTM network to capture the temporal relationship between the above information, and then infer the logical chain of "weather → output → supply chain → price increase", and finally generate a more interpretable associated feature vector.

[0148] S6. Aggregate the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merge the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set.

[0149] It can be understood that merging the associated feature vector sets corresponding to each corpus channel in the corpus channel set means: putting all the elements in each associated feature vector set in multiple associated feature vector sets into the same set, and the set after completion of the merge is the target agricultural product vector set.

[0150] S7. Identify multiple agricultural product names in the corpus channel set, and perform data structuring on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain an agricultural product structure data set.

[0151] It can be understood that the agricultural product name refers to the name of the agricultural product that appears in the corpus channel set. The agricultural product structure data set includes multiple agricultural product structure data, where the agricultural product structure data refers to the key-value pair between the agricultural product name and the feature vector of this agricultural product name.

[0152] Specifically, performing data structuring on each agricultural product name in the target agricultural product vector set to obtain an agricultural product structure data set, including:

[0153] Classifying the target agricultural product vector set based on multiple agricultural product names to obtain multiple same-name agricultural product vector sets;

[0154] Performing the following operations on each same-name agricultural product vector set in the multiple same-name agricultural product vector sets:

[0155] Sequentially extracting same-name agricultural product vectors in the same-name agricultural product vector set and calculating the vector center distance of the same-name agricultural product vectors;

[0156] If it is confirmed that the vector center distance is not greater than a preset standard center distance, then recording the same-name agricultural product vector as a valid agricultural product vector, and summarizing the valid agricultural product vectors in the same-name agricultural product vector set to obtain a valid agricultural product vector set;

[0157] Performing average calculation of vectors based on the valid agricultural product vector set to obtain a target feature vector, and performing key-value pairing of the target feature vector and the agricultural product name to obtain agricultural product structure data, where the agricultural product structure data is expressed as:

[0158]

[0159] Among them, represents the agricultural product structure data, represents the agricultural product name, represents the target feature vector;

[0160] Summarizing the agricultural product structure data of each agricultural product name in multiple agricultural product names to obtain an agricultural product structure data set.

[0161] It is understandable that the same-name agricultural product vector set refers to a set of target agricultural product vectors whose represented agricultural products have the same agricultural product name. The vector center distance refers to the distance between the same-name agricultural product vector and the center of the same-name agricultural product vector set. The standard center distance refers to a constant set by humans. When the vector center distance is greater than the standard center distance, it indicates that the agricultural product characteristics represented by the same-name agricultural product vector have a large difference from the overall characteristics of the agricultural products represented by the same-name agricultural product vector set, and there may be a problem of inaccurate information. Therefore, this same-name agricultural product vector needs to be discarded. The average calculation of vectors based on the valid agricultural product vector set means: calculating the average value of the elements at the same position of all valid agricultural product vectors in the valid agricultural product vector set.

[0162] Specifically, the calculation of the vector center distance of the same-name agricultural product vectors includes:

[0163] Calculate the vector center distance of the same-name agricultural product vectors using the following formula:

[0164]

[0165] where, represents the vector center distance, represents the same-name agricultural product vector, represents the number of same-name agricultural product vectors in the same-name agricultural product vector set, represents the th same-name agricultural product vector in the same-name agricultural product vector set, represents the modulus of the vector.

[0166] S8. Store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

[0167] It should be explained that the data space platform refers to a data management platform integrating data analysis and storage functions. In this solution, the data space platform is a data management platform integrating the Longformer model, the LSTM network, and the cloud space. The agricultural product data space platform refers to the data space platform storing the agricultural product structure data set. Among them, the cloud space can be used to store the agricultural product structure data set. When receiving the agricultural product description information input by the user, the platform will construct a description agricultural product vector of the agricultural product description information. This step is the same as the construction method of the above-mentioned associated feature vector and requires the participation of the Longformer model and the LSTM network. Then, compare the similarity between the description agricultural product vector and each target feature vector in the agricultural product structure data set. The similarity comparison method can be: calculate the cosine value between these two vectors, and use the agricultural product name corresponding to the target feature vector with the largest cosine value as the output agricultural product name. Finally, output the output agricultural product name to complete the user's retrieval requirement.

[0168] To solve the problems described in the background art, the present invention first determines a corpus channel set, which can ensure the diversity of the sources of agricultural product data, provide rich basic data support for subsequent agricultural product data processing, and thus enhance the reliability and coverage of platform construction. Then, multi-dimensional key value analysis is performed on each agricultural product sentence in the agricultural product sentence set, and a comprehensive key value set is calculated based on the analysis results. Through text segmentation and multi-dimensional key value analysis, this step can accurately extract the core information in the agricultural product text, and the calculation of the comprehensive key value set further improves the degree of information structuring, significantly improving the efficiency of constructing the agricultural product data space platform. Then, the key agricultural product sentence set is screened and the key agricultural product sentence set is merged to obtain the key agricultural product text, and the key agricultural product text is overlapped and segmented to obtain the agricultural product text fragment set. Through screening and overlapping segmentation, this step can extract the most valuable agricultural product information fragments while retaining the relevance of the context, providing high-quality input data for the subsequent generation of feature vectors. Further, through the combination of the Longformer model and the LSTM network, the feature vectors of the agricultural product text can be efficiently extracted, and the associated feature vectors can be further generated, so that the associated feature vectors can better represent the agricultural product feature information contained in the agricultural product text. Then, multiple agricultural product names in the corpus channel set are confirmed, and data structuring is performed on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain the agricultural product structure data set. By performing data structuring on the agricultural product names using the target agricultural product vector set, complex text information can be converted into a structured data set, significantly improving the readability and usability of the data, and providing a standardized data basis for the final construction of the platform. Finally, the agricultural product structure data set is stored in the data space platform to obtain the agricultural product data space platform. By storing the structured data in the data space platform, this step can achieve the efficient management and retrieval of agricultural product data, provide an intuitive and convenient agricultural product data access interface for users, and thus enhance the practicality and user experience of the platform. Therefore, the present invention can improve the efficiency of constructing the agricultural product data platform and the relevance of the agricultural product data in the agricultural product data platform.

[0169] As Figure 2 shown, it is a functional module diagram of a system for constructing an agricultural product data space platform based on a large-scale corpus provided by an embodiment of the present invention.

[0170] The agricultural product data space platform construction system 100 based on a large-scale corpus according to the present invention can be installed in an electronic device. According to the functions achieved, the agricultural product data space platform construction system 100 based on a large-scale corpus can include an agricultural product text extraction module 101, an agricultural product segment segmentation module 102, an agricultural product vector calculation module 103, and an agricultural product platform construction module 104. The modules described in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0171] The agricultural product text extraction module 101 is configured to receive a data platform construction instruction, determine a corpus channel set based on the data platform construction instruction, sequentially extract corpus channels in the corpus channel set, obtain an agricultural product text set in the corpus channels, and sequentially extract agricultural product texts in the agricultural product text set, where the agricultural product texts include agricultural product introduction articles;

[0172] The agricultural product segment segmentation module 102 is configured to segment the agricultural product texts to obtain an agricultural product sentence set, perform multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, calculate a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set, screen out a key agricultural product sentence set in the agricultural product sentence set according to a preset standard key value and the comprehensive key value set, merge the key agricultural product sentence sets to obtain key agricultural product texts, and perform overlapping segmentation on the key agricultural product texts to obtain an agricultural product text segment set;

[0173] The agricultural product vector calculation module 103 is configured to input the agricultural product text segment set into a pre-obtained Longformer model to obtain agricultural product feature vectors, input the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors, summarize the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merge the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set;

[0174] The agricultural product platform construction module 104 is configured to identify multiple agricultural product names in the corpus channel set, perform data structuring on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain an agricultural product structure data set, and store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

[0175] Specifically, each module in the agricultural product data space platform construction system 100 based on a large-scale corpus in the embodiment of the present invention is used in the same way as the aboveFigure 1 The technical means are the same as those of the method for constructing an agricultural product data space platform based on a large-scale corpus described in , and can produce the same technical effects, which will not be elaborated here.

[0176] As Figure 3 shown, it is a schematic structural diagram of an electronic device for implementing the method for constructing an agricultural product data space platform based on a large-scale corpus provided by an embodiment of the present invention.

[0177] The electronic device 1 may include a processor 10, a memory 11, and a bus 12, and may further include a computer program stored in the memory 11 and operable on the processor 10, such as a program for the method for constructing an agricultural product data space platform based on a large-scale corpus.

[0178] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as the mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 1. Further, the memory 11 also includes the internal storage unit of the electronic device 1 and the external storage device. The memory 11 can not only be used to store application software installed in the electronic device 1 and various types of data, such as the code of the program for the method for constructing an agricultural product data space platform based on a large-scale corpus, but also be used to temporarily store data that has been output or will be output.

[0179] The processor 10 may be composed of integrated circuits in some embodiments. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as the program for the method for constructing an agricultural product data space platform based on a large-scale corpus, etc.), and calling data stored in the memory 11, to execute various functions of the electronic device 1 and process data.

[0180] The bus 12 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is arranged to implement connection communication between the memory 11 and at least one processor 10, etc.

[0181] Figure 3 Only an electronic device with components is shown. Those skilled in the art can understand that Figure 3 the shown structure does not constitute a limitation on the electronic device 1, and it may include fewer or more components than shown, or combine certain components, or have a different component arrangement.

[0182] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering each component. Preferably, the power source can be logically connected to the at least one processor 10 through a power management system, so as to implement functions such as charge management, discharge management, and power consumption management through the power management system. The power source may also include any components such as one or more DC or AC power sources, a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0183] Furthermore, the electronic device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0184] Optionally, the electronic device 1 may further include a user interface. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the electronic device 1 and to display a visual user interface.

[0185] The program of the method for constructing an agricultural product data space platform based on a large-scale corpus stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0186] Receive a data platform construction instruction, determine a corpus channel set based on the data platform construction instruction, and sequentially extract corpus channels in the corpus channel set;

[0187] Obtain an agricultural product text set in the corpus channel, and sequentially extract agricultural product texts in the agricultural product text set, where the agricultural product texts include agricultural product introduction articles;

[0188] Segment the agricultural product texts to obtain an agricultural product sentence set, perform multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, and calculate a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set;

[0189] According to a preset standard key value and the comprehensive key value set, screen out a key agricultural product sentence set in the agricultural product sentence set, merge the key agricultural product sentence set to obtain a key agricultural product text, and perform overlapping segmentation on the key agricultural product text to obtain an agricultural product text segment set;

[0190] Input the agricultural product text segment set into a pre-obtained Longformer model to obtain an agricultural product feature vector, and input the agricultural product feature vector into a pre-trained LSTM network to obtain an associated feature vector;

[0191] Summarize the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merge the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set;

[0192] Identify multiple agricultural product names in the corpus channel set, and perform data structuring on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain an agricultural product structure data set;

[0193] Store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

[0194] Specifically, the specific implementation method of the above instructions by the processor 10 can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0195] Furthermore, if the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or system capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0196] The present invention also provides a computer-readable storage medium. The readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, it can implement:

[0197] Receiving a data platform construction instruction, determining a corpus channel set based on the data platform construction instruction, and sequentially extracting corpus channels in the corpus channel set;

[0198] Obtaining an agricultural product text set in the corpus channel, and sequentially extracting agricultural product texts in the agricultural product text set, where the agricultural product texts include agricultural product introduction articles;

[0199] Segmenting the agricultural product texts to obtain an agricultural product sentence set, performing multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, and calculating a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set;

[0200] According to a preset standard key value and the comprehensive key value set, screening out a key agricultural product sentence set in the agricultural product sentence set, merging the key agricultural product sentence sets to obtain key agricultural product texts, and performing overlapping segmentation on the key agricultural product texts to obtain an agricultural product text segment set;

[0201] Inputting the agricultural product text segment set into a pre-obtained Longformer model to obtain agricultural product feature vectors, and inputting the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors;

[0202] Summarizing the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merging the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set;

[0203] Identifying multiple agricultural product names in the corpus channel set, and performing data structuring on each agricultural product name in the multiple agricultural product names by using the target agricultural product vector set to obtain an agricultural product structure data set;

[0204] Store the agricultural product structure dataset in a preset data space platform to obtain an agricultural product data space platform.

[0205] In several embodiments provided by the present invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiment described above is only illustrative, and there may be other partitioning methods in actual implementation.

[0206] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a hardware plus software functional module.

[0208] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for constructing an agricultural product data space platform based on a large-scale corpus, characterized in that The method includes: Receiving a data platform construction instruction, determining a corpus channel set based on the data platform construction instruction, and sequentially extracting corpus channels in the corpus channel set; Obtaining an agricultural product text set in the corpus channel, and sequentially extracting agricultural product texts in the agricultural product text set, where the agricultural product texts include agricultural product introduction articles; Segmenting the agricultural product texts to obtain an agricultural product sentence set, performing multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, and calculating a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set; The performing multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set includes: Performing paragraph division on the agricultural product sentence set to obtain multiple paragraph agricultural product sentence sets, where the paragraph agricultural product sentences in the same paragraph agricultural product sentence set come from the same paragraph; Performing the following operations on each paragraph agricultural product sentence set in the multiple paragraph agricultural product sentence sets: Identifying the agricultural product paragraph where the paragraph agricultural product sentence set is located, and counting the paragraph serial number of the agricultural product paragraph in the agricultural product text; Sequentially extracting the paragraph agricultural product sentences in the paragraph agricultural product sentence set, and counting the sentence serial number of the paragraph agricultural product sentence in the agricultural product paragraph, where the sentence serial number is the position of the paragraph agricultural product in the agricultural product paragraph; Summarizing the sentence serial numbers corresponding to each paragraph agricultural product sentence in the paragraph agricultural product sentence set to obtain a sentence serial number set, and identifying the median sentence serial number in the sentence serial number set; Calculating a sentence sorting value set based on the median sentence serial number and the sentence serial number set; Summarizing the paragraph serial numbers corresponding to each paragraph agricultural product sentence set in the multiple paragraph agricultural product sentence sets to obtain multiple paragraph serial numbers, identifying the median paragraph serial number in the multiple paragraph serial numbers, and calculating multiple paragraph sorting values based on the median paragraph serial number and the multiple paragraph serial numbers; Calculating the sorting key value of each agricultural product sentence in the agricultural product sentence set based on the multiple paragraph sorting values and the sentence sorting value set to obtain a sorting key value set; Performing information content analysis on the agricultural product sentence set to obtain an information key value set, and performing title association analysis on the agricultural product sentence set to obtain a title key value set; According to a preset standard key value and the comprehensive key value set, screening out a key agricultural product sentence set in the agricultural product sentence set, merging the key agricultural product sentence sets to obtain a key agricultural product text, and performing overlapping segmentation on the key agricultural product text to obtain an agricultural product text segment set; Inputting the agricultural product text segment set into a pre-obtained Longformer model to obtain agricultural product feature vectors, and inputting the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors; Summarizing the associated feature vectors corresponding to each agricultural product text in the agricultural product text set to obtain an associated feature vector set, and merging the associated feature vector sets corresponding to each corpus channel in the corpus channel set to obtain a target agricultural product vector set; Identify multiple agricultural product names with concentrated corpus channels, and perform data structuring on each agricultural product name in the multiple agricultural product names by using the target agricultural product vector set to obtain an agricultural product structure data set; Store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

2. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 1, characterized in that, The calculating the sentence sorting value set based on the median sentence serial number and the sentence serial number set includes: Sequentially extract the sentence serial numbers in the sentence serial number set, and calculate the sentence sorting value by using the following formula: ， Among them, represents the statement sorting value, represents the median statement serial number, represents the statement serial number, represents taking the absolute value, represents the number of statement serial numbers in the set of statement serial numbers; Summarize the sentence sorting values to obtain a sentence sorting value set.

3. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 2, wherein The performing information content analysis on the agricultural product sentence set to obtain an information key value set includes: Sequentially extract agricultural product sentences in the agricultural product sentence set, perform word segmentation on the agricultural product sentences to obtain an original word segmentation group, and remove stop words in a preset stop word list from the original word segmentation group to obtain a target word segmentation group, where the target word segmentation group includes multiple target words; Sequentially extract target words in the target word segmentation group, and respectively count the number of target words of the target word in the agricultural product text and the total number of words in the agricultural product text; Identify multiple agricultural product text sets of the corpus channel set, and respectively count the number of target texts containing the target word and the total number of texts in the multiple agricultural product text sets, where the corpus channel set includes multiple corpus channels, and the corpus channels correspond to the agricultural product text sets one by one; Respectively summarize the number of target words and the number of target texts corresponding to each target word in the target word segmentation group to obtain a target word number array and a target text number array; Calculate the information key value according to the target word number array, the target text number array, the total number of words and the total number of texts; Summarize the information key values of each agricultural product sentence in the agricultural product sentence set to obtain an information key value set.

4. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 3, wherein, The calculating the information key value according to the target word number array, the target text number array, the total number of words and the total number of texts includes: Calculate the information key value by using the following formula: ， Among them, represents the information key value, represents the number of target words in the target word group, represents the th target word count in the target word array, represents the total number of words, represents the natural logarithm, represents the total number of texts, represents the th target text count in the target text array.

5. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 4, wherein The performing title association analysis on the agricultural product sentence set to obtain a title key value set includes: Identify the text title of the agricultural product text, perform word segmentation on the text title to obtain a title word group, sequentially extract title words in the title word group, count the number of title texts containing the title word in the multiple agricultural product text sets, and summarize the number of title texts to obtain a title text array; Construct a title word vector of the title word group according to the title text array and the total number of texts, where the title word vector is expressed as: ， Among them, represents the title word vector, represents the first vector element in the title word vector, represents the th vector element in the title word vector, represents the first title text number in the title text array, represents the th title text number in the title text array, represents the vector dimension of the title word vector; Sequentially extract agricultural product sentences in the agricultural product sentence set, construct a single sentence word vector of the agricultural product sentence, calculate the vector cosine of the single sentence word vector and the title word vector, and record the vector cosine as the title key value, where the single sentence word vector and the title word vector need to be converted to the same dimension before calculating the vector cosine; Summarize the title key values of each agricultural product sentence in the agricultural product sentence set to obtain a title key value set.

6. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 5, wherein The performing overlapping segmentation on the key agricultural product text to obtain an agricultural product text segment set includes: According to a preset input character length, perform first and last text extraction on the key agricultural product text to obtain an agricultural product text segment, where the character length of the agricultural product text segment is the same as the input character length; Based on a preset text overlap ratio, perform last text extraction on the agricultural product text segment to obtain an agricultural product overlap segment, where the ratio of the character length of the agricultural product overlap segment to the input character length is the text overlap ratio; Delete the agricultural product text segment from the key agricultural product text to obtain the agricultural product text after deletion; If it is confirmed that the step of performing first text extraction in the key agricultural product text is not the first time to perform first text extraction, supplement the agricultural product overlap segment to the agricultural product text after deletion to obtain the supplemented agricultural product text; Take the supplemented agricultural product text as the key agricultural product text, and return to the step of performing first text extraction in the key agricultural product text according to the preset input character length until the agricultural product text after deletion does not contain any characters; Summarize the agricultural product text segments to obtain an agricultural product text segment set.

7. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 6, wherein Performing data structuring on each agricultural product name in the multiple agricultural product names by using the target agricultural product vector set to obtain an agricultural product structure data set, including: Classify the target agricultural product vector set based on the multiple agricultural product names to obtain multiple same-name agricultural product vector sets; Perform the following operations on each same-name agricultural product vector set in the multiple same-name agricultural product vector sets: Successively extract same-name agricultural product vectors in the same-name agricultural product vector set, and calculate the vector center distance of the same-name agricultural product vectors; If it is confirmed that the vector center distance is not greater than the preset standard center distance, record the same-name agricultural product vector as a valid agricultural product vector, and summarize the valid agricultural product vectors in the same-name agricultural product vector set to obtain a valid agricultural product vector set; Perform average calculation of vectors based on the valid agricultural product vector set to obtain a target feature vector, and pair the target feature vector with the agricultural product name to obtain agricultural product structure data, where the agricultural product structure data is expressed as: ， Among them, represents the agricultural product structure data, represents the agricultural product name, represents the target feature vector; Summarize the agricultural product structure data of each agricultural product name in the multiple agricultural product names to obtain an agricultural product structure data set.

8. The method for constructing an agricultural product data space platform based on a large-scale corpus according to claim 7, wherein The calculation of the vector center distance of the same-name agricultural product vectors includes: Calculate the vector center distance of the same-name agricultural product vectors by using the following formula: ， Among them, represents the vector center distance, represents the vector of the same name agricultural products, represents the number of vectors of the same name agricultural products in the vector set of the same name agricultural products, represents the th vector of the same name agricultural product in the vector set of the same name agricultural products, represents the modulus of the vector.

9. A system for constructing an agricultural product data space platform based on a large-scale corpus, which is used to implement the method described in any one of claims 1 to 8, characterized in that, The system includes: An agricultural product text extraction module, configured to receive a data platform construction instruction, determine a corpus channel set based on the data platform construction instruction, successively extract corpus channels in the corpus channel set, obtain an agricultural product text set in the corpus channel, and successively extract agricultural product texts in the agricultural product text set, where the agricultural product text includes agricultural product introduction articles; An agricultural product segment segmentation module, configured to segment the agricultural product text to obtain an agricultural product sentence set, perform multi-dimensional key value analysis on each agricultural product sentence in the agricultural product sentence set to obtain a sorting key value set, an information key value set, and a title key value set, calculate a comprehensive key value set of the agricultural product sentence set based on the sorting key value set, the information key value set, and the title key value set, screen out a key agricultural product sentence set in the agricultural product sentence set according to a preset standard key value and the comprehensive key value set, merge the key agricultural product sentence sets to obtain a key agricultural product text, and perform overlapping segmentation on the key agricultural product text to obtain an agricultural product text segment set; An agricultural product vector calculation module is used to input the set of agricultural product text fragments into a pre-acquired Longformer model to obtain agricultural product feature vectors, and input the agricultural product feature vectors into a pre-trained LSTM network to obtain associated feature vectors. The associated feature vectors corresponding to each agricultural product text in the agricultural product text set are summarized to obtain an associated feature vector set, and the associated feature vector sets corresponding to each corpus channel in the corpus channel set are merged to obtain a target agricultural product vector set; An agricultural product platform construction module is used to identify multiple agricultural product names in the corpus channel set, perform data structuring on each agricultural product name in the multiple agricultural product names using the target agricultural product vector set to obtain an agricultural product structure data set, and store the agricultural product structure data set in a preset data space platform to obtain an agricultural product data space platform.

Images