A standard think tank data retrieval method and system based on big data

Abstract

This invention discloses a standard think tank data retrieval method and system based on big data. The method includes collecting user search data and think tank data from standard think tanks; preprocessing the user search data and the think tank data; semantically extracting core features from the search data based on keyness; embedding text information into the think tank data to obtain complete data; matching the core features and the complete data based on semantic similarity to obtain pooled data; combining and filtering the pooled data to obtain supplementary data; constructing a standard think tank data retrieval model based on the supplementary data; inputting the data to be retrieved into the standard think tank data retrieval model; and outputting the retrieval results. This method not only improves the accuracy of standard think tank data retrieval based on big data but also has good interpretability and can be directly applied to standard think tank data retrieval systems.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a standard think tank data retrieval method and system based on big data. Background Art

[0002] With the rapid development of information technology, big data has become an important resource in modern society. Data retrieval technology plays a vital role in all walks of life, especially in the field of standard think tanks. Standard think tanks refer to databases or knowledge bases that centrally store, manage and provide various types of standard information. Due to the huge, diverse and dynamic nature of standard information, how to efficiently retrieve data in standard think tanks has become a hot topic and difficulty in research.

[0003] The Standard Think Tank is a database that stores various standards, specifications, guidelines and other documents. The documents are usually complex in structure and rich in content. At the same time, due to the rapid update speed of standard documents, new standards continue to emerge, and old standards may be abandoned or modified, which further increases the difficulty of data retrieval. Traditional data retrieval methods often cannot meet the accuracy of standard think tank data retrieval. Therefore, how to efficiently and accurately retrieve valuable information from these data has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide a standard think tank data retrieval method based on big data.

[0005] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0006] The present invention comprises the following steps:

[0007] Collecting user search data and think tank data of a standard think tank, and preprocessing the user search data and the think tank data;

[0008] Perform semantic extraction on the search data according to the criticality to obtain core features, and perform text information embedding on the think tank data to obtain complete data;

[0009] Matching the core features and the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data;

[0010] A standard think tank data retrieval model is constructed based on the supplementary data, the data to be retrieved is input into the standard think tank data retrieval model, and the retrieval results are output.

[0011] Furthermore, the method of performing semantic extraction on the search data to obtain core features according to the criticality includes:

[0012] Use word clustering algorithm to extract named entities from search data, obtain entity vectors, and input entity vectors into semantic extraction model;

[0013] Get the entity vector information of the location and the semantic vector of the relationship, and calculate the judgment probability of the semantic relationship:

[0014]

[0015] The judgment probability of the semantic relation w is The bias of semantic relation w is v w , the weight of the semantic relation w is η w , the index value matrix is ​​G, the entity value matrix is ​​U, the entity index value matrix is ​​H, the transpose of the matrix is ​​T, and the dimension of the entity index value matrix H is b H , the association between the semantic relation w and the named entity y is ζ(w,y), the activation function is softmax(·), the deviation coefficient is σ, and the natural constant is e;

[0016] The semantic relations with a probability greater than 0.763 are regarded as valid semantic relations.

[0017] Calculate the attention weights:

[0018]

[0019] The ath attention weight is The ath valid semantic relation is f a , the κth character of the ath span semantic vector is f a,κ , the key is x, and the dimension of key x is b x , the number of characters is M, and the correct probability of the a-th semantics is d a ;

[0020] Calculate the span semantic vector:

[0021]

[0022] The span semantic vector is f sn , through character-level granularity and deep semantic understanding capabilities, it captures fine-grained semantic information within the card semantic vector;

[0023] Calculate the probability of span vector and valid semantic relationship matching entity pair, the expression is:

[0024]

[0025] The valid semantic relation f a and the span semantic vector f sn The probability of matching entity pairs is Q(f sn ,w a);

[0026] Given the loss function, the expression is:

[0027]

[0028] The number of valid semantic relations is Valid semantic relation f a The judgment probability is

[0029] The span semantic vector with a loss function value less than 0.17 is output as the core feature.

[0030] Furthermore, the method of embedding text information into the think tank data to obtain complete data includes:

[0031] Collect the forward block core information and current module of the think tank data after segmentation, obtain the entities of the forward block core information, and calculate and improve the entity embedding vector based on the entities of the forward block core information:

[0032]

[0033] The c-th entity embedding vector of the ι-th layer is The latent text vector is E τ , the embedding matrix of the entity is expressed as The number of entities is N, the transpose is T, and the bias is δ τ , the number of layers is The attention score of the cth entity in the ιth layer is The cth improved entity embedding vector is

[0034] Calculate the attention score:

[0035]

[0036] The parameter of the attention function is μ, and the jump embedding is represented by z ι ;

[0037] The decision tree algorithm is used to extract the semantic relationship of the core information of the forward block, obtain the semantic relationship vector, and calculate the comprehensive text information:

[0038]

[0039] The comprehensive text information is The cth improved entity embedding vector of the ιth layer is The semantic relationship vector of the ιth layer is K ι ;

[0040] The comprehensive text information is embedded in the current module, and the embedded current module outputs complete data.

[0041] Furthermore, the method of matching the core features and the complete data according to semantic similarity to obtain pooled data includes:

[0042] The core features and complete data are represented by bag-of-words to obtain the core feature vector and complete data vector respectively;

[0043] The core feature vector and the complete data vector are matrix decomposed by using truncated singular decomposition to obtain the word semantic vector of the core feature vector and the text semantic vector of the complete data vector;

[0044] Calculate the cosine similarity between the word semantic vector and the text semantic vector:

[0045]

[0046] The word semantic vector group is u, and the rth text semantic vector is k r , word semantic vector group u and text semantic vector k r The cosine similarity is The semantic vector u of the i-th word in the m-th dimension i,m , the number of word semantic vectors is n, and the rth text semantic vector in the mth dimension is k r,m , the number of dimensions is P;

[0047] Calculate the standard deviation:

[0048]

[0049] The semantic vector of the i-th word is in the P-dimensional semantic space. The dimension attribute value is The rth text semantic vector is in the P-dimensional semantic space. The average value of the dimension attribute is The attribute value of the Pth dimension of the word semantic vector is uf P , the rth text semantic vector is in the P-dimensional semantic space The dimension attribute value is Attribute valueuf P and attribute values The standard deviation of

[0050] Calculate the information entropy of the text semantic vector:

[0051]

[0052] The attribute value The information entropy is Attribute average The number of word semantic vectors with zero attributes is Property Value The number of text semantic vectors whose attributes are not zero is Property Value The number of text semantic vectors with zero attributes is

[0053] Calculate the distinction between text semantics and word semantics:

[0054]

[0055] The distinction between text semantics and word semantics is ψ, and the semantic similarity between the word semantic vector and the text semantic vector is calculated:

[0056]

[0057] The first hyperparameter is δ1, the second hyperparameter is δ2, and the feature word of the rth text semantic vector is uy r , the characteristic word of the semantic vector of the i-th word in the P-dimensional semantic space is uy i,P , the feature word of the rth text semantic vector in the P-dimensional semantic space is ky r,P , the frequency of word semantics in text semantics is V(uy i,P ,ky r,P ), the average word length of the text semantic text is avgl, the word semantic vector u and the text semantic vector k r The semantic similarity is

[0058] The complete data with semantic similarity greater than 0.673 are output as pooled data.

[0059] Furthermore, the method of combining and screening the pooled data to obtain supplementary data comprises:

[0060] Randomly combine the pooled data to obtain combined sentences, and calculate the semantic deviation between the combined sentences and the retrieved data:

[0061]

[0062] The jth combination sentence is Retrieve data for Combining sentences and retrieve data The semantic deviation of Combining sentences The semantic vector of Retrieve Data The semantic vector of Combining sentences and retrieve data The semantic similarity is

[0063] The pooled data with a semantic deviation less than 0.091 are output as supplementary data.

[0064] Furthermore, the method for constructing a standard think tank data retrieval model based on the supplementary data includes:

[0065] Standard think tank data retrieval models include random forest algorithms, deep learning algorithms, and graph retrieval enhancement generation algorithms;

[0066] The random forest algorithm randomly divides user search data into training data and test data in a ratio of 6:1;

[0067] The deep learning algorithm retains historical information through a loop structure, captures the dependency rules of different named entities in the training data, and extracts the semantic themes of the training data based on the dependency rules;

[0068] The graph retrieval enhancement generation algorithm constructs a knowledge graph and uses a graph machine learning algorithm to semantically aggregate semantic topics, identify and query the topic modules after semantic aggregation, and output retrieval results;

[0069] The test data is input into the standard think tank data retrieval model, the predicted retrieval results are output, the semantic variance between the predicted retrieval results and the actual retrieval results is calculated, and the hyperparameters of the standard think tank data retrieval model are adjusted according to the semantic variance until the variance is less than 0.17.

[0070] The second aspect is a standard think tank data retrieval system based on big data, comprising:

[0071] Data collection module: used to collect user search data and think tank data of the standard think tank, and pre-process the user search data and the think tank data;

[0072] Data processing module: used to perform semantic extraction on the search data according to the criticality to obtain core features, and to embed text information on the think tank data to obtain complete data;

[0073] Retrieval and matching module: used for matching the core features with the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data;

[0074] Model building module: used to build a standard think tank data retrieval model based on the supplementary data, input the data to be retrieved into the standard think tank data retrieval model, and output the retrieval results.

[0075] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0076] A processor; and a memory arranged to store computer executable instructions, which when executed cause the processor to perform the method steps described in the first aspect.

[0077] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the method steps described in the first aspect.

[0078] The beneficial effects of the present invention are:

[0079] The present invention is a standard think tank data retrieval method and system based on big data. Compared with the prior art, the present invention has the following technical effects:

[0080] The present invention can improve the accuracy of standard think tank data retrieval based on big data through preprocessing, semantic extraction, text information embedding, data matching, combined screening and model construction steps, thereby improving the precision of standard think tank data retrieval based on big data, optimizing standard think tank data retrieval based on big data, greatly saving resources and improving work efficiency, and can realize intelligent retrieval of standard think tank data based on big data, and perform retrieval correction on standard think tank data retrieval based on big data in real time, which is of great significance to standard think tank data retrieval based on big data, can adapt to different standards of standard think tank data retrieval based on big data and different requirements of standard think tank data retrieval based on big data, and has a certain universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flowchart of the steps of a standard think tank data retrieval method based on big data in the present invention;

[0082] Figure 2 It is a schematic diagram of the structure of an electronic device in an embodiment of this specification. DETAILED DESCRIPTION

[0083] The present invention is further described below by means of specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0084] The present invention provides a standard think tank data retrieval method and system based on big data, comprising the following steps:

[0085] like Figure 1 As shown, in this embodiment, the following steps are included:

[0086] Collecting user search data and think tank data of a standard think tank, and preprocessing the user search data and the think tank data;

[0087] In the actual evaluation, user search data includes historical search data, current search data, and search habits; the current search data of a user is the charging standard of electric vehicles;

[0088] Perform semantic extraction on the search data according to the criticality to obtain core features, and perform text information embedding on the think tank data to obtain complete data;

[0089] In the actual evaluation, the core features are electric vehicles and charging standards;

[0090] Matching the core features and the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data;

[0091] In the actual evaluation, the pool configuration data are electric vehicles, charging standards, 2024 electric vehicle charging standards, national electric vehicle charging standards, international electric vehicle charging standards, GB / T XXXX-XXXX electric vehicle conductive charging system, and the supplementary data are 2024 electric vehicle charging standards, national electric vehicle charging standards, and international electric vehicle charging standards;

[0092] Constructing a standard think tank data retrieval model based on the supplementary data, inputting the data to be retrieved into the standard think tank data retrieval model, and outputting the retrieval results;

[0093] In the actual assessment, when the current search data is food safety testing, the search results are XX / TZZZZ-2020 Technical Specifications for Food Safety Testing and XX / T AAAA-2022 General Requirements for Food Safety Rapid Detection Methods.

[0094] In this embodiment, the method of performing semantic extraction on the search data to obtain core features according to the criticality includes:

[0095] Use word clustering algorithm to extract named entities from search data, obtain entity vectors, and input entity vectors into semantic extraction model;

[0096] Get the entity vector information of the location and the semantic vector of the relationship, and calculate the judgment probability of the semantic relationship:

[0097]

[0098] The judgment probability of the semantic relation w is The bias of semantic relation w is v w , the weight of the semantic relation w is η w , the index value matrix is ​​G, the entity value matrix is ​​U, the entity index value matrix is ​​H, the transpose of the matrix is ​​T, and the dimension of the entity index value matrix H is b H, the association between the semantic relation w and the named entity y is ζ(w,y), the activation function is softmax(·), the deviation coefficient is σ, and the natural constant is e;

[0099] The semantic relations with a probability greater than 0.763 are regarded as valid semantic relations.

[0100] Calculate the attention weights:

[0101]

[0102] The ath attention weight is The ath valid semantic relation is f a , the κth character of the ath span semantic vector is f a,κ , the key is x, and the dimension of key x is b x , the number of characters is M, and the correct probability of the a-th semantics is d a ;

[0103] Calculate the span semantic vector:

[0104]

[0105] The span semantic vector is f sn , through character-level granularity and deep semantic understanding capabilities, it captures fine-grained semantic information within the card semantic vector;

[0106] Calculate the probability of span vector and valid semantic relationship matching entity pair, the expression is:

[0107]

[0108] The valid semantic relation f a and the span semantic vector f sn The probability of matching entity pairs is Q(f sn ,w a );

[0109] Given the loss function, the expression is:

[0110]

[0111] The number of valid semantic relations is Valid semantic relation f a The judgment probability is

[0112] The span semantic vector with a loss function value less than 0.17 is output as the core feature.

[0113] In this embodiment, the method of embedding text information into the think tank data to obtain complete data includes:

[0114] Collect the forward block core information and current module of the think tank data after segmentation, obtain the entities of the forward block core information, and calculate and improve the entity embedding vector based on the entities of the forward block core information:

[0115]

[0116] The c-th entity embedding vector of the ι-th layer is The latent text vector is E τ , the embedding matrix of the entity is expressed as The number of entities is N, the transpose is T, and the bias is δ τ , the number of layers is The attention score of the cth entity in the ιth layer is The cth improved entity embedding vector is

[0117] Calculate the attention score:

[0118]

[0119] The parameter of the attention function is μ, and the jump embedding is represented by z ι ;

[0120] The decision tree algorithm is used to extract the semantic relationship of the core information of the forward block, obtain the semantic relationship vector, and calculate the comprehensive text information:

[0121]

[0122] The comprehensive text information is The cth improved entity embedding vector of the ιth layer is The semantic relationship vector of the ιth layer is K ι ;

[0123] The comprehensive text information is embedded in the current module, and the embedded current module outputs complete data.

[0124] In this embodiment, the method for obtaining pooled data by matching the core features with the complete data according to semantic similarity includes:

[0125] The core features and complete data are represented by bag-of-words to obtain the core feature vector and complete data vector respectively;

[0126] The core feature vector and the complete data vector are matrix decomposed by truncated singular decomposition to obtain the word semantic vector of the core feature vector and the text semantic vector of the complete data vector;

[0127] Calculate the cosine similarity between the word semantic vector and the text semantic vector:

[0128]

[0129] The word semantic vector group is u, and the rth text semantic vector is k r , word semantic vector group u and text semantic vector k r The cosine similarity is The semantic vector u of the i-th word in the m-th dimension i,m , the number of word semantic vectors is n, and the rth text semantic vector in the mth dimension is k r,m , the number of dimensions is P;

[0130] Calculate the standard deviation:

[0131]

[0132] The semantic vector of the i-th word is in the P-dimensional semantic space. The dimension attribute value is The rth text semantic vector is in the P-dimensional semantic space. The average value of the dimension attribute is The attribute value of the Pth dimension of the word semantic vector is uf P , the rth text semantic vector is in the P-dimensional semantic space The dimension attribute value is Attribute valueuf P and attribute values The standard deviation of

[0133] Calculate the information entropy of the text semantic vector:

[0134]

[0135] The attribute value The information entropy is Attribute average The number of word semantic vectors with zero attributes is Property Value The number of text semantic vectors whose attributes are not zero is Property Value The number of text semantic vectors with zero attributes is

[0136] Calculate the distinction between text semantics and word semantics:

[0137]

[0138] The distinction between text semantics and word semantics is ψ, and the semantic similarity between the word semantic vector and the text semantic vector is calculated:

[0139]

[0140] The first hyperparameter is δ1, the second hyperparameter is δ2, and the feature word of the rth text semantic vector is uy r , the characteristic word of the semantic vector of the i-th word in the P-dimensional semantic space is uy i,P , the feature word of the rth text semantic vector in the P-dimensional semantic space is ky r,P , the frequency of word semantics in text semantics is V(uy i,P ,ky r,P ), the average word length of the text semantic text is avgl, the word semantic vector u and the text semantic vector k r The semantic similarity is

[0141] The complete data with semantic similarity greater than 0.673 are output as pooled data.

[0142] In this embodiment, the method of combining and screening the pooled data to obtain supplementary data includes:

[0143] Randomly combine the pooled data to obtain combined sentences, and calculate the semantic deviation between the combined sentences and the retrieved data:

[0144]

[0145] The jth combination sentence is Retrieve data for Combining sentences and retrieve data The semantic deviation of Combining sentences The semantic vector of Retrieve Data The semantic vector of Combining sentences and retrieve data The semantic similarity is

[0146] The pooled data with a semantic deviation less than 0.091 are output as supplementary data.

[0147] In this embodiment, the method for constructing a standard think tank data retrieval model based on the supplementary data includes:

[0148] Standard think tank data retrieval models include random forest algorithms, deep learning algorithms, and graph retrieval enhancement generation algorithms;

[0149] The random forest algorithm randomly divides user search data into training data and test data in a ratio of 6:1;

[0150] The deep learning algorithm retains historical information through a loop structure, captures the dependency rules of different named entities in the training data, and extracts the semantic themes of the training data based on the dependency rules;

[0151] The graph retrieval enhancement generation algorithm constructs a knowledge graph and uses a graph machine learning algorithm to semantically aggregate semantic topics, identify and query the topic modules after semantic aggregation, and output retrieval results;

[0152] The test data is input into the standard think tank data retrieval model, the predicted retrieval results are output, the semantic variance between the predicted retrieval results and the actual retrieval results is calculated, and the hyperparameters of the standard think tank data retrieval model are adjusted according to the semantic variance until the variance is less than 0.17.

[0153] The second aspect is a standard think tank data retrieval system based on big data, comprising:

[0154] Data collection module: used to collect user search data and think tank data of the standard think tank, and pre-process the user search data and the think tank data;

[0155] Data processing module: used to perform semantic extraction on the search data according to the criticality to obtain core features, and to embed text information on the think tank data to obtain complete data;

[0156] Retrieval and matching module: used for matching the core features with the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data;

[0157] Model building module: used to build a standard think tank data retrieval model based on the supplementary data, input the data to be retrieved into the standard think tank data retrieval model, and output the retrieval results.

[0158] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 2 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0159] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0160] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0161] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a standard think tank data retrieval device based on big data at the logical level. The processor executes the program stored in the memory and is specifically used to execute any of the above-mentioned standard think tank data retrieval methods based on big data.

[0162] The above application Figure 1A standard think tank data retrieval method based on big data disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0163] The electronic device may also perform Figure 1 A standard think tank data retrieval method based on big data is implemented Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0164] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple applications, they execute any of the aforementioned standard think tank data retrieval methods based on big data.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A standard think tank data retrieval method based on big data, characterized in that: The following steps are involved: Collecting user search data and think tank data of a standard think tank, and preprocessing the user search data and the think tank data; According to the criticality, semantic extraction is performed on the search data to obtain core features, and text information is embedded in the think tank data to obtain complete data; including: Use word clustering algorithm to extract named entities from search data, obtain entity vectors, and input entity vectors into semantic extraction model; Get the entity vector information of the location and the semantic vector of the relationship, and calculate the judgment probability of the semantic relationship: The judgment probability of the semantic relation w is The bias of semantic relation w is v w , the weight of the semantic relation w is η w , the index value matrix is ​​G, the entity value matrix is ​​U, the entity index value matrix is ​​H, the transpose of the matrix is ​​T, and the dimension of the entity index value matrix H is b H , the association between the semantic relation w and the named entity y is ζ(w,y), the activation function is softmax(·), the deviation coefficient is σ, and the natural constant is e; The semantic relations with a probability greater than 0.763 are regarded as valid semantic relations. Calculate the attention weights: The ath attention weight is The ath valid semantic relation is f a , the κth character of the ath span semantic vector is f a,κ , the key is x, and the dimension of key x is b x , the number of characters is M, and the correct probability of the a-th semantics is d a ; Calculate the span semantic vector: The span semantic vector is f sn , through character-level granularity and deep semantic understanding capabilities, it captures fine-grained semantic information within the card semantic vector; Calculate the probability of span vector and valid semantic relationship matching entity pair, the expression is: The valid semantic relation f a and the span semantic vector f sn The probability of matching entity pairs is Q(f sn ,w a ); Given the loss function, the expression is: The number of valid semantic relations is Valid semantic relation f a The judgment probability is Output the span semantic vector with loss function value less than 0.17 as the core feature; Collect the forward block core information and current module of the think tank data after segmentation, obtain the entities of the forward block core information, and calculate and improve the entity embedding vector based on the entities of the forward block core information: The c-th entity embedding vector of the ι-th layer is The latent text vector is E τ , the embedding matrix of the entity is expressed as The number of entities is N, the transpose is T, and the bias is δ τ , the number of layers is The attention score of the cth entity in the ιth layer is The cth improved entity embedding vector is Calculate the attention score: The parameter of the attention function is μ, and the jump embedding is represented by z ι ; The decision tree algorithm is used to extract the semantic relationship of the core information of the forward block, obtain the semantic relationship vector, and calculate the comprehensive text information: The comprehensive text information is The cth improved entity embedding vector of the ιth layer is The semantic relationship vector of the ιth layer is K ι ; Embed the comprehensive text information into the current module, and output the complete data of the embedded current module; Matching the core features and the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data; A standard think tank data retrieval model is constructed based on the supplementary data, the data to be retrieved is input into the standard think tank data retrieval model, and the retrieval results are output.

2. According to the standard think tank data retrieval method based on big data as described in claim 1, it is characterized in that: The method for obtaining pooled data by matching the core features with the complete data according to semantic similarity includes: The core features and complete data are represented by bag-of-words to obtain the core feature vector and complete data vector respectively; The core feature vector and the complete data vector are matrix decomposed by using truncated singular decomposition to obtain the word semantic vector of the core feature vector and the text semantic vector of the complete data vector; Calculate the cosine similarity between the word semantic vector and the text semantic vector: The word semantic vector group is u, and the rth text semantic vector is k r , word semantic vector group u and text semantic vector k r The cosine similarity is The semantic vector u of the i-th word in the m-th dimension i,m , the number of word semantic vectors is n, and the rth text semantic vector in the mth dimension is k r,m , the number of dimensions is P; Calculate the standard deviation: The l-th dimension attribute value of the semantic vector of the i-th word in the P-dimensional semantic space is The average value of the l-th dimension attribute of the r-th text semantic vector in the P-dimensional semantic space is The attribute value of the Pth dimension of the word semantic vector is uf P , the l-th dimension attribute value of the r-th text semantic vector in the P-dimensional semantic space is Attribute valueuf P and attribute values The standard deviation of Calculate the information entropy of the text semantic vector: The attribute value The information entropy is Attribute average The number of word semantic vectors with zero attributes is Property Value The number of text semantic vectors whose attributes are not zero is Property Value The number of text semantic vectors with zero attributes is Calculate the distinction between text semantics and word semantics: The distinction between text semantics and word semantics is ψ, and the semantic similarity between the word semantic vector and the text semantic vector is calculated: The first hyperparameter is δ1, the second hyperparameter is δ2, and the feature word of the rth text semantic vector is uy r , the characteristic word of the semantic vector of the i-th word in the P-dimensional semantic space is uy i,P , the feature word of the rth text semantic vector in the P-dimensional semantic space is ky r,P , the frequency of word semantics in text semantics is The average word length of the text semantics text is avgl, the word semantic vector u and the text semantic vector k r The semantic similarity is The complete data with semantic similarity greater than 0.673 are output as pooled data.

3. According to the standard think tank data retrieval method based on big data as described in claim 1, it is characterized in that: The method of combining and screening the pooled data to obtain supplementary data comprises: Randomly combine the pooled data to obtain combined sentences, and calculate the semantic deviation between the combined sentences and the retrieved data: The jth combination sentence is Retrieve data for Combining sentences and retrieve data The semantic deviation of Combining sentences The semantic vector of Retrieve Data The semantic vector of Combining sentences and retrieve data The semantic similarity is The pooled data with a semantic deviation less than 0.091 are output as supplementary data.

4. According to the standard think tank data retrieval method based on big data as described in claim 1, it is characterized in that: The method for constructing a standard think tank data retrieval model based on the supplementary data includes: Standard think tank data retrieval models include random forest algorithms, deep learning algorithms, and graph retrieval enhancement generation algorithms; The random forest algorithm randomly divides user search data into training data and test data in a ratio of 6:1; The deep learning algorithm retains historical information through a loop structure, captures the dependency rules of different named entities in the training data, and extracts the semantic themes of the training data based on the dependency rules; The graph retrieval enhancement generation algorithm constructs a knowledge graph and uses a graph machine learning algorithm to semantically aggregate semantic topics, identify and query the topic modules after semantic aggregation, and output retrieval results; The test data is input into the standard think tank data retrieval model, the predicted retrieval results are output, the semantic variance between the predicted retrieval results and the actual retrieval results is calculated, and the hyperparameters of the standard think tank data retrieval model are adjusted according to the semantic variance until the variance is less than 0.

17.

5. A standard think tank data retrieval system based on big data, used to execute the method described in any one of claims 1 to 4, characterized in that: include: Data collection module: used to collect user search data and think tank data of the standard think tank, and pre-process the user search data and the think tank data; Data processing module: used to perform semantic extraction on the search data according to the criticality to obtain core features, and to embed text information on the think tank data to obtain complete data; including: Use word clustering algorithm to extract named entities from search data, obtain entity vectors, and input entity vectors into semantic extraction model; Get the entity vector information of the location and the semantic vector of the relationship, and calculate the judgment probability of the semantic relationship: The judgment probability of the semantic relation w is The bias of semantic relation w is v w , the weight of the semantic relation w is η w , the index value matrix is ​​G, the entity value matrix is ​​U, the entity index value matrix is ​​H, the transpose of the matrix is ​​T, and the dimension of the entity index value matrix H is b H , the association between the semantic relation w and the named entity y is ζ(w,y), the activation function is softmax(·), the deviation coefficient is σ, and the natural constant is e; The semantic relations with a probability greater than 0.763 are regarded as valid semantic relations. Calculate the attention weights: The ath attention weight is The ath valid semantic relation is f a , the κth character of the ath span semantic vector is f a,κ , the key is x, and the dimension of key x is b x , the number of characters is M, and the correct probability of the a-th semantics is d a ; Calculate the span semantic vector: The span semantic vector is f sn , through character-level granularity and deep semantic understanding capabilities, it captures fine-grained semantic information within the card semantic vector; Calculate the probability of span vector and valid semantic relationship matching entity pair, the expression is: The valid semantic relation f a and the span semantic vector f sn The probability of matching entity pairs is Q(f sn ,w a ); Given the loss function, the expression is: The number of valid semantic relations is Valid semantic relation f a The judgment probability is Output the span semantic vector with loss function value less than 0.17 as the core feature; Collect the forward block core information and current module of the think tank data after segmentation, obtain the entities of the forward block core information, and calculate and improve the entity embedding vector based on the entities of the forward block core information: The c-th entity embedding vector of the ι-th layer is The latent text vector is E τ , the embedding matrix of the entity is expressed as The number of entities is N, the transpose is T, and the bias is δ τ , the number of layers is The attention score of the cth entity in the ιth layer is The cth improved entity embedding vector is Calculate the attention score: The parameter of the attention function is μ, and the jump embedding is represented by z ι ; The decision tree algorithm is used to extract the semantic relationship of the core information of the forward block, obtain the semantic relationship vector, and calculate the comprehensive text information: The comprehensive text information is The cth improved entity embedding vector of the ιth layer is The semantic relationship vector of the ιth layer is K ι ; Embed the comprehensive text information into the current module, and output the complete data of the embedded current module; Retrieval and matching module: used for matching the core features with the complete data according to semantic similarity to obtain pooled data, and combining and screening the pooled data to obtain supplementary data; Model building module: used to build a standard think tank data retrieval model based on the supplementary data, input the data to be retrieved into the standard think tank data retrieval model, and output the retrieval results.

6. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute the method according to any one of claims 1 to 4.

Images