Information adaptive distribution strategy and process automatic arrangement system of message middleware

Abstract

The application discloses a message middleware information adaptive distribution strategy and a process automatic arrangement method, and belongs to the field of data processing.The method is user demand-oriented, and an information distribution adaptive strategy is designed to change a traditional user passive subscription message mode, so that a user actively receives interested information.User interest labels are obtained based on an LDA topic modeling technology, and a user interest knowledge graph is constructed based on Neo4j.A mapping relationship between global semantic information of the user interest knowledge graph and global semantic information of to-be-distributed message texts is constructed based on an attention mechanism, information content semantic adaptive distribution is realized, an immune optimization algorithm is used to construct information modal adaptive distribution, and the channel utilization rate of information distribution is improved.Finally, an information adaptive distribution system and process automatic arrangement are constructed based on a process engine technology, the modularization customization of a distribution data node is realized, and the automatic generation and arrangement of a distribution process are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an information adaptive distribution strategy of a message middleware and a process automatic arrangement system. BACKGROUND

[0002] There are many technical ways to realize data sharing and data distribution among nodes in a system, the most important of which are the following three: federal database system, data warehouse and middleware technology. But the most popular one is the middleware technology, which can shield the differences between various heterogeneous data, realize the interconnection, intercommunication and interoperation between nodes, and provide users with a unified interface to access heterogeneous data. However, with the increasing network conditions of information and data, there are two problems in sending information by using the traditional subscription / distribution mode: passive reception of the subscription end leads to an increasing amount of junk information; and the point-to-point topic matching of the publishing user and the subscription user in the data link domain has the problem of low efficiency of channel resource utilization. Therefore, the present application proposes an information adaptive distribution strategy of a message middleware and a process automatic arrangement system, which aims to solve the problems of passive reception of users in the traditional subscription distribution mechanism, the inability of the message content obtained by the user to adapt to the user's demand, and the inflexible information distribution strategy causing the user terminal to be difficult to realize on-demand information acquisition. SUMMARY

[0003] The present application aims to solve the problems of passive reception of the subscription end leading to an increasing amount of junk information, and the point-to-point topic matching of the publishing user and the subscription user in the data link domain having the problem of low efficiency of channel resource utilization.

[0004] To solve the problems existing in the prior art, the present application proposes an information adaptive distribution strategy of a message middleware and a process automatic arrangement system, which can realize adaptive distribution of corresponding messages based on user interest and demand, and can realize automatic process arrangement of data distribution according to network conditions and adaptive distribution of different message modalities. The adaptive information distribution strategy has the following specific steps:

[0005] Step S10: A topic content extraction model based on TF-IDF, TextRank and LDA algorithm is constructed to obtain the user interested topic content of the historical message text received by the user, and further feature selection is performed based on the AdaBoost algorithm to obtain the user interested topic word;

[0006] Step S20: A triple of <user, relationship, interest label word group> is constructed for the user and the interest topic group, and a user interest knowledge graph is constructed based on the Neo4j tool;

[0007] Step S30: based on the Bert model and the attention mechanism, a semantic mapping relationship between the user interest knowledge graph and the to-be-received message is constructed, and information content is adaptively distributed to users interested in the information content;

[0008] Step S40: in a network change or limited scene, based on user priority and network link state, an information modal (text, graph, video, voice, etc.) adaptive distribution strategy is realized based on an immune optimization algorithm, an optimal combination of information distribution modes is selected, and the utilization efficiency of the channel is improved;

[0009] Step S50: based on ETL and process engine design, a data adaptive distribution module is designed, and an automatic arrangement system of the information distribution process is constructed.

[0010] In the above technical solution, the specific steps of step S10 are as follows:

[0011] Step S101, the TF-IDF algorithm uses a word segmentation tool to perform word segmentation processing on the input document, removes stop words and low-frequency words, and only retains noun words as a candidate word set. The TF-IDF value of the candidate word is calculated, wherein the TF value (Term Frequency) calculation formula is as follows:

[0012]

[0013] Wherein, m is the number of times the word w appears in the text, and n is the total number of words in the text. The IDF value (Inverse Document Frequency) of the candidate word, that is, the inverse document frequency, is calculated by dividing the total number of files N by the number of files M containing the word, and then taking the logarithm of the quotient:

[0014]

[0015] Step S102, the TextRank algorithm uses a word segmentation tool to perform word segmentation processing on the input document, removes stop words and low-frequency words, and only retains noun words as a candidate word set. The retained words are constructed into a semantic relationship undirected graph, and the TextRank value of the candidate word is calculated:

[0016]

[0017] Wherein, is a damping coefficient, set to 0.85, is a word set pointing to the word , is a word set pointed to by the word , is the weight of the edge between the word node and the word node ;

[0018] Step S103, the LDA algorithm uses a word segmentation tool to perform word segmentation processing on the input document, removes stop words and low-frequency words, and only retains noun words as a candidate word set. A P (word / topic) is calculated according to the probability of the topic word appearing in the topic, a P (topic / document) is calculated according to the probability of a topic appearing in the document, and the probability of the word appearing is calculated according to P (word / topic) and P (topic / document) :

[0019]

[0020] Step S104, the weak classifiers (TF-IDF, TextRank, LDA algorithm classifier) are combined using the AdaBoost algorithm to construct a strong classifier, the idea of which is to change the weight of the training samples in the data set to learn multiple classifiers, and the performance of the classification is improved by integrating the classifiers according to certain rules; finally, the obtained topic words are taken as the label words of interest of the user.

[0021] In the above technical solution, the specific steps of step S20 are as follows:

[0022] Step S201, extract the named entity words (users, user interested topic words) and relationship words, and add them to the knowledge graph database, and import Neo4j to realize the visualization construction of the knowledge graph.

[0023] In the above technical solution, the specific steps of step S30 are as follows:

[0024] Step S301, the message text is originally represented as , , where n is the number of words in the text, the interest label entity word of the knowledge graph is represented as , and the sentence context feature vector representation , is obtained using the pre-trained language model BERT.

[0025] Step S302, based on the attention mechanism, the cosine similarity between the message text and the interest label entity word of the knowledge graph is calculated, and the calculation formula is as follows:

[0026]

[0027] Wherein, represents the user interest feature representation, represents the message text feature representation.

[0028] Step S303, based on the semantic mapping model of the user interest word and the message text, set the cosine similarity threshold , and the value greater than Value of the user-message composition to be distributed list .

[0029] In the technical solution, the specific steps of step S40 are as follows:

[0030] Step S401, based on the network probe, the user channel communication state is acquired, and the to-be-distributed list is used as the basis The to-be-distributed information contains the modalities (text, picture, video, etc.) and specifications of the data.

[0031] Step S402, the user and the to-be-distributed data are defined as [text size, picture size, video size, voice size] = .

[0032] Step S403, based on the immune algorithm, the optimal sending data list is selected .

[0033] In the technical solution, the specific steps of step S60 are as follows:

[0034] Step S501, the ETL tool is used to process the data conversion process in the following manner: including the extraction, cleaning conversion, and loading of test data.

[0035] Step S502, the process definition file is designed to contain the information required for visual display of each node (node position, size, shape, etc.).

[0036] Step S503, the processing nodes and flow transfer mode are defined based on the workflow engine.

[0037] Step S504, the required process definition file is saved and deployed to the running environment of the workflow engine, and the process is executed according to the identification information in the currently parsed and converted data to determine the subsequent data processing process.

[0038] The beneficial effects of the present application are:

[0039] Firstly, the LDA topic modeling technology is used to obtain the theme content of interest in the user historical messages, and a user interest knowledge graph is constructed based on Neo4j; secondly, the mapping relationship between the global semantic information of the user interest knowledge graph and the global semantic information of the to-be-distributed message text is constructed based on the attention mechanism, realizing semantic adaptive distribution of information content; thirdly, considering the scenario of limited network state, an immune optimization algorithm is used to construct adaptive distribution of information modalities (text, picture, video, voice, etc.), so as to improve the channel utilization rate of information distribution; finally, based on the process engine technology, an information adaptive distribution system and process automatic arrangement are constructed, realizing modular customization of distribution data node modules and automatic generation and arrangement of processes. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The overall technical flowchart of the present application.

[0041] Figure 2 The implementation technical roadmap of the information adaptive distribution strategy of the present application.

[0042] Figure 3 The implementation technical roadmap of the information distribution automatic flowchart of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] Referring to Figure 1 The present application provides an overall technical solution:

[0045] The information source text is input into the information adaptive distribution strategy, the information distribution strategy performs content adaptive selection according to user interest, performs modal adaptive selection according to network condition, and finally constructs an information automatic distribution system through flowchart technology. The system is finally used for message distribution management of a message middleware.

[0046] Referring to Figure 2 The detailed steps of the information adaptive strategy are as follows:

[0047] Step 1: a topic content extraction model based on TF-IDF, TextRank and LDA algorithm is constructed as follows, which is used to obtain user interested topic content of historical message text received by the user, and further based on AdaBoost algorithm for feature selection, so as to obtain user interested topic words;

[0048] Step 1-1, the TF-IDF algorithm uses a word segmentation tool to perform word segmentation processing on the input document, removes stop words and low-frequency words, and only keeps the noun words as a candidate word set. The TF-IDF value of the candidate word is calculated, wherein the TF value (Term Frequency) calculation formula is as follows:

[0049]

[0050] Wherein, m is the number of times the word w appears in the text, and n is the total number of words in the text. The IDF value (Inverse Document Frequency) of the candidate word, i.e. the inverse document frequency, is calculated by dividing the total number of files N by the number of files M containing the word, and then taking the logarithm of the quotient:

[0051]

[0052] Step 1-2, the TextRank algorithm uses a word segmentation tool to process the input document, removes stop words and low-frequency words, and only keeps the noun words as the candidate word set. The retained words are constructed into a semantic relationship undirected graph, and the TextRank value of the candidate words is calculated:

[0053]

[0054] where d is the damping coefficient, usually set to 0.85, is the set of words pointing to word i, is the set of words pointing to. is the word node and the word node edge weight.

[0055] Step 1-3, the LDA algorithm uses a word segmentation tool to process the input document, removes stop words and low-frequency words, and only keeps the noun words as the candidate word set. According to the probability of the topic word appearing in the topic, a P (word / topic) is calculated, according to the probability of a topic appearing in the document, a P (topic / document) is calculated, and according to P (word / topic) and P (topic / document), the probability of the word appearing is calculated:

[0056]

[0057] Step 1-4, use AdaBoost algorithm to combine weak classifiers (TF-IDF, TextRank, LDA algorithm classifier) to construct strong classifier, the idea is to change the weight of the training samples in the data set to learn multiple classifiers, and integrate these classifiers according to certain rules to improve the performance of classification; Finally, the topic word obtained is taken as the label word of interest of the user.

[0058] Step 2: Build a triple <user, relationship, interest label word group> of user and interest theme group, and build a user interest knowledge graph based on Neo4j tool;

[0059] Step 2-1, extract named entity words (users, user interest topic words) and relationship words, and add them to the knowledge graph database, and import Neo4j to realize knowledge graph visualization construction.

[0060] Step 3: Based on the Bert model and attention mechanism, build the semantic mapping relationship between the user interest knowledge graph and the received message, and realize the adaptive distribution of information content to the interested users;

[0061] Step S301, the message text is originally represented as , is the number of words in the text, the interest label entity word of the knowledge graph is represented as , and the sentence context feature vector representation is obtained using the pre-trained language model BERT , .

[0062] Step 3-2, based on the attention mechanism, the cosine similarity between the message text and the interest label entity word of the knowledge graph is calculated, and the calculation formula is as follows:

[0063]

[0064] wherein, represents the user interest feature representation, represents the message text feature representation.

[0065] Step 3-3, based on the semantic mapping model of the user interest word and the message text, set the cosine similarity threshold , the user-message with a value greater than forms a list to be distributed .

[0066] Step 4: In the network change, limited scene, based on the user priority, network link state, based on the immune optimization algorithm to realize the information modal (text, graph, video, voice, etc.) adaptive distribution strategy, select the optimal combination of information distribution mode, improve the utilization efficiency of the channel;

[0067] Step 4-1, based on the network probe to obtain the user channel communication state, according to the to-be-distributed list to obtain the modal (text, picture, video, etc.) and its specification of the to-be-distributed information containing data.

[0068] Step 4-2, define the above user and to-be-distributed data as [text size, picture size, video size, voice size]= .

[0069] Step 4-3, based on the immune algorithm to select the optimal sending data list , the detailed steps are as follows:

[0070] ① Antigen recognition: input the objective function and various constraints as the antigen of the immune algorithm.

[0071] ② Initial antibody generation: randomly generate an initial antibody population.

[0072] ③ Affinity calculation: calculate the fitness value of the antibody.

[0073] ④ Immune processing: immune processing includes immune selection, cloning, variation and inhibition.

[0074] ⑤ Immune selection: selecting antibodies with higher affinity according to the affinity of antibodies.

[0075] ⑥ Cloning: copying the selected antibodies with higher affinity.

[0076] ⑦ Variation: cross and variation of cloned individuals to change their affinity.

[0077] ⑧ Inhibition: selecting the antibodies after variation and keeping the antibodies with higher affinity.

[0078] ⑨ Population refresh: combining the antibodies after immune selection and immune inhibition into a set, keeping the antibodies with higher affinity and making these antibodies enter a new population. The insufficient part in the new population is randomly generated to increase diversity.

[0079] Please refer to Figure 3 , the detailed steps of building an information adaptive distribution system based on process engine technology are as follows:

[0080] Step 5: Design a data adaptive distribution module based on ETL and process engine to build an automatic arrangement system of information distribution process.

[0081] Step 5-1, use ETL tools to process data conversion process: including extraction, cleaning and conversion, and loading of test data.

[0082] Step 5-2, design process definition file to contain the information needed for visual display of each node (node position, size, shape, etc.).

[0083] Step 5-3, define processing nodes and flow mode based on workflow engine, including:

[0084] ① Start node: indicates the start of a processing process. Start node can only have one.

[0085] ② End node: indicates the end of a processing process. There can be multiple end nodes, but there must be at least one (without end node, the program can run, but it must be unreasonable).

[0086] ③ Task node: core node, indicating an approval node. The process will be stalled at these task nodes, and API needs to be called to push the process forward. Task nodes include automatic tasks and manual tasks. We mainly process data, so we mainly use automatic tasks.

[0087] (4) Gateway node: this is a node for process flow control, for example, all nodes are audited by the gateway node, and only when all nodes are audited, the gateway node will continue to push forward, the gateway node can also be designed to have multiple exits.

[0088] Step 5-4: Save the required process definition file and deploy it to the running environment of the workflow engine, and execute the process according to the identification information in the current parsed and converted data to determine the subsequent data processing process.

[0089] An information adaptive distribution strategy and process automatic arrangement system of a message middleware can be used for message distribution management of the message middleware.

[0090] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some of the technical features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A message middleware information adaptive distribution strategy and automatic process orchestration system, characterized in that, The system implementation includes the following steps: Step S10: Construct a topic content extraction model based on TF-IDF, TextRank, and LDA algorithms to obtain user-interested topic content from historical message texts received by the user, and further perform feature filtering based on the AdaBoost algorithm to obtain user-interested topic words; Step S20: Construct triples of users and interest topic groups <user, relationship, interest tag phrase>, and build a user interest knowledge graph based on Neo4j tool; Step S30: Construct a semantic mapping relationship between a user interest knowledge graph and the message to be received based on the BERT model and attention mechanism, so as to realize the adaptive distribution of information content to users who are interested; Step S40: In scenarios with network changes and constraints, based on user priority and network link status, an adaptive information mode distribution strategy is implemented using an immune optimization algorithm to select the optimal combination of information distribution modes and improve channel utilization efficiency. Step S50: Based on ETL and process engine, design a data adaptive distribution module to build an automatic orchestration system for information distribution processes; The specific steps of step S30 are as follows: Step S301, the original representation of the message text is as follows: , The number of words in the text, and the interest tag entity words in the knowledge graph are represented as: And use the pre-trained language model BERT to obtain the sentence context feature vector representation. , ; Step S302: Based on the attention mechanism, calculate the cosine similarity between the message text and the interest tag entity words in the knowledge graph. The calculation formula is as follows: in, Represents user interest characteristics. Representation of message text features; Step S303: Based on the semantic mapping model between user interest words and message text, set a cosine similarity threshold. , will be greater than The value of the user-message composition is the list to be distributed. ; The specific steps of step S40 are as follows: Step S401: Obtain the user channel communication status based on the network probe, and according to the list to be distributed... Obtain the modality and specifications of the data contained in the information to be distributed; Step S402, define the user and the data to be distributed as [text size, image size, video size, voice size] = ; Step S403: Select the optimal list of data to be sent based on the immune algorithm. .

2. The message middleware adaptive distribution strategy and automatic process orchestration system according to claim 1, characterized in that: The specific steps of step S10 are as follows: Step S101: The TF-IDF algorithm uses a word segmentation tool to segment the input document, removes stop words and low-frequency words, and retains only noun words as the candidate word set. The TF-IDF value of the candidate words is calculated, and the formula for calculating the TF value is as follows: Where m is the number of times the word w appears in the text, and n is the total number of words in the text. The IDF value, or inverse document frequency, of the candidate word is calculated by dividing the total number of documents N by the number of documents M containing the word, and then taking the logarithm of the quotient. Step S102: The TextRank algorithm uses a word segmentation tool to segment the input document, removing stop words and low-frequency words, retaining only noun words as a candidate word set, constructing an undirected semantic relation graph from the retained words, and calculating the TextRank value of the candidate words. Where d is the damping coefficient, set to 0.

85. For pointing to words A collection of words, For words The set of words it points to. For word nodes With word nodes Edge weights; Step S103: The LDA algorithm uses a word segmentation tool to segment the input document, removing stop words and low-frequency words, retaining only noun words as the candidate word set. It calculates P(Word / Topic) based on the probability of a topic word appearing in that topic, and P(Topic / Text) based on the probability of a specific topic appearing in the document. Finally, it calculates the probability of word occurrence based on P(Word / Topic) and P(Topic / Text). Step S104: Use the AdaBoost algorithm to combine weak classifiers to construct a strong classifier. Change the weights of the training samples in the dataset to learn multiple classifiers. Then integrate these classifiers according to certain rules to improve classification performance. Finally, use the obtained topic words as the tag words that users are interested in.

3. The message middleware adaptive distribution strategy and automatic process orchestration system according to claim 1, characterized in that: The specific steps of step S20 are as follows: Step S201: Extract named entity words and relation words, add them to the knowledge graph database, and import them into Neo4j to realize the visualization construction of the knowledge graph. Named entity words include users and topics of interest to users.

4. The message middleware adaptive distribution strategy and automatic process orchestration system according to claim 1, characterized in that: The specific steps of step S50 are as follows: Step S501, use ETL to process the data transformation process: experimental data extraction, cleaning and transformation, loading; Step S502: Design a process definition file to include the information needed for the visualization of each node, including node position, size, and shape; Step S503: Define processing nodes and flow modes based on the workflow engine; Step S504: Save the required process definition file and deploy it to the workflow engine's runtime environment. The execution process determines the subsequent data processing flow based on the identification information in the currently parsed and transformed data.

Images