Distributed material compilation service system
By adopting a distributed task scheduling framework and modular classification model in computer-aided compilation systems, the performance bottlenecks and single-point failure problems of centralized architecture in large-scale data processing are solved, and efficient and reliable material compilation services are achieved.
Patent Information
- Application Number
- CN202411605697.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-02-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing computer-assisted compilation system adopts a centralized architecture and is difficult to deal with large-scale data processing, resulting in system performance bottlenecks and single-point failure problems.
A distributed material compilation service system was designed, and the distributed task scheduling framework was used to split the compilation tasks into multiple subtasks and process them in parallel between multiple work nodes. At the same time, a preset classification model and user historical preference information were introduced. Conduct modular classification.
Through distributed processing and modular classification, the compilation efficiency and system scalability are significantly improved, omissions, errors and duplication problems are reduced, and the quality and credibility of compilation results are improved.
Smart Images
Figure CN119440851A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed technology, and in particular to a distributed material compilation service system. Background Art
[0002] In modern society, with the rapid development of information technology and the sharp increase in the amount of information, material compilation is facing unprecedented challenges. Traditional material compilation methods often rely on manual operations, which are not only inefficient, but also difficult to ensure the quality and consistency of compilation. Especially when dealing with a large amount of complex information, manual compilation is often prone to omissions, errors and duplications, which seriously affects the accuracy and credibility of material compilation.
[0003] In order to solve these problems, people began to try to use computer technology to assist in the compilation of materials. However, most of the existing computer-assisted compilation systems adopt a centralized architecture, that is, all compilation tasks are completed on a central server. This architecture can barely cope with processing small-scale data, but when the amount of data increases, it will face performance bottlenecks and scalability issues. In addition, the centralized architecture is prone to single point failures. Once the central server fails, the entire compilation system will be paralyzed.
[0004] In view of this, the art needs a new distributed material compilation service system to solve the above problems. Summary of the invention
[0005] In order to solve the above technical problem, that is, to solve the problem that the existing computer-aided compilation system adopts a centralized architecture and is difficult to cope with the situation of large data volume, resulting in the system being prone to failure.
[0006] The present invention provides a distributed material compilation service system, the distributed material compilation service system comprising: A requirement acquisition module, which is used to obtain input material compilation requirement information; A primary information generation module, which is used to generate primary information of material compilation according to the material compilation requirement information; A selection and correction module, which is used to select and correct the primary information of material compilation, wherein the selection and correction module adopts a distributed task scheduling framework, and the primary information of material compilation is assigned to different working nodes for processing according to categories; A material compilation module, which is used to compile materials based on selected and revised information; In the process of material compilation, the compiled content is modularly classified according to the preset classification model and user historical preference information.
[0007] In some preferred embodiments, the selection and correction module includes a summary node, which is used to receive and summarize the information processed by all the working nodes, and send the summarized information to the material compilation module.
[0008] In certain preferred embodiments, the distributed material compilation service system also includes a distributed database, which is used to store the classification model, the user historical preference information, the material compilation demand information, the material compilation primary information and the material compilation results.
[0009] In some preferred embodiments, the step of generating primary information of material compilation according to the material compilation requirement information includes: Inputting the material compilation requirement information into a material compilation primary information generation model; The material compilation primary information is generated based on the material compilation primary information generation model, wherein the formula of the material compilation primary information generation model is: ; in, , First step in compiling primary information for the material Features, The number of features for which primary information is compiled for the material, , Compile requirements for materials Features, The number of characteristics for which the required information is compiled for the material, and are weight matrices, and are bias vectors, is the attention mechanism parameter, is the context information vector, To dynamically adjust parameters, is the regularization parameter, is the feature extraction function parameter, is the generator parameter, is the historical information fusion parameter.
[0010] In certain preferred embodiments, The formula is: ; in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of a feature.
[0011] In certain preferred embodiments, The formula is: ; in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of a feature is The relevant weight coefficient.
[0012] In certain preferred embodiments, the formula of the classification model is: ; in, , For the classification categories, is the number of classification categories in the classification results, , The information is selected and corrected. Features, is the number of features of the selected and modified information, is the classification weight matrix, is the classification bias vector, is the tree structure parameter, are the support vector machine parameters, are neural network parameters, is the fusion coefficient, Enhance the parameters for the feature.
[0013] In certain preferred embodiments, The formula is: ; in, is the tree structure parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the support vector machine parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the neural network parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, , and The selected and corrected information The weight coefficient associated with each feature.
[0014] In certain preferred embodiments, The formula is: ; in, Enhance parameters for features The The parameters correspond to the selected and corrected information The weighted value of the feature, and The selected and corrected information The weight coefficient associated with each feature.
[0015] In certain preferred embodiments, The formula is: ; in, and The selected and corrected information The weight coefficient associated with each feature.
[0016] It can be seen from the above that the distributed material compilation service system provided by the present invention has the following beneficial technical effects: The distributed material compilation service system provided by the present invention adopts a distributed task scheduling framework, which makes it possible to efficiently split the material compilation task into multiple subtasks and flexibly allocate them to different work nodes for parallel processing, thereby greatly improving the compilation efficiency and the scalability of the system, and easily coping with large-scale data challenges; the design of the distributed architecture enhances the fault tolerance of the system, and even in the face of the failure of a single work node, it can quickly realize task transfer, ensure the continuity and efficiency of the compilation work, and ensure the stability and reliability of the system; in the process of material compilation, the system introduces a preset classification model to carry out refined modular classification of the compilation content, which not only improves the structural clarity and logic of the compilation results, but also greatly facilitates subsequent search, modification and reuse, and improves the convenience and efficiency of the compilation work; combined with the user's historical preference information, the system can generate compilation content that is more in line with the user's expectations, meet the user's personalized needs, and improve the user experience and satisfaction. In addition, due to the use of distributed processing, modular classification, and user preference analysis, the system of the present invention significantly reduces problems such as omissions, errors, and duplications in the compilation process, effectively improves the quality and credibility of the compilation results, and provides strong support for fields that require highly accurate and reliable compilation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings: Figure 1 It is a structural schematic diagram of the distributed material compilation service system of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0019] Based on the problem pointed out in the background technology that the existing computer-aided compilation system adopts a centralized architecture and is difficult to cope with the situation of large data volume, resulting in the problem that the system is prone to failure. The present invention provides a distributed material compilation service system, which aims to efficiently split the material compilation task into multiple subtasks through a system adopting a distributed task scheduling framework, and flexibly allocate them to different working nodes for parallel processing, thereby greatly improving the compilation efficiency and the scalability of the system, easily coping with large-scale data challenges, and improving the system fault tolerance, modular classification, user personalized demand satisfaction, and compilation quality and credibility.
[0020] like Figure 1 As shown, the distributed material compilation service system of the present invention includes: a demand acquisition module 100, a primary information generation module 200, a selection and correction module 300 and a material compilation module 400. The functions of each module are described below: A requirement acquisition module 100 is used to acquire input material compilation requirement information; A primary information generation module 200, which is used to generate primary information of material compilation according to material compilation requirement information; A selection and correction module 300 is used to select and correct the primary information of material compilation, wherein the selection and correction module adopts a distributed task scheduling framework, and the primary information of material compilation is assigned to different working nodes for processing according to categories; A material compilation module 400, which is used to compile materials according to the selected and revised information; In the process of material compilation, the compiled content is modularly classified according to the preset classification model and user historical preference information.
[0021] In some preferred embodiments, the selection and correction module 300 includes a summary node, which is used to receive and summarize the information processed by all working nodes, and send the summarized information to the material compilation module 400. Alternatively, the selection and correction module 300 can also send the information processed by all working nodes directly to the material compilation module 400, which will summarize and process it. Further preferably, a message queue is used to realize asynchronous communication and task decoupling between various modules. For example, the demand acquisition module 100 sends the acquired material compilation demand information to the message queue, and the primary information generation module 200 obtains the material compilation demand information from the message queue and processes it. After the processing is completed, the primary information of material compilation is sent back to the message queue for processing by the selection and correction module 300. In this way, direct coupling between modules can be avoided, and the scalability and stability of the system can be improved. At the same time, the message queue can also play a buffering role to cope with sudden high-concurrency requests and prevent the system from collapsing due to excessive instantaneous pressure.
[0022] In some preferred embodiments, the distributed material compilation service system also includes a distributed database, which is used to store classification models, user historical preference information, material compilation requirement information, material compilation primary information and material compilation results. The distributed database provides high availability, scalability and fault tolerance to ensure safe and reliable storage and fast access to data. For example, a relational distributed database can be used to store structured data, such as user historical preference information, material compilation requirement information, etc.; while a non-relational distributed database (such as a NoSQL database) is used to store some large-scale, unstructured data, such as parameters of classification models, etc. Further preferably, in a distributed environment, it is necessary to ensure the consistency of data on each node. Specifically, a combination of periodic synchronization and real-time synchronization is used. For data that changes frequently and has high real-time requirements (such as preference data generated by real-time user operations, etc.), real-time synchronization is used; for relatively stable data (such as parameters of classification models, etc.), periodic synchronization is used.
[0023] Preferably, the aforementioned step of generating primary information of material compilation according to material compilation requirement information comprises: Inputting material compilation demand information into the material compilation primary information generation model; The primary information of material compilation is generated based on the primary information generation model of material compilation, wherein the formula of the primary information generation model of material compilation is: ; in, , First step in compiling primary information for the material Features, The number of features for which primary information is compiled for the material, , Compile requirements for materials Features, The number of characteristics for which the required information is compiled for the material, and are weight matrices, and are bias vectors, is the attention mechanism parameter, is the context information vector, To dynamically adjust parameters, is the regularization parameter, is the feature extraction function parameter, is the generator parameter, is the historical information fusion parameter.
[0024] In the above, is the attention mechanism parameter, ,in The attention weight of each feature in the corresponding material compilation requirement information, is the number of attention features, calculated by ,in It is calculated by a small neural network. , here represents a simple feed-forward neural network for determining the importance of different parts of the information required for material compilation so as to give different attention when generating primary information for material compilation; is the context information vector, , is the number of context information features. Context information can be obtained from external databases, knowledge graphs, etc. to enrich the content of primary information; To dynamically adjust parameters, , Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, To dynamically adjust the number of factor features, ,in, is a function of time, is the difference calculation function, and are the adjustment coefficients respectively; is the regularization parameter, , and is the regularization coefficient, and are the weight matrices respectively Norm and norm; is the feature extraction function parameter, let , It is a function for extracting specific features of material compilation requirement information. is the number of functions for specific feature extraction, such as feature extraction based on word frequency statistics, feature extraction based on topic models, etc.; is the generator parameter, , The number of generation rule features in the primary information for the corresponding materials compilation, the generation rules can be template-based generation, language model-based generation, etc.; is the historical information fusion parameter, , The number of primary information features generated by history, historical information fusion can be calculated by ,in is the fusion coefficient, which is used to combine the primary information of historical generation to improve the accuracy and consistency of the current generation.
[0025] Further preferably, in the foregoing, The formula is: ; in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of each feature; The formula is: ; in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of a feature is The relevant weight coefficient.
[0026] Preferably, the formula of the classification model in the above is: ; in, , For the classification categories, is the number of classification categories in the classification results, , The information after selection and correction Features, is the number of features of the selected and modified information, is the classification weight matrix, is the classification bias vector, is the tree structure parameter, are the support vector machine parameters, are neural network parameters, is the fusion coefficient, Enhance the parameters for the feature.
[0027] In the above, it is further preferred that The formula is: ; in is the sigmoid function, , and is the adjustment factor.
[0028] More preferably, The formula is: ; in, is the tree structure parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the support vector machine parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the neural network parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, , and The selected and corrected information The weight coefficient associated with each feature; The formula is: ; in, Enhance parameters for features The The parameters correspond to the selected and corrected information The weighted value of the feature, and The selected and corrected information The weight coefficient associated with each feature; The formula is: ; in, and The selected and corrected information The weight coefficient associated with each feature.
[0029] The present invention provides a distributed material compilation service system, which has the following beneficial effects: The present invention adopts a distributed task scheduling framework, so that the material compilation task can be efficiently split into multiple subtasks, and flexibly assigned to different working nodes for parallel processing, thereby greatly improving the compilation efficiency and the scalability of the system, and easily coping with large-scale data challenges; the design of the distributed architecture enhances the fault tolerance of the system, even in the face of a single working node failure, it can quickly realize task transfer, ensure the continuity and efficiency of the compilation work, and ensure the stability and reliability of the system; in the material compilation process, the system introduces a preset classification model to carry out refined modular classification of the compilation content, which not only improves the structural clarity and logic of the compilation result, but also greatly facilitates subsequent search, modification and reuse, and improves the convenience and efficiency of the compilation work; combined with user historical preference information, the system can generate compilation content that is more in line with user expectations, meet the user's personalized needs, and improve user experience and satisfaction. In addition, due to the use of distributed processing, modular classification and user preference analysis, the distributed material compilation service system of the present invention significantly reduces the problems of omissions, errors and duplications in the compilation process, effectively improves the quality and credibility of the compilation results, and provides strong support for the fields that require highly accurate and reliable compilation results.
[0030] It should be noted that the above describes a specific embodiment of the present invention. Other embodiments are within the scope of the present invention. In some cases, the actions or steps described above can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0031] For the convenience of description, the above device is described as being divided into various modules according to their functions. Of course, when implementing the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0032] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0033] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0034] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.
[0035] While the disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art in light of the foregoing description.
[0036] One or more embodiments of the present invention are intended to cover all such replacements, modifications and variations that fall within the broad scope of the present invention. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present invention should be included in the protection scope of this disclosure.
Claims
1. A distributed material compilation service system, characterized in that: The distributed material compilation service system includes: A requirement acquisition module, which is used to obtain input material compilation requirement information; A primary information generation module, which is used to generate primary information of material compilation according to the material compilation requirement information; A selection and correction module, which is used to select and correct the primary information of material compilation, wherein the selection and correction module adopts a distributed task scheduling framework, and the primary information of material compilation is assigned to different working nodes for processing according to categories; A material compilation module, which is used to compile materials based on selected and revised information; In the process of material compilation, the compiled content is modularly classified according to the preset classification model and user historical preference information.
2. The distributed material compilation service system according to claim 1, characterized in that: The selection and correction module includes a summary node, which is used to receive and summarize the information processed by all the working nodes, and send the summarized information to the material compilation module.
3. The distributed material compilation service system according to claim 1, characterized in that: The distributed material compilation service system also includes a distributed database, which is used to store the classification model, the user's historical preference information, the material compilation demand information, the material compilation primary information and the material compilation results.
4. The distributed material compilation service system according to claim 1, characterized in that: The step of generating primary information of material compilation according to the material compilation requirement information comprises: Inputting the material compilation requirement information into a material compilation primary information generation model; The material compilation primary information is generated based on the material compilation primary information generation model, wherein the formula of the material compilation primary information generation model is: ； in, , First step in compiling primary information for the material Features, The number of features for which primary information is compiled for the material, , Compile requirements for materials Features, The number of characteristics for which the required information is compiled for the material, and are weight matrices, and are bias vectors, is the attention mechanism parameter, is the context information vector, To dynamically adjust parameters, is the regularization parameter, is the feature extraction function parameter, is the generator parameter, is the historical information fusion parameter.
5. The distributed material compilation service system according to claim 4, characterized in that: The formula is: ； in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of a feature.
6. The distributed material compilation service system according to claim 4, characterized in that: The formula is: ； in, , , and Respectively The relevant mapping function, Compile the required information for the corresponding materials The attention weight of each feature, is the context information vector Through the mapping function Determined corresponding material compilation requirements information The context information value of a feature, Compile the required information for the corresponding materials The dynamic adjustment factor of each feature, is the regularization parameter Through the mapping function Determined corresponding material compilation requirements information The regularized correlation value of the features, Extract function parameters for features Through the mapping function Determined corresponding material compilation requirements information The feature extraction function value of the feature, For the generator parameters Through the mapping function Determined corresponding material compilation requirements information The generation rule value of features, is the historical information fusion parameter Corresponding material compilation requirements information The historical information value of a feature is The relevant weight coefficient.
7. The distributed material compilation service system according to claim 1, characterized in that: The formula of the classification model is: ； in, , For the classification categories, is the number of classification categories in the classification results, , The information after selection and correction Features, is the number of features of the selected and modified information, is the classification weight matrix, is the classification bias vector, is the tree structure parameter, are the support vector machine parameters, are neural network parameters, is the fusion coefficient, Enhance the parameters for the feature.
8. The distributed material compilation service system according to claim 7, characterized in that: The formula is: ； in, is the tree structure parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the support vector machine parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, is the neural network parameter The The parameters correspond to the selected and corrected information The weighted value of the feature, , and The selected and corrected information The weight coefficient associated with each feature.
9. The distributed material compilation service system according to claim 7, characterized in that: The formula is: ； in, Enhance parameters for features The The parameters correspond to the selected and corrected information The weighted value of the feature, and The selected and corrected information The weight coefficient associated with each feature.
10. The distributed material compilation service system according to claim 7, characterized in that: The formula is: ； in, and The selected and corrected information The weight coefficient associated with each feature.