Content generation system, method, and program

Abstract

In light of the above-mentioned problems with the conventional technology, the objective is to simultaneously achieve content generation that is of high quality on average and has little variation, and furthermore, to have the system itself autonomously and continuously optimize itself during operation. [Solution] The present invention solves the above problems by the configurations described in claims 1 to 3. In particular, 1. Self-evaluation vector S for each stage, 2. Update W by online weight learning unit. 3. Convergence test of σ and ternary gate test, 4. Control logic for looping these It is characterized by the fact that the above is an essential component.

Description

[Technical Field]

[0001] This invention relates to a content generation technology that automatically generates text or multimodal content by cooperating with multiple heterogeneous LLMs, and dynamically learns and quantitatively evaluates its quality. In particular, it applies to systems and methods including AI workflow management, self-evaluation scoring, online weight learning, and ternary gate judgment. [Background technology]

[0002] Although generation using a single LLM is widely used, the quality is unstable due to fluctuations in model characteristics and prompts. Traditionally, (i) objective indicators were lacking, and (ii) repeated testing and manual tuning were required, resulting in high operational costs. Frameworks for linking multiple LLMs, such as AutoGen, and self-assessment technologies, such as LLM as a Judge, have been reported. Dynamic weight learning that integrates multi-stage self-evaluation and Loop control based on the standard deviation convergence condition that represents the quality variation, - Automatic operation mechanism combining three-value disclosure / regeneration / discard gates · A structure that directly incorporates external KPIs (time spent, exit rate, search ranking, etc.) into the loss function No system is disclosed that includes both of these. **External KPIs are non-stationary, noisy data acquired in real time, and require scaling and delay correction with the internal self-assessment score, making it technically difficult to simply incorporate them into a loss function.** [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 1,101,7780 [Patent Document 2] U.S. Patent No. 12,111,859 [Non-patent literature]

[0004] [Non-Patent Document 1] Contentstack website, 2024 [Non-patent document 2] Kontent.ai website, 2024 [Non-patent document 3] CrafterCMS official documentation, 2024 Summary of the Invention [Problem to be solved by the invention]

[0005] In light of the above-mentioned problems with the conventional technology, the objective is to simultaneously achieve content generation that is of high quality on average and has little variation, and furthermore, to have the system itself autonomously and continuously optimize itself during operation. [Means for solving the problem]

[0006] The present invention solves the above problems by providing the configurations described in claims 1 to 3. In particular, 1. Self-evaluation vector S for each stage, 2. Update W by online weight learning unit. 3. Convergence test of σ and ternary gate test, 4. Control logic for looping these It is characterized by the fact that the above is an essential component. [Effects of the Invention]

[0007] Improved the average discussion score by +17 points (e.g., from 75 to 92), reduced the standard deviation by -12 points (from 15 to 3), and stabilized the quality. Automatic regeneration reduces intermediate rejection rate from 35% to 8%, significantly reducing operational costs. Convergence judgment prevents overfitting, and performance does not deteriorate even over long-term operation. According to the present invention, autonomous weight updating is performed based on actual performance indicators, (i) Search rankings improved by an average of 1.4 positions, (ii) A 30% reduction in the number of LLM calls; The technical effect was confirmed. [Brief explanation of the drawings]

[0008] [Figure 1] The collaborative pipeline control unit (1) receives the input topic and sequentially executes the pipeline processing by the LLMs: draft generation -> structuring and grammar correction -> expression and tone adjustment -> final audit. The results of each stage are passed to the stage-specific scoring unit (3), which records the self-evaluation score S_i calculated by the LLMs themselves. In parallel, the operational indicator acquisition unit (8) acquires KPIs such as dwell time, exit rate, and search result ranking from an external access analysis service, normalizes them, and inputs them into the dynamic weight learning unit (7a). 7a online updates the weight vector W using a loss function that includes S_i and the KPIs, and the convergence judgment unit (6b) monitors the convergence of the standard deviation σ. After convergence, the weighted aggregation unit (4) calculates the consensus score S_final using the updated W, and the gate decision unit (6a) automatically decides whether to publish, regenerate, or discard the data by referring to the pre-set (or dynamically updated) threshold T_pub / T_rej. [Figure 2] Stage 1 -> ... -> Stage N -> Score acquisition -> Weight learning -> σ convergence? -> Gate judgment -> Branching for publication / regeneration / discard. When regeneration is performed, an arrow is shown looping back to Stage 1. [Figure 3] Scoring and weight update sequence diagram Messages for S_i calculation, W update processing, σ calculation, and gate judgment are shown on the timeline. [Figure 4] Gradient descent based on equation (1) -> Standard deviation calculation -> Convergence check ε comparison -> Branching to fix W / update. [Figure 5] Quality Convergence Graph: Horizontal axis = number of generations, vertical axis = average and standard deviation of consensus score. Convergence is visualized with two lines. DETAILED DESCRIPTION OF THE INVENTION

[0009] (1) Hardware configuration Each LLM container is deployed on a Kubernetes(R) cluster. The control unit (1) performs asynchronous orchestration using the NATS message queue, and metadata such as S, W, and σ is stored in PostgreSQL(R). Recommended resources: 4x A100 GPUs, 256 GB RAM, average latency 1.2 s / stage.

[0010] (2) Software flow As shown in Figure 2, Stage 1 LLM receives an input topic T and generates a draft C_1, followed by C_2...C_n. At the end of each stage, a self-evaluation S_i is given. At the end of Stage N, the weight learning unit (7a) updates W using equation (1) and checks the convergence of σ.

[0011] (3) Online learning parameters Learning rate eta=0.002 (bias correction coefficients beta_1=0.9, beta_2=0.999) Convergence threshold epsilon=0.5 Batch size M=50 generations [Example]

[0012] The actual measured values ​​of 7-1 to 7-4 are shown in Table 1 and Figure 5, and the improvement in the average value and convergence of the standard deviation were confirmed. In a preferred embodiment of the present invention, the initial value of the weight vector W was set to {0.35, 0.20, 0.20, 0.25}. The convergence threshold epsilon was set to 0.5, the publication threshold T_pub was set to 90, and the regeneration threshold T_rej was set to 60. Furthermore, the publication approval condition was set to a standard deviation σ of 3.0 or less for every 100 generations. These values ​​are merely examples, and other values ​​can also be set. The obtained search result ranking r and average visit time t (seconds) are normalized to the range of 0 to 1. The loss function is defined as L' = 1 / 2(S_target - Σw_i S_i)^2 + beta_1·(1-t) + beta_2·r. Beta_1 and beta_2 are set appropriately within the range of 0.0 to 1.0 as an example, and the present invention is not limited to these coefficient values.

[0013] (5) Variations Includes variable stage configurations for N>=2, dynamic changes to T_pub / T_rej, multilingual support, and multimodal LLM.

[0014] Table 1 below shows examples of self-assessment scores hypothetically obtained based on a simulation to demonstrate the validity of the present invention. | Table 1: Validity assessment of self-assessment scores (provisional data) | |---|---|---| |Evaluation items|Average AI self-assessment score|Average human expert evaluation score| |Content validity (accuracy of content)|87.8|88.5| |Grammar & Structure Consistency|90.0|89.5| |Natural Expression / Tone Adjustment|94.0|93.2| |SEO Optimization|82.0|83.1| |Overall rating (collective score)|88.5|88.6| (Note: This data is a hypothetical simulation result and may differ from actual values.) "The experimental results shown in Table 1 are hypothetical numerical examples based on simulations to demonstrate the validity of the present invention. Measurement results during actual system operation may differ slightly from these example values, but they are listed as representative values ​​to demonstrate the effects and validity of the present invention." [Industrial Applicability]

[0015] This system can be applied to a wide range of fields, including CMS, news distribution, and automatic generation of e-commerce product descriptions, and can be used in both SaaS and on-premise formats. [Explanation of symbols]

[0016] Code Name 1 Cooperative Pipeline Control Unit 2a~2d LLM group 3. Stage-specific scoring section 4 Weighted aggregation part 5 AI communication log database 6a Gate Judgment Section 6b Convergence judgment section 7a Dynamic weight learning unit 8. Means for obtaining actual operational indicators (KPI acquisition section)

Claims

1. The system consists of a collaborative pipeline control unit that executes different types of large-scale language models (hereinafter referred to as "LLM") sequentially in a predetermined order in multiple stages and passes the generation results of each stage to the next stage; a stage-specific scoring unit that acquires the evaluation score S_i (a real value between 0 and 100 points) calculated by at least one LLM at the end of each stage and records it in a database; a dynamic weight learning unit that calculates the standard deviation σ of the evaluation score vector S for each generation over the most recent M times (M>=10) and updates the weight vector W by online learning according to the formula w_j <- w_j - eta dL / dw_j (where L = 1 / 2(S_target - Σw_i S_i)^2, eta is a predetermined learning rate); a convergence judgment unit that fixes W and proceeds to the next batch processing if the difference between the previous σ_prev and the current σ_now is equal to or less than a predetermined threshold epsilon; and a consensus score S_final=Σw_i A content generation system characterized by comprising a weighted aggregation unit that calculates S_i, and a gate determination unit that determines whether to approve publication, re-generation, or discard by referring to a pre-set or dynamically updated publication threshold T_pub and a re-generation threshold T_rej.

2. A content generation method using the content generation system described in claim 1, wherein input topics are input to a collaborative pipeline control unit to perform sequential generation, a self-assessment score is obtained for each stage, a weight learning unit updates W using the formula while monitoring the convergence of σ, and a gate judgment unit automatically decides whether to publish, regenerate, or discard based on T_pub / T_rej, and each of the above processes is looped until the convergence judgment condition is met.

3. A computer-readable non-transitory recording medium having a program recorded thereon for causing a computer to execute the method of claim 2.

4. 2. The content creation system according to claim 1, wherein the stage structure comprises four stages: draft generation, structuring and grammar correction, expression and tone adjustment, and final audit.

5. 5. The content generation system according to claim 4, wherein the weight vector W has a predetermined initial value that is set in advance.

6. 2. The content generation system according to claim 1, wherein the convergence threshold epsilon is equal to or less than a predetermined threshold.

7. 2. The content generation system according to claim 1, wherein the publication threshold T_pub and the regeneration threshold T_rej are each a predetermined value that has been set in advance.

8. 8. The content creation system according to claim 7, wherein the disclosure approval condition is added such that the standard deviation σ is equal to or less than a predetermined condition for each predetermined number of cases.

9. 2. The content creation system according to claim 1, wherein the self-assessment score and the natural language comment with a time stamp are stored in the log database.

10. 2. The content generation system according to claim 1, wherein each component is implemented as a microservice on a Kubernetes(R) cluster and is called via an external API.

11. 2. The content generation system according to claim 1, wherein the weight learning unit updates the weight vector W by using reinforcement learning in addition to the gradient descent method.

12. 2. The content generation system of claim 1, wherein the LLM is a multimodal configuration including an image generation model.

13. 2. The content generation system according to claim 1, wherein when a regeneration signal is output, regeneration is performed up to three times, and if T_pub is still not reached, the content is discarded.

14. 2. The content creation system according to claim 1, further comprising a monitoring mechanism for transmitting the collegial scores, σ, and W to an external dashboard in real time.

15. 2. The content generation system according to claim 1, further comprising an actual operation index acquisition means, wherein the actual operation index acquisition means acquires at least one of page visit time, bounce rate, or search result ranking via an external access analysis service, and the dynamic weight learning unit updates the weight vector using a loss function including the actual operation index.

Images