Application scene-oriented big data acquisition and analysis method

Abstract

The invention relates to an application scene-oriented big data acquisition and analysis method, and relates to the technical field of computer processing, and the method comprises the following steps: S01, capturing dominant and implicit demands of a user through multi-modal input, and generating a dynamic intention tree; s02, automatically disassembling the task flow based on the intention tree, and calling a cross-domain data model to generate an end-to-end decision scheme; s03, realizing cross-domain scene migration through a pluggable adapter module, and dynamically optimizing a decision path; wherein the decision-making scheme is automatically executed through a decision-making confidence coefficient threshold value or returned to manual confirmation. According to the method, complex intentions are captured through nested semantic analysis, unattended decision making is achieved through a confidence coefficient threshold value, a pluggable adapter supports zero-cost cross-domain migration, a two-dimensional index tree guarantees billion-level data real-time response, the three pain points that traditional recommendation systems are one-sided in intention recognition, decision-making depends on manpower and cross-scene failure are comprehensively solved, and the recommendation efficiency is improved. And an end-to-end automatic decision closed loop is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a big data collection and analysis method oriented to application scenarios. Background Art

[0002] The core process of current intelligent recommendation systems can be summarized as follows: user inputs text - the system recognizes semantics - retrieves related data - filters redundant information - generates a recommendation list - and the user manually screens and makes decisions. Although the system achieves automation in the data processing phase, significant bottlenecks still exist in the final decision-making stage:

[0003] 1. Incomplete intent recognition: Traditional text parsing technologies (such as NLP models based on keyword matching) struggle to capture the embedded semantics and context within user input. For example, if a user enters "earphones suitable for outdoor sports in the rain," the system will only recognize "earphones" and ignore implicit requirements such as "sweat-proof in the rain" and "anti-loosening during exercise," leading to biased recommendations.

[0004] 2. Lack of automated decision-making: Mainstream systems, such as the Chinese patent publication number CN120104881A, which discloses a smart community life service recommendation system and method based on big data analysis, rely on "demand-resource matching calculation" but ultimately require users to manually confirm the service provider;

[0005] 3. Weak scenario transfer capabilities: When performing cross-domain recommendations (e.g., from e-commerce to healthcare), the model fails due to its lack of dynamic task decomposition. Although negative transfer can be mitigated through layered adapters (MoA), an end-to-end decision flow is not generated.

[0006] In summary, the deficiency of existing technologies is that the system only completes "information supply" but fails to achieve "decision-making". Therefore, how to solve this problem is expected to be well solved. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention adopts a method for collecting and analyzing big data for application scenarios, which includes the following steps:

[0008] S01. Capture users’ explicit and implicit needs through multimodal input and generate a dynamic intent tree;

[0009] S02: Automatically disassemble the task flow based on the intent tree and call the cross-domain data model to generate an end-to-end decision solution;

[0010] S03, realize cross-domain scenario migration through pluggable adapter modules and dynamically optimize decision paths;

[0011] The decision-making scheme is automatically executed or returned for manual confirmation based on the decision confidence threshold, and a scenario-based indexing engine is used to perform real-time retrieval and association of multi-source heterogeneous data.

[0012] Preferably, the step S01 of capturing the user's explicit and implicit needs through multimodal input includes:

[0013] S11, collecting user input text, voice, behavior sequence and environmental sensor data;

[0014] S12. Using a nested semantic parsing model to identify the main requirements and sub-requirements in the input, the nested semantic parsing model integrates keyword matching, context dependency graph and hidden Markov chain;

[0015] S13. When conflicting demands are detected, the demand game algorithm is started to calculate the optimal compromise weight;

[0016] S14. Output a dynamic intent tree with priority labels, where the tree nodes include requirement types, weight coefficients, and associated scenario labels.

[0017] Preferably, the dynamic intent tree in step S01 includes: if the user input contains scene keywords, then the associated constraints are extracted from the scene knowledge graph, and then the constraints are injected into the intent tree child nodes to generate an executable requirement framework with boundary conditions, wherein the constraints include physical rules, social rules and user historical preferences.

[0018] Preferably, the step of generating an end-to-end decision solution in step S02 includes:

[0019] S21. Decompose the dynamic intent tree into atomic task chains, and associate each atomic task with a domain execution engine.

[0020] S22. Select the optimal engine through the cross-domain routing module, where:

[0021] General tasks call public cloud APIs;

[0022] Specialized tasks call private domain models;

[0023] For uncovered areas, a zero-shot strategy generator is launched to reason about the decision flow based on the knowledge graph;

[0024] The final output is a complete decision plan with execution time window, resource list and risk assessment.

[0025] Preferably, the dynamic optimization decision path in step S03 includes:

[0026] S31. When the comprehensive confidence of the decision plan is greater than the threshold α, the execution is directly triggered;

[0027] S32. When the confidence level ∈ [β, α], return a simplified version of the solution for the user to confirm with one click, where β is one-tenth of the threshold value α;

[0028] S33, when the confidence level is less than β, starting the counterfactual enhancement module to generate a comparison view of alternative options;

[0029] Among them, confidence = data completeness * model accuracy * scenario matching.

[0030] Preferably, the pluggable adapter module in step S03 includes:

[0031] S34, converting general task parameters into entity relationships within the domain;

[0032] S35, suppressing cross-domain conflicting features through gradient masking;

[0033] S36. Update local decision rules based on user feedback;

[0034] When the application scenario switches, only the pluggable adapter module of the target domain is activated and the intent tree trunk is retained to enable seamless migration of the end-to-end decision flow.

[0035] Preferably, in step S03, cross-domain scenario migration is achieved through the pluggable adapter module, including:

[0036] S37. Label scene events from multi-source data and extract event types, time and space stamps, and impact factors;

[0037] S38. Using scene events as leaf nodes, construct a spatiotemporal-semantic dual-dimensional index tree, where the parent node is the scene category and the ancestor node is the domain category;

[0038] S39. When the decision layer initiates a request, the index tree is searched according to the priority of scene similarity > timeliness > data popularity, and the compressed data block is returned to the execution engine.

[0039] Preferably, the multi-source data includes user logs, IoT flows and knowledge bases.

[0040] The present invention has at least the following beneficial effects:

[0041] 1. By nesting semantic parsing models and scenario knowledge graphs, it breaks through the limitations of traditional text parsing and can simultaneously identify explicit requirements and implicit constraints;

[0042] 2. Based on the confidence threshold mechanism, high-credibility solutions are automatically executed, and counterfactual comparison views are generated for low-credibility solutions, achieving a leap from "information supply" to "decision-making on behalf of others."

[0043] 3. Pluggable adapter plug-ins use domain ontology mapping and negative transfer blocking to retain the intent tree trunk while switching domain engines, shortening the time required to build cross-scenario decision flows.

[0044] 4. The spatiotemporal-semantic dual-dimensional index tree uses a cascade retrieval strategy based on scenario similarity > timeliness > data popularity, reducing the latency of retrieval of 10TB multi-source data to milliseconds, supporting end-to-end decision making. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a big data collection and analysis method for application scenarios provided in the first embodiment of the present invention;

[0047] Figure 2 This is a flowchart of S01 provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] Example 1

[0051] This embodiment provides a big data collection and analysis method for application scenarios, which includes the following steps: Figure 1 As shown:

[0052] S01. Capture users’ explicit and implicit needs through multimodal input and generate a dynamic intent tree;

[0053] Specifically, the above-mentioned step of capturing the user's explicit and implicit needs through multimodal input includes:

[0054] S11, collecting user input text, voice, behavior sequence and environmental sensor data;

[0055] S12. Use a nested semantic parsing model to identify the main requirements and sub-requirements in the input. The nested semantic parsing model integrates keyword matching, context dependency graph and hidden Markov chain;

[0056] S13. When conflicting demands are detected, the demand game algorithm is started to calculate the optimal compromise weight;

[0057] S14. Output a dynamic intent tree with priority labels, where the tree nodes include requirement types, weight coefficients, and associated scenario labels.

[0058] Furthermore, the dynamic intent tree includes: if the user input contains scene keywords, the associated constraints are extracted from the scene knowledge graph, and then the constraints are injected into the intent tree child nodes to generate an executable requirement framework with boundary conditions. The constraints include physical rules, social rules and user historical preferences.

[0059] S02: Automatically disassemble the task flow based on the intent tree and call the cross-domain data model to generate an end-to-end decision solution;

[0060] Specifically, the steps of generating an end-to-end decision solution include:

[0061] S21. Decompose the dynamic intent tree into atomic task chains, and associate each atomic task with a domain execution engine.

[0062] S22. Select the optimal engine through the cross-domain routing module, where:

[0063] General tasks call public cloud APIs;

[0064] Specialized tasks call private domain models;

[0065] For uncovered areas, a zero-shot strategy generator is launched to reason about the decision flow based on the knowledge graph;

[0066] The final output is a complete decision plan with execution time window, resource list and risk assessment.

[0067] S03, realize cross-domain scenario migration through pluggable adapter modules and dynamically optimize decision paths;

[0068] Among them, the decision-making plan is automatically executed or returned for manual confirmation through the decision confidence threshold, and a scenario-based indexing engine is used to perform real-time retrieval and association of multi-source heterogeneous data.

[0069] Furthermore, the above dynamic optimization decision path includes:

[0070] S31. When the comprehensive confidence of the decision plan is greater than the threshold α, the execution is directly triggered;

[0071] S32. When the confidence level ∈ [β, α], return a simplified version of the solution for the user to confirm with one click, where β is one-tenth of the threshold value α;

[0072] S33, when the confidence level is less than β, starting the counterfactual enhancement module to generate a comparison view of alternative options;

[0073] Among them, confidence = data completeness * model accuracy * scenario matching.

[0074] Secondly, the aforementioned pluggable adapter module includes:

[0075] S34, converting general task parameters into entity relationships within the domain;

[0076] S35, suppressing cross-domain conflicting features through gradient masking;

[0077] S36. Update local decision rules based on user feedback;

[0078] When the application scenario switches, only the pluggable adapter module of the target domain is activated and the intent tree trunk is retained to enable seamless migration of the end-to-end decision flow.

[0079] Furthermore, in the above step S03, the cross-domain scenario migration is achieved through the pluggable adapter module, including

[0080] S37. Label scene events from multi-source data and extract event types, time and space stamps, and impact factors;

[0081] S38. Using scene events as leaf nodes, construct a spatiotemporal-semantic dual-dimensional index tree, where the parent node is the scene category and the ancestor node is the domain category;

[0082] S39. When the decision layer initiates a request, the index tree is searched according to the priority of scene similarity > timeliness > data popularity, and the compressed data block is returned to the execution engine.

[0083] It should be noted that multi-source data includes user logs, IoT streams, and knowledge bases.

[0084] This first embodiment breaks through the limitations of traditional text parsing by nesting semantic parsing models and scenario knowledge graphs, enabling simultaneous identification of explicit requirements and implicit constraints. Secondly, based on a confidence threshold mechanism, it automatically executes high-confidence solutions and generates counterfactual comparison views for low-confidence solutions, achieving a leap from "information supply" to "decision-making on behalf of others." Furthermore, the pluggable adapter plugin uses domain ontology mapping and negative transfer blocking to retain the intent tree backbone while switching domain engines, shortening the time required to build cross-scenario decision flows. Furthermore, the spatiotemporal-semantic dual-dimensional index tree uses a cascaded search strategy based on scenario similarity > timeliness > data popularity, reducing the latency of retrieval of 10TB-level multi-source data to milliseconds, supporting end-to-end decision generation.

[0085] Example 2

[0086] An embodiment of the present invention provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0087] Capture users' explicit and implicit needs through multimodal input and generate a dynamic intent tree;

[0088] Automatically disassemble task flows based on the intent tree and call cross-domain data models to generate end-to-end decision solutions;

[0089] Achieve cross-domain scenario migration through pluggable adapter modules and dynamically optimize decision paths;

[0090] Among them, the decision-making plan is automatically executed or returned for manual confirmation through the decision confidence threshold, and a scenario-based indexing engine is used to perform real-time retrieval and association of multi-source heterogeneous data.

[0091] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0092] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0093] Example 3

[0094] An embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the following steps:

[0095] Capture users' explicit and implicit needs through multimodal input and generate a dynamic intent tree;

[0096] Automatically disassemble task flows based on the intent tree and call cross-domain data models to generate end-to-end decision solutions;

[0097] Achieve cross-domain scenario migration through pluggable adapter modules and dynamically optimize decision paths;

[0098] Among them, the decision-making plan is automatically executed or returned for manual confirmation through the decision confidence threshold, and a scenario-based indexing engine is used to perform real-time retrieval and association of multi-source heterogeneous data.

[0099] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A big data collection and analysis method for application scenarios, characterized in that: The method comprises the following steps: S01. Capture users’ explicit and implicit needs through multimodal input and generate a dynamic intent tree; S02: Automatically disassemble the task flow based on the intent tree and call the cross-domain data model to generate an end-to-end decision solution; S03, realize cross-domain scenario migration through pluggable adapter modules and dynamically optimize decision paths; The decision-making scheme is automatically executed or returned for manual confirmation based on the decision confidence threshold, and a scenario-based indexing engine is used to perform real-time retrieval and association of multi-source heterogeneous data.

2. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The step S01 of capturing the user's explicit and implicit needs through multimodal input includes: S11, collecting user input text, voice, behavior sequence and environmental sensor data; S12. Using a nested semantic parsing model to identify the main requirements and sub-requirements in the input, the nested semantic parsing model integrates keyword matching, context dependency graph and hidden Markov chain; S13. When conflicting demands are detected, the demand game algorithm is started to calculate the optimal compromise weight; S14. Output a dynamic intent tree with priority labels, where the tree nodes include requirement types, weight coefficients, and associated scenario labels.

3. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The dynamic intent tree in step S01 includes: if the user input contains scene keywords, then the associated constraints are extracted from the scene knowledge graph, and then the constraints are injected into the intent tree child nodes to generate an executable requirement framework with boundary conditions, and the constraints include physical rules, social rules and user historical preferences.

4. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The step of generating an end-to-end decision solution in step S02 includes: S21. Decompose the dynamic intent tree into atomic task chains, and associate each atomic task with a domain execution engine. S22. Select the optimal engine through the cross-domain routing module, where: General tasks call public cloud APIs; Specialized tasks call private domain models; For uncovered areas, a zero-shot strategy generator is launched to reason about the decision flow based on the knowledge graph; The final output is a complete decision plan with execution time window, resource list and risk assessment.

5. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The dynamic optimization decision path in step S03 includes: S31. When the comprehensive confidence of the decision plan is greater than the threshold α, the execution is directly triggered; S32. When the confidence level ∈ [β, α], return a simplified version of the solution for the user to confirm with one click, where β is one-tenth of the threshold value α; S33, when the confidence level is less than β, starting the counterfactual enhancement module to generate a comparison view of alternative options; Among them, confidence = data completeness × model accuracy × scene matching.

6. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The pluggable adapter module in step S03 includes: S34, converting general task parameters into entity relationships within the domain; S35, suppressing cross-domain conflicting features through gradient masking; S36. Update local decision rules based on user feedback; When the application scenario switches, only the pluggable adapter module of the target domain is activated and the intent tree trunk is retained to enable seamless migration of the end-to-end decision flow.

7. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: In step S03, the pluggable adapter module is used to implement cross-domain scenario migration, including: S37. Label scene events from multi-source data and extract event types, time and space stamps, and impact factors; S38. Using scene events as leaf nodes, construct a spatiotemporal-semantic dual-dimensional index tree, where the parent node is the scene category and the ancestor node is the domain category; S39. When the decision layer initiates a request, the index tree is searched according to the priority of scene similarity > timeliness > data popularity, and the compressed data block is returned to the execution engine.

8. The method for collecting and analyzing big data for application scenarios according to claim 1, characterized in that: The multi-source data includes user logs, IoT streams and knowledge bases.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the steps of the application scenario-oriented big data collection and analysis method as described in any one of claims 1-8.

10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the steps of the application scenario-oriented big data collection and analysis method as described in any one of claims 1 to 8.

Images