Big data based integrated service information processing system

Abstract

The present application relates to big data processing and information service technical field, especially integrated service information processing system based on big data, including data acquisition module, service arrangement module, service output module and quality estimation module. The system realizes the collaborative optimization of service accuracy and calculation efficiency through the multidimensional dynamic coupling analysis of multi-source heterogeneous data, real-time traffic state data, data source schema snapshot data and real-time data quality residual data, and the quality bottom unit of the quality estimation module carries out quality bottom for the service output process of the service output module, solves the technical problems of quality estimation and output execution disconnection, poor data non-interception mechanism in the prior art, and realizes the closed-loop control from quality strategy to output blocking.

Description

Technical Field

[0001] This invention relates to the field of big data processing and information service technology, and in particular to an integrated service information processing system based on big data. Background Technology

[0002] Existing big data integration service technologies have the following main shortcomings: First, the data cleaning module relies on static schema rules to perform format verification on multi-source heterogeneous data, failing to consider the sudden changes in field structure caused by upstream system upgrades or business changes. In schema drift scenarios, the cleaning rules become ineffective, causing dirty data that does not conform to the new contract to penetrate into downstream systems, leading to distorted analysis results and decision-making biases. Second, the service orchestration module uses a fixed concurrency level to pre-allocate computing resources, failing to perceive the non-linear amplification effect of data quality degradation on processing time. This quality degradation includes a surge in field missing rate, a jump in format anomaly rate, and changes in encoding methods. First, the cleaning time doubles during schema mutations, but the concurrency is not reduced, leading to thread pool exhaustion and cascading timeouts. Second, data quality control and service resource scheduling are executed serially as independent stages, lacking dynamic feedback coupling from quality degradation to resource constraints. When semantic drift triggers cleaning rollback, the resource layer does not synchronously reduce input traffic, causing queue accumulation and memory overflow. Third, schema change detection and resource adjustment are configured offline manually, without establishing an automatic contract negotiation mechanism based on real-time semantic similarity. This results in a service window period between schema change and manual intervention, during which service quality is uncontrollable.

[0003] For example, Chinese patent CN121481439A discloses an information integration management system based on big data. However, it still has problems such as the security protection module only performing static threat feature matching, not establishing a dynamic security threat identification mechanism based on real-time semantic drift index to include data quality degradation caused by schema changes into the scope of security threats, not considering the security boundary failure problem when the data source structure changes suddenly, and not dynamically adjusting the security strategy according to the data quality residual. This leads to dirty data bypassing static security protection and being injected downstream in scenarios of field addition, deletion or type change. Furthermore, it lacks a mechanism to pre-circuit the resources of the information analysis module based on the SLA default risk, resulting in the simultaneous failure of the security boundary and performance boundary under high load conditions of schema changes.

[0004] Chinese patent CN119806724A discloses an enterprise information management integration platform based on big data. However, it still has some shortcomings. When mapping heterogeneous data using a hierarchical autoencoder network, it uses static weights, fails to establish a dynamic contract negotiation mechanism based on real-time schema similarity to adapt to changes in field structure, does not consider the mapping failure problem of the autoencoder network in schema drift scenarios, and does not dynamically compensate for the mapping intensity of heterogeneous data based on semantic drift index. This leads to a surge in autoencoder reconstruction error in scenarios of source system DDL changes or encoding method conversion. Furthermore, the biological cell division mechanism and entropy balance mechanism are coarse-grained biomimetic scheduling and do not establish fine-grained feedback coupling with data quality residuals. It lacks a mechanism for pre-emptive elastic contraction of resource scheduling based on SLA default risk, resulting in the problem of dual loss of control over resource allocation and quality management under schema mutation and sudden traffic concurrency conditions. Summary of the Invention

[0005] To address this, the present invention provides an integrated service information processing system based on big data, which overcomes the problems of service crashes caused by static schema verification failure, semantic drift and resource orchestration disconnect, as well as the lack of dynamic coupling between quality degradation and resource constraints in the prior art.

[0006] To achieve the above objectives, the present invention provides an integrated service information processing system based on big data, comprising: The data acquisition module is used to collect multi-source heterogeneous data, real-time traffic status data, and data source schema snapshot data of the target business. The service orchestration module is used to generate a basic service scheme based on the multi-source heterogeneous data, and to perform traffic adaptation on the basic service scheme based on the real-time traffic status data to obtain an adapted basic service scheme. It is also used to perform semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain a target service scheme. The service output module is used to output the target service according to the target service plan. The quality estimation module is used to provide a quality safety net for the service output process of the service output module.

[0007] Furthermore, the step of generating a basic service solution based on the multi-source heterogeneous data includes: The basic resource allocation vector is calculated based on the data source identifier, data volume, business priority, and initial service window included in the multi-source heterogeneous data; The basic service parameters are set according to the basic resource allocation vector, the preset concurrency level, and the preset response threshold, and the basic service parameters are output as the basic service scheme.

[0008] Further, the step of adapting the basic service scheme to the real-time traffic status data to obtain the adapted basic service scheme includes: The traffic fluctuation index is calculated based on the current request rate, historical peak rate, first rate weight, and second burst weight included in the real-time traffic status data. The traffic fluctuation index is compared with a preset fluctuation index. The traffic status is judged based on the comparison result. The traffic is adapted to the basic service scheme based on the judgment result to obtain the adapted concurrency. The adapted concurrency is then replaced in the basic service scheme to obtain the adapted basic service scheme.

[0009] Further, the step of performing semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme includes: The semantic drift index is calculated based on the field missing rate and type conflict rate included in the schema semantic index in the schema snapshot data of the data source; The semantic drift index is compared with a preset drift index. The semantic state is judged based on the comparison result. The semantic compensation is performed on the adapted basic service scheme based on the judgment result to obtain the compensated cleaning intensity. The compensated cleaning intensity is then replaced in the adapted basic service scheme to obtain the target service scheme.

[0010] Furthermore, the step of performing semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme also includes: setting the rollback duration Trq according to the schema difference type, setting Trq=300s when the difference is identified as a field addition or deletion, setting Trq=60s when the difference is identified as a type narrowing, and setting Trq=0s when the difference is identified as an encoding change, and supplementing the rollback duration Trq into the target service scheme.

[0011] Furthermore, the quality estimation module's quality assurance mechanism for the service output process of the service output module specifically includes: generating a basic quality estimation strategy based on the target service plan; The target quality estimation strategy is obtained by performing drift correction on the basic quality estimation strategy based on the real-time data quality residual data. The service output process of the service output module is given a quality safety net based on the target quality estimation strategy.

[0012] Further, the step of generating a basic quality estimation strategy based on the target service plan includes: The basic quality level is calculated based on the steady-state response error, backoff time, current throughput, and current default rate included in the target service plan; Based on the aforementioned basic quality level, a data confidence threshold is set to obtain a basic quality estimation strategy that includes the data confidence threshold.

[0013] Further, the step of performing drift correction on the basic quality estimation strategy based on real-time data quality residual data to obtain the target quality estimation strategy includes: The real-time data quality residual data is acquired, and the real-time data quality residual data includes semantic deviation and cleaning rollback rate; The quality residual index is calculated based on the semantic deviation, the cleaning rollback rate, the first deviation weight, and the second rollback weight. The quality residual index is compared with a preset residual index. The data quality status is judged based on the comparison result. The basic quality estimation strategy is then drift-corrected based on the judgment result to obtain the corrected data confidence threshold. The corrected data confidence threshold is then replaced in the basic quality estimation strategy to obtain the target quality estimation strategy.

[0014] Furthermore, the step of providing a quality safety net for the service output process of the service output module based on the target quality estimation strategy includes: The corrected data confidence threshold included in the target quality estimation strategy is compared with the data quality score of the current output batch of the service output module, and the service output status is judged based on the comparison result, wherein: When Pq≥J1, the output quality is deemed to meet the standard, and the service output module is allowed to continue performing service output. When Pq < J1, the output quality is determined to be substandard, the current output batch of the service output module is blocked, and the current output batch is transferred to the isolation queue. At the same time, the service orchestration module is triggered to re-execute semantic compensation.

[0015] Furthermore, the execution cycle Te=1s for the quality fallback, the blocking delay Tdly≤50ms, and the maximum capacity of the isolation queue is 10,000 records.

[0016] This invention has the following beneficial effects: The system collects multi-source heterogeneous data, real-time traffic status data, and data source schema snapshot data of the target service through a data acquisition module, solving the technical problems of existing technologies that lack data source semantic benchmarks and real-time schema change awareness data for service strategies. This provides a complete data foundation for subsequent service orchestration and quality optimization, realizing the transformation from static configuration to dynamic awareness. The system generates service plans based on the multi-source heterogeneous data through a service orchestration module, solving the problem of static setting deviations caused by service parameters relying on fixed experience values ​​and not considering data volume characteristics in existing technologies, making the initial service orchestration business-adaptable. The system collects multi-source heterogeneous data through the data acquisition module to generate basic service plans. By adapting the basic service plans to traffic, it solves the problem of resource allocation imbalance caused by relying solely on static rule scheduling and ignoring real-time traffic fluctuation correction in existing technologies, realizing synchronous control of data aggregation and traffic awareness, and eliminating false service phenomena. By performing semantic compensation on the adapted basic service plans, it solves the problem of existing technologies not considering data source schema dynamics. To address service failures caused by state changes and fixed contract orchestration, the system dynamically decouples semantic contracts from resource scheduling, ensuring service continuity. The system outputs services to the target business through a service output module, resolving the issues of disconnect between service orchestration and output execution, and time synchronization lag in existing technologies. This achieves closed-loop execution from service plan to service output, ensuring stable business response within the target accuracy range. The system also generates a quality estimation strategy based on the target service plan through a quality estimation module, addressing the disconnect between quality estimation and data characteristics, and inaccurate preset verification thresholds in existing technologies. This achieves a reasonable mapping from service quality to verification parameter configuration. Furthermore, by generating a basic quality estimation strategy and performing drift correction on this strategy, the system addresses the issues of semantic drift fluctuations and cleaning estimation distortion caused by fixed verification rules in existing technologies, achieving adaptive error optimization for semantic quality. Finally, by providing quality fallback for the service output process of the service output module, the system addresses the technical problems of disconnect between quality estimation and output execution, and the lack of interception mechanisms for inferior data in existing technologies, achieving closed-loop control from quality strategy to output blocking. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the big data-based integrated service information processing system in this embodiment; Figure 2 A structural diagram of the service orchestration module; Figure 3 This is a schematic diagram of the quality estimation module. Figure 4 This is a logic diagram of the quality fallback unit. Detailed Implementation

[0018] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0019] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0020] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0021] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0022] Please see Figure 1 As shown, this is a schematic diagram of the structure of the big data-based integrated service information processing system of this embodiment. The system includes: The data acquisition module is used to collect multi-source heterogeneous data, real-time traffic status data, and data source schema snapshot data of the target business. The service orchestration module is used to generate a basic service scheme based on the multi-source heterogeneous data, and to perform traffic adaptation on the basic service scheme based on the real-time traffic status data to obtain an adapted basic service scheme. It is also used to perform semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain a target service scheme. The service orchestration module is connected to the data acquisition module. A service output module is used to output the target service according to the target service scheme, and the service output module is connected to the service orchestration module. The quality estimation module is used to generate a basic quality estimation strategy based on the target service plan, to perform drift correction on the basic quality estimation strategy based on real-time data quality residual data to obtain a target quality estimation strategy, and to provide quality fallback for the service output process of the service output module based on the target quality estimation strategy. The quality estimation module is connected to the service output module.

[0023] Specifically, the big data-based integrated service information processing system is applied to enterprise-level data platforms or government information sharing platforms equipped with distributed data acquisition nodes and microservice orchestration engines. The system achieves synergistic optimization of service accuracy and computational efficiency through multi-dimensional dynamic coupling analysis of multi-source heterogeneous data, real-time traffic status data, data source schema snapshot data, and real-time data quality residual data. The system collects multi-source heterogeneous data, real-time traffic status data, and data source schema snapshot data for the target business through a data acquisition module, addressing the lack of data source semantic benchmarks and real-time schema change awareness in existing technologies. The system addresses the technical challenges of data knowledge, providing a complete data foundation for subsequent service orchestration and quality optimization, and enabling a shift from static configuration to dynamic awareness. The system generates service plans based on the multi-source heterogeneous data through a service orchestration module, resolving the static setting deviations caused by service parameters relying on fixed empirical values ​​and failing to consider data volume characteristics in existing technologies. This ensures the initial service orchestration is business-adaptive. A data acquisition module collects multi-source heterogeneous data to generate basic service plans. By performing traffic adaptation on these basic service plans, the system solves the resource allocation imbalance problems caused by relying solely on static rule scheduling and neglecting real-time traffic fluctuation correction in existing technologies, thus achieving data-driven... Synchronous control of aggregation and traffic awareness eliminates false service phenomena. Semantic compensation of the adapted basic service scheme addresses service failures caused by unconsidered dynamic changes in the data source schema and fixed contract orchestration in existing technologies. This achieves dynamic decoupling of semantic contracts and resource scheduling, ensuring service continuity. The system outputs services to the target business through a service output module, resolving the disconnect between service orchestration and output execution, and time synchronization lag issues in existing technologies. This achieves closed-loop execution from service scheme to service output, ensuring stable business response within the target accuracy range. The system also uses a quality estimation module to assess quality based on the target service scheme. The estimation strategy is generated to address the problems of disconnect between quality estimation and data characteristics and inaccurate preset verification thresholds in existing technologies, achieving a reasonable mapping from service quality to verification parameter configuration. By generating a basic quality estimation strategy and performing drift correction on the basic quality estimation strategy, the problem of cleaning estimation distortion caused by the failure to consider semantic drift fluctuations and fixed verification rules in existing technologies is solved, achieving error optimization with semantic quality adaptation. By providing quality fallback for the service output process of the service output module, the technical problems of disconnect between quality estimation and output execution and lack of interception mechanism for inferior data in existing technologies are solved, achieving closed-loop control from quality strategy to output blocking.

[0024] Specifically, the target business refers to specific business applications deployed on enterprise-level data platforms or government information sharing platforms, including transaction settlement systems, real-time analysis platforms, and log auditing systems. The multi-source heterogeneous data includes data source identifiers, data volume, business priority, and initial service window. The data source identifier is a unique code distinguishing different upstream data systems, composed of letters and numbers, with a length of 8 to 32 characters, representing the traceability attribute of the data source. In this embodiment, the data source identifier is collected through a unified resource identifier allocator, which is a distributed registration component that generates globally unique identifiers based on a hash algorithm. The data volume refers to the amount of data flowing into the system from various data sources per unit time. The raw data volume, measured in megabytes per second, represents the physical scale of the data load. In this embodiment, the data volume is collected through a traffic probe or log collection agent. The traffic probe is a monitoring component that collects data on the number of bytes per packet based on network layer packet capture or application layer data entry. The sampling frequency of the traffic probe is 1 Hz to 10 Hz. The service priority refers to a service level identifier pre-set by the service provider, with a value ranging from 1 to 5. A higher value indicates a greater loss due to service interruption. In this embodiment, the service priority is configured through a service level protocol management console, which is a configuration center that maintains service metadata based on key-value storage. The initial service window refers to the window allocated to each data source by the system at startup. The default number of concurrent processing slots is set to 10 ≤ Wb ≤ 1000, representing the initial resource allocation state before the system detects traffic. The real-time traffic status data includes the current request rate and historical peak rate. The current request rate refers to the number of HTTP or RPC requests arriving at the system gateway per unit time, measured in requests per second, representing the instantaneous intensity of business access. The historical peak rate refers to the maximum request rate recorded in the past 24 hours or 7 days, measured in requests per second, representing the historical upper limit benchmark of business load. In this embodiment, the current request rate is collected through a gateway counter or a distributed tracing system. The gateway counter is an embedded gateway that counts request traffic based on a token bucket or sliding window algorithm. The module, wherein the gateway counter's sampling frequency is 1 Hz to 10 Hz, and the data source schema snapshot data includes field missing rate and type conflict rate. The field missing rate refers to the proportion of fields missing relative to the baseline schema in the current data batch, with a value range of 0 ≤ Fm ≤ 1, representing the degree of degradation of schema structural integrity. The type conflict rate refers to the proportion of records in the current data batch whose field values ​​are incompatible with the data type defined in the baseline schema, with a value range of 0 ≤ Ft ≤ 1, representing the degree of violation of data type contracts. In this embodiment, the data source schema snapshot data is collected through a schema comparison engine or an Avro / Protobuf contract checker.The schema comparison engine is a metadata governance component that compares differences in field tree structures based on depth-first traversal. The engine's detection period is 5 to 30 seconds.

[0025] Please see Figure 2 As shown, this is a structural diagram of the service orchestration module, which includes: A basic orchestration unit is used to generate a basic service scheme based on the multi-source heterogeneous data; Traffic adaptation unit is used to adapt the basic service scheme to traffic based on the real-time traffic status data to obtain the adapted basic service scheme. The traffic adaptation unit is connected to the basic orchestration unit. A semantic compensation unit is used to perform semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme. The semantic compensation unit is connected to the traffic adaptation unit.

[0026] Specifically, the method for generating basic service solutions based on the multi-source heterogeneous data includes: Based on the data source identifier Ds, data volume Vd, business priority Pb, and initial service window Wb included in the multi-source heterogeneous data, the basic resource allocation vector Rb is calculated, and Rb is set as (Vd1 / Vdt,Vd2 / Vdt,...,Vdn / Vdt)⊙Pb×Wb, where Vdt is the total data volume, ⊙ is the Hadamard product operator, and Pb is the business priority vector; The basic service parameter Sb is set according to the basic resource allocation vector Rb, the preset concurrency Cb and the preset response threshold Tb, and Sb=(Rb,Cb,Tb) is set. The basic service parameter Sb is then output as the basic service scheme.

[0027] Specifically, the total data volume refers to the arithmetic sum of the data flowing into the system from all data sources per unit time, measured in megabytes per second. In this embodiment, the total data volume is calculated by summing the traffic probe readings from each data source. The Hadamard product operator refers to the element-wise multiplication operation of two vectors of the same dimension. The business priority vector is an n-dimensional column vector composed of the business priorities corresponding to each data source, where n is the total number of data sources. Vd1 represents the original data volume flowing into the system from the first data source per unit time, Vd2 represents the original data volume flowing into the system from the second data source per unit time, Vd3 represents the original data volume flowing into the system from the third data source per unit time, and so on. n represents the amount of raw data flowing into the system from the nth data source per unit time. The preset concurrency level refers to the number of data processing tasks that the system is allowed to execute simultaneously at startup, with a value of Cb=100. The reason for this preset concurrency level is that this value corresponds to the default thread pool capacity of a medium-sized data platform, which can maintain a response time of less than 500 milliseconds under normal load, while reserving resource margin to cope with sudden traffic. The preset response threshold refers to the maximum service response time limit promised by the system at startup, with a value of Tb=1000ms. The preset response threshold corresponds to the service level agreement baseline requirements of enterprise-level applications and can be further tightened or relaxed as an initial performance constraint through subsequent dynamic optimization.

[0028] Specifically, the step of adapting the basic service plan to the real-time traffic status data to obtain the adapted basic service plan includes: The traffic fluctuation index Ef is calculated based on the current request rate Rc, historical peak rate Rp, first rate weight α1 and second burst weight α2 included in the real-time traffic status data. Ef is set as Ef = α1 × Rc / Rp + α2 × (1 - Na / Nm), where Na is the number of currently active sessions and Nm is the maximum session capacity. The traffic fluctuation index Ef is compared with the preset fluctuation index E0. The traffic status is judged based on the comparison result, and the basic service scheme is adapted to the traffic based on the judgment result, wherein: When Ef≤E0, the traffic status is determined to be stable, and no traffic adaptation is performed on the basic service scheme. When Ef > E0, the traffic state is determined to be bursty. Traffic adaptation is performed on the basic service scheme to obtain the adapted concurrency C1. The adapted concurrency C1 is then replaced in the basic service scheme to obtain the adapted basic service scheme.

[0029] Specifically, the first rate weight refers to the weighting coefficient assigned to the ratio of the current request rate to the historical peak rate when calculating the traffic fluctuation index, with a value of α1=0.6. The rationale for this first rate weight is that in an integrated service system, the request rate ratio Rc / Rp directly reflects the degree to which the traffic approaches the historical peak. When this ratio is close to 1, it indicates that the system is about to enter the overload edge. This rate risk accounts for 60% to 70% of the total congestion risk, so a higher weight is assigned to prioritize the identification of traffic surge scenarios. The second burst weight refers to the weighting coefficient assigned to the session saturation gap when calculating the traffic fluctuation index, with a value of α2=0.4. The rationale for this second burst weight is that the session saturation gap reflects the resource consumption status of the connection pool. In the scenario of connection pool exhaustion, although the request rate has not reached the peak, the inability of new requests to access the system still leads to service interruption. This capacity degradation accounts for 60% to 70% of the total congestion risk. The risk is 30% to 40%, so a weight of 0.4 is assigned to complement the first rate weight. The preset fluctuation index is the critical threshold for determining whether the traffic status has entered the burst area, with a value of E0=0.7. The preset fluctuation index is a dimensionless normalized threshold. When Ef=0.7, the corresponding request rate reaches 70% of the historical peak and the number of sessions reaches 70% of the maximum capacity. Exceeding this value indicates that the system resources are insufficient to support reliable service orchestration. At this time, elastic expansion must be introduced to provide resource backup. The adapted concurrency refers to the concurrency increment calculated by the linear growth algorithm when the traffic status is determined to be bursty, with a value of C1=Cb×(1+d×(Ef-E0) / Emax), where d is the expansion coefficient and Emax is the maximum fluctuation index. The linear growth algorithm is a conservative strategy that increases the concurrency by a fixed step size each period, which is suitable for predictable growth scenarios.

[0030] Specifically, the step of performing semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme includes: The semantic drift index Sd is calculated based on the field missing rate Fm and type conflict rate Ft included in the schema semantic index in the schema snapshot data of the data source. Sd is set as β1×Fm / F0+β2×Ft / T0, where F0 is the baseline missing rate, T0 is the baseline conflict rate, β1 is the first semantic weight, and β2 is the second semantic weight. The semantic drift index Sd is compared with the preset drift index S0. The semantic state is judged based on the comparison result, and semantic compensation is performed on the adapted basic service scheme based on the judgment result, wherein: When Sd≤S0, the semantic state is determined to be contract stable, and no semantic compensation is performed on the adapted basic service scheme. When Sd > S0, the semantic state is determined to be a contract break. Semantic compensation is performed on the adapted basic service scheme to obtain the compensated cleaning intensity D2. D2 is set as D1 × (1 + h × (Sd - S0) / Smax), where D1 is the cleaning intensity in the adapted basic service scheme, h is the semantic suppression coefficient, and Smax is the maximum semantic drift. The rollback duration Trq is set according to the schema difference type. When the identification is a field addition or deletion, Trq is set to 300s. When the identification is a type narrowing, Trq is set to 60s. When the identification is an encoding change, Trq is set to 0s. The compensated cleaning intensity D2 and the rollback duration Trq are replaced in the adapted basic service scheme to obtain the target service scheme.

[0031] Specifically, the schema difference type refers to the category of structural changes in the current schema relative to the baseline schema detected by the schema comparison engine, including field additions / deletions, type narrowing, and encoding changes. This embodiment identifies schema difference types using a schema difference classifier. Field additions / deletions refer to structural expansion or contraction caused by the upstream system adding business fields or obsolete historical fields. Type narrowing refers to the contraction of compatibility range caused by a change in field data type from a broad type to a strict type. Encoding changes refer to parsing failures caused by conversion of the character set or serialization method from UTF-8 to GBK or from JSON to Protobuf. The rollback duration Trq refers to the time taken to complete a specific task within a given timeframe. Under the schema difference type, the system maintains the degradation and cleaning mode until the contract is restored, for the predicted duration. When the semantic state is contract stable, the rollback duration Trq is set to the default rollback duration Tr0=0s in the adapted basic service solution, indicating no rollback is needed. When the semantic state is contract broken, the rollback duration Trq is set differently according to the schema difference type. The rollback duration for adding or deleting fields is set to Trq=300s. The reason for this value is that typical field additions and deletions involve downstream ETL link transformation, report field mapping adjustment, and front-end display adaptation, with a total time consumption of approximately 180s to 240s. Setting it to 300s provides a margin to ensure full-link contract restoration. The rollback duration for the narrower type is set to Trq=60s. The reasons are as follows: Type narrowing typically triggers data truncation or accuracy loss alarms, with an operation and maintenance intervention cycle of approximately 30 to 60 seconds. A value of 60 seconds covers the typical manual confirmation and rollback operation cycle. The rollback duration for the encoding change is set to Trq=0 seconds. The reason for this value is that encoding changes cause batch parsing failures, necessitating immediate blocking of data inflow and triggering emergency repairs, with no rollback buffer period. The first semantic weight refers to the weight coefficient assigned to the field missing rate when calculating the semantic drift index, with a value of β1=0.7. The reason for this first semantic weight is that the field missing rate Fm directly reflects upstream system DDL changes or field obsolescence. A higher missing rate indicates a more severe breach of the semantic contract, posing a threat to downstream data integrity in over 70% of cases. Therefore, it is assigned a weight of β1=0.7. The highest weight; the second semantic weight refers to the weight coefficient assigned to the type conflict rate when calculating the semantic drift index, with a value of β2=0.3. The reason for the value of the second semantic weight is that the type conflict rate Ft mainly corresponds to implicit type conversion failure or precision overflow. Although it causes a single record to fail, it can be partially recovered by filling in the default value. Therefore, it is given a lower weight to form a complementary diagnosis. The semantic state refers to the classification and judgment result of the degree of deviation of the data source schema from the benchmark contract based on the comparison result of the semantic drift index Sd and the preset drift index S0. The semantic state includes two types: contract stability and contract breakage. The preset drift index is the critical threshold for determining whether the semantic state has entered the contract breakage region, with a value of S0=1.The preset drift index is set to 0 because when Sd = 1.0, it corresponds to a situation where the missing field rate reaches 100% of the baseline missing rate and the type conflict rate reaches 100% of the baseline conflict rate. Exceeding this value indicates that the schema has undergone structural changes, and the original cleaning rules cannot be reused. The post-compensation cleaning intensity refers to the enhanced verification level obtained by quantifying the strictness of cleaning through the semantic drift index when the semantic state is determined to be a contract break. The value is D2 = D1 × (1 + h × (Sd - S0) / Smax), where D1 is the cleaning intensity in the adapted basic service scheme, h is the semantic suppression coefficient, and Smax is the maximum semantic drift. The semantic suppression coefficient is an adjustment parameter that controls the increase in cleaning intensity during semantic compensation. The value is h = 0.5. The reason for the semantic suppression coefficient is that, according to the conservative data governance strategy, it is necessary to ensure that under extreme semantic drift conditions, the post-compensation cleaning intensity D2 ≤ 1.5D1. When Sd = Smax, D2 = 1.5D1. This approach effectively blocks dirty data penetration caused by contract breakage while avoiding data gaps due to excessive cleaning. The maximum semantic drift refers to the upper limit of the semantic drift index Sd, which is set to Smax = 3.0. The rationale for this maximum semantic drift value is as follows: Under extreme conditions, such as a major version upgrade of the upstream system with Fm≈F0, and simultaneous full-scale encoding changes with Ft≈T0, Sd = 0.7×1 + 0.3×1 = 1.0. This takes into account simultaneous changes at multiple points and cascading schema failures. A semantic fluctuation range of 3.0 is taken to cover all normal operating conditions. The compensated cleaning intensity D2 is determined only by the semantic drift index Sd and has no direct coupling relationship with the corrected data confidence threshold J1. The cleaning intensity D2 characterizes the strictness of data cleaning by the service orchestration module, with a value of D2 ≥ D1. When Sd = S0, D2 = D1; when Sd = Smax, D2 = 1.5D1. The cleaning intensity is positively correlated with the degree of semantic drift, ensuring that the cleaning rules are synchronously enhanced when the schema changes abruptly.

[0032] Specifically, the service output module is used to output services to the target business according to the target service plan, and specifically includes: The service output mode of the target service is switched to the converged output mode. The service output frequency Fs is set according to the post-compensation cleaning intensity D2 in the target service scheme. Fs is set as min(Fdc, Fse), where Fdc is the sampling frequency of the data acquisition module and Fse is the scheduling frequency of the service orchestration engine. The service output frequency is then output to the downstream business system through the service mesh bus to realize service output. The service status is determined by comparing the rollback duration Trq in the target service plan with the current rollback duration Td. When Td < Trq, a high-precision fusion service is output. When Td ≥ Trq, the service is switched to the basic service fallback mode and the service uncertainty estimate σs is output.

[0033] Specifically, the service output mode refers to the classification identifier of the service computing working state determined by the system based on the data processing stage and resource fusion strategy, including pure acquisition mode, lightweight mode, full acquisition mode, and fusion output mode, used to characterize the orchestration architecture and data processing method adopted by the system at the current moment; the fusion output mode refers to the working state of the system outputting the final service after weighted fusion of data acquisition results and service orchestration results, setting the cleaning intensity to be dynamically adjusted according to the semantic drift index, realizing the complementary advantages and error suppression of multi-source heterogeneous data; the high-precision fusion service refers to the service obtained through tight coupling and fusion of data acquisition and service orchestration. The service response results, with a response time satisfying σs≤500ms and a throughput satisfying σt≥1000TPS, characterize the optimal service performance of the system under effective service conditions; the basic service fallback mode refers to the emergency working state in which the system switches to relying only on local caching or degradation strategies when the semantic contract breaks or the rollback duration Td≥Trq. At this time, the parsing of upstream schema change data is stopped, and service continuity is maintained by relying on the baseline contract, accompanied by a risk warning that data freshness degrades linearly over time; the service uncertainty estimation refers to the current service confidence metric calculated by the system according to the service level agreement, and sets... ,in To estimate the covariance matrix posteriorly, The matrix trace operation is used to characterize the statistical reliability and error boundary range of the service results. The sampling frequency of the data acquisition module refers to the number of data pulls or pushes performed per second by the data acquisition node, in Hertz, representing the update rate of the data source, with a value of 1Hz≤Fdc≤10Hz. The reason for this value is that this range corresponds to the typical acquisition frequency from timed batch processing to real-time stream processing. A higher value is used for high-time-sensitivity services to ensure data freshness, and a lower value is used for low-time-sensitivity services to reduce network bandwidth consumption. The service orchestration engine scheduling frequency refers to the number of service orchestration cycles completed per second by the service orchestration module, in Hertz, representing the refresh rate of the service policy, with a value of 1Hz≤Fse≤5Hz. The reason for this value is that this range corresponds to the standard heartbeat frequency of the microservice orchestration engine, taking into account both policy sensitivity and computational overhead limitations. The service output frequency Fs takes the smaller of the two values ​​to ensure time synchronization and data alignment.

[0034] Please see Figure 3 As shown, this is a schematic diagram of the quality estimation module, which includes: A basic estimation unit is used to generate a basic quality estimation strategy based on the target service scheme; A drift correction unit is used to perform drift correction on the basic quality estimation strategy based on real-time data quality residual data to obtain a target quality estimation strategy. The drift correction unit is connected to the basic estimation unit. A quality fallback unit is used to perform quality fallback on the service output process of the service output module according to the target quality estimation strategy. The quality fallback unit is connected to the drift correction unit.

[0035] Specifically, the step of generating a basic quality estimation strategy based on the target service plan includes: The baseline quality level Qm is calculated based on the steady-state response error Es, backoff time Trq, current throughput Pt, and current default rate Dv included in the target service scheme. Qm is set as Qm=(Es / E0)×(Pt0 / Pt)×(Dv / D0) / (1+Trq / T0), where E0 is the baseline response error, Pt0 is the baseline throughput, D0 is the baseline default rate, and T0 is the baseline backoff time. Based on the basic quality level Qm, the data confidence threshold Jc is set to obtain the basic quality estimation strategy that includes the data confidence threshold Jc.

[0036] Specifically, the steady-state response error refers to the average deviation between the actual response time and the target response time under stable operating conditions, measured in milliseconds. This embodiment obtains the steady-state response error through sliding window statistics, which involves calculating the arithmetic mean and standard deviation of the response times over the past Nr service cycles, where Nr ranges from 100 to 1000. The current throughput refers to the number of data records successfully processed and output by the system per unit time, measured in transactions per second. This embodiment obtains the current throughput through a throughput counter, a performance monitoring component based on atomic operations to count successfully processed transactions. The current default rate refers to the percentage of business requests that violate the service level agreement (SLA) commitments regarding response time or availability metrics per unit time, ranging from 0 to Dv to 1. This embodiment calculates the current default rate using the ratio of the SLA monitoring probe to the timeout counter. The baseline throughput refers to the reference value used as a normalized baseline when calculating the basic quality level, with a value of Pt0 = 100. 00TPS, the benchmark throughput corresponds to the typical throughput of a medium-sized distributed system, and is used as a normalized benchmark to establish a relative proportional relationship of quality degradation. The benchmark response error is a reference value used as a normalized benchmark when calculating the basic quality level, with a value of E0=1000ms. The benchmark response corresponds to the baseline requirements of the service level agreement of enterprise-level applications, and is used as a normalized benchmark to establish a relative proportional relationship of response error. The benchmark throughput is a reference value used as a normalized benchmark when calculating the basic quality level, with a value of Pt0=10000TPS. The benchmark throughput corresponds to the typical throughput of a medium-sized distributed system. The benchmark default rate is a reference value used as a normalized benchmark when calculating the basic quality level, with a value of D0=0.001. The benchmark default rate corresponds to the typical default level of enterprise-level service level agreements. The benchmark rollback duration is a reference value used as a normalized benchmark when calculating the basic quality level, with a value of T0=60s. The benchmark rollback duration corresponds to the repair cycle of typical narrow-scenarios.

[0037] Specifically, setting the data confidence threshold Jc based on the basic quality level Qm refers to configuring parameters for the data quality verification rules according to the quality level segmentation strategy, specifically including: When Qm≤100, it is judged as a low-quality risk condition. The data confidence threshold Jc=0.01 is set, the original data is trusted first, and only the format is checked. When 100 < Qm ≤ 500, it is judged as a medium quality risk condition. Jc = 0.05 is set to balance data integrity and processing efficiency, and format verification, range verification and correlation verification are performed. When Qm > 500, it is judged as a high-quality risk condition. Jc = 0.1 is set to increase the verification depth to tolerate short-term quality degradation of the data source, while reducing the trust in the potentially distorted original data. Full verification, fuzzy matching and manual review are performed.

[0038] Specifically, the threshold value for the low-quality risk condition is Qm≤1, the reason for which is: when Es=E0=1000ms and Pt=Pt0=10000TPS and Dv=D0=0.001 and Trq=0, Qm=1×1×1 / 1=1, this state corresponds to the system being at the baseline performance boundary, exceeding this boundary indicates that the risk begins to accumulate; the threshold value for the medium-quality risk condition is 1<Qm≤10, the reason for which is: when Es=2000ms and Pt=5000TPS and Dv=D0=0.001 and Trq=0, Qm=1×1×1 / 1=1, this state corresponds to the system being at the baseline performance boundary, exceeding this boundary indicates that the risk begins to accumulate; When v=0.01 and Trq=0, Qm=2×2×10 / 1=40. This state has far exceeded the upper limit of medium risk. Taking 10 as the boundary between medium and high risk can cover typical deterioration scenarios. The threshold value of the high-quality risk condition is Qm>10. The reason for this value is: when the system experiences severe congestion or schema mutation, Es≥5000ms and Pt≤2000TPS and Dv≥0.05, Qm≥5×5×50 / 1=1250, which is far beyond 10, indicating that the system has entered a high-risk out-of-control state.

[0039] Specifically, the step of performing drift correction on the basic quality estimation strategy based on real-time data quality residual data to obtain the target quality estimation strategy includes: The real-time data quality residual data is acquired, and the real-time data quality residual data includes semantic deviation Δs and cleaning rollback rate Δc. The quality residual index Qr is calculated based on the semantic deviation Δs, the cleaning rollback rate Δc, the first deviation weight γ1 and the second rollback weight γ2. Qr is set as Qr = γ1 × Δs / Sm + γ2 × Δc / Cm, where Sm is the maximum allowable deviation and Cm is the maximum allowable rollback rate. The quality residual index Qr is compared with the preset residual index Q0. The data quality status is judged based on the comparison result, and the basic quality estimation strategy is drift-corrected based on the judgment result, wherein: When Qr≤Q0, the data quality status is determined to be excellent, and no drift correction is applied to the basic quality estimation strategy. When Qr > Q0, the data quality status is determined to be poor. The basic quality estimation strategy is then drift-corrected to obtain the corrected data confidence threshold J1. The corrected data confidence threshold J1 is then replaced in the basic quality estimation strategy to obtain the target quality estimation strategy.

[0040] Specifically, the semantic deviation refers to the average semantic distance between field values ​​in the current data batch and the benchmark semantic dictionary, with a value range of 0 ≤ Δs ≤ 1. In this embodiment, the semantic deviation is obtained through a word embedding model or ontology alignment algorithm. The word embedding model is a semantic analysis component based on Word2Vec or BERT that calculates the cosine distance between field values ​​and benchmark concept vectors. The cleaning rollback rate refers to the proportion of the cleaning process that rolls back from the standard mode to the degraded mode due to schema mismatch within a unit of time, with a value range of 0 ≤ Δc ≤ 1. In this embodiment, the cleaning rollback rate is calculated through cleaning state machine monitoring. The first deviation weight refers to the weight coefficient assigned to the semantic deviation degree when calculating the quality residual index, with a value of γ1=0.6. The reason for this first deviation weight is that the semantic deviation degree Δs directly reflects the degree of deviation between the data semantic layer and the benchmark contract. The larger the deviation, the worse the data usability, and the greater the impact on downstream analysis, accounting for more than 60%. Therefore, a higher weight is assigned to prioritize suppressing data trust under high deviation. The second backoff weight refers to the weight coefficient assigned to the cleaning backoff rate when calculating the quality residual index, with a value of γ2=0.4. The reason for this second backoff weight is that the cleaning backoff rate Δc reflects the sc The frequency of hema changes triggering downgrades in the cleaning process, while affecting processing efficiency, is less significant than semantic deviation itself; therefore, it is assigned a weight of 0.4 to complement the first deviation weight. The maximum allowable deviation refers to the upper limit threshold of semantic deviation acceptable to the system, with a value of Sm = 0.2. The rationale for this maximum allowable deviation is that, according to data governance standards, the semantic deviation of key fields should be kept within 0.05 ≤ σ ≤ 0.1. Using 0.2 as the normalization benchmark can cover normal semantic fluctuations and leave a margin. The maximum allowable rollback rate refers to the highest rollback ratio to ensure the stability of the cleaning process, with a value of Cm = 0.15. The rationale for setting the maximum allowable rollback rate is as follows: when the cleaning rollback rate Δc > 15%, it indicates that more than 85% of the cleaning tasks are in standard mode, and this threshold corresponds to the boundary where the cleaning process enters an unstable state; the data quality status refers to the classification and judgment result of the current data quality reliability based on the comparison result of the quality residual index Qr and the preset residual index Q0, and the data quality status includes two types: excellent and poor; the preset residual index is the critical threshold for determining whether the data quality status is in the excellent range, with a value of Q0 = 1.0, and the preset residual index is a dimensionless normalized threshold, when Qr = 1.At 0, the corresponding semantic deviation reaches 50% of the maximum allowable value and the cleaning rollback rate drops to 50% of the maximum allowable value. Exceeding this value indicates a significant decline in data quality, requiring an increase in the verification threshold to reduce the system's response to poor-quality data. The corrected data confidence threshold refers to the value after adjusting the verification threshold using the quality residual index when the data quality status is determined to be poor. The value is J1=Jc×(1-g×(Qr-Q0) / Qmax), where g is the drift gain coefficient and Qmax is the maximum quality residual. The drift gain coefficient is a gain parameter that controls the rate of decrease of the data confidence threshold during the drift correction process, with a value of g=0.5. The reason for this drift gain coefficient value is: according to the conservative governance strategy, it is necessary to ensure that the maximum... Under the condition of quality residuals, the corrected data confidence threshold J1 ≥ 0.5Jc. When Qr = Qmax, J1 = 0.5Jc, which effectively improves the verification strictness under poor data and avoids the exhaustion of analysis samples caused by completely rejecting data. The maximum quality residual refers to the theoretical upper limit that the quality residual index Qr may reach, which is taken as Qmax = 3.0. The reason for taking the maximum quality residual is as follows: Under extreme operating conditions, such as changes in upstream system business rules and Δs ≈ Sm, and the existence of full schema rollback and Δc ≈ Cm, Qr = 0.6 × 0.2 / 0.2 + 0.4 × 0.15 / 0.15 = 1.0. Considering multiple failures and extreme environments, taking 3.0 can cover all normal operating conditions and extreme boundary conditions.

[0041] Please see Figure 4 As shown, this is a logical diagram of the quality fallback unit. The quality fallback unit performs quality fallback on the service output process of the service output module according to the target quality estimation strategy, including: The corrected data confidence threshold J1 included in the target quality estimation strategy is compared with the data quality score Pq of the current output batch of the service output module. The service output status is judged based on the comparison result, wherein: When Pq≥J1, the output quality is deemed to meet the standard, and the service output module is allowed to continue performing service output. When Pq < J1, the output quality is determined to be substandard, the current output batch of the service output module is blocked, and the current output batch is transferred to the isolation queue. At the same time, the service orchestration module is triggered to re-execute semantic compensation.

[0042] Specifically, the data quality score Pq refers to the comprehensive quality index obtained after a rapid quality sampling inspection of the current output batch of the service output module, with a value range of 0 ≤ Pq ≤ 1. In this embodiment, a lightweight validation rule engine is used to calculate the data quality score. The lightweight validation rule engine is a fast filtering component based on regular expression matching, field non-empty checks, and numerical range verification. The processing time of the lightweight validation rule engine is 1ms ≤ t ≤ 5ms. The isolation queue is a dedicated storage area used to temporarily store output batches that do not meet the quality standards. The capacity limit of the isolation queue is 10,000 records. When this limit is exceeded, an alarm is triggered and the output operation of the service output module is suspended. The re-execution of semantic compensation means that when the output quality does not meet the standards, the service orchestration module recalculates the semantic drift index Sd based on the latest schema snapshot data and updates the compensated cleaning intensity D2, forming a closed-loop quality repair.

[0043] Specifically, the corrected data confidence threshold J1 is used only for the output interception determination of the quality fallback unit and works independently of the compensated cleaning intensity D2. The value range of J1 is 0.01≤J1≤0.1. When Qr=Q0, J1=Jc, and when Qr=Qmax, J1=0.5Jc. The smaller J1 is, the stricter the quality constraint and the tighter the interception threshold. When J1=0.01, only the top 1% of data in the quality score is allowed to pass, and when J1=0.1, the top 10% of data in the quality score is allowed to pass.

[0044] Specifically, the execution cycle Te=1s for the quality fallback, the blocking delay Tdly≤50ms, and the maximum capacity of the isolation queue is 10,000 records.

[0045] Specifically, the execution cycle of the quality fallback unit is synchronized with the output cycle of the service output module, and the execution cycle Te is set to 1s. The reason for this value is that this cycle corresponds to the typical health check heartbeat interval in a microservice architecture, which ensures the real-time performance of the quality fallback while avoiding excessively frequent comparison operations that consume computing resources. The blocking delay of the quality fallback refers to the time difference between the determination that the output quality is substandard and the actual blocking operation, and the value Tdly≤50ms is taken. The reason for this value is that this delay corresponds to the interception response time of the sidecar proxy in the service mesh, ensuring that the blocking is completed before the data packet reaches the downstream system, thus avoiding dirty data penetration.

[0046] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A big data-based integrated service information processing system, characterized in that, include: The data acquisition module is used to collect multi-source heterogeneous data, real-time traffic status data, and data source schema snapshot data of the target business. The service orchestration module is used to generate a basic service scheme based on the multi-source heterogeneous data, and to perform traffic adaptation on the basic service scheme based on the real-time traffic status data to obtain an adapted basic service scheme. It is also used to perform semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain a target service scheme. The service output module is used to output the target service according to the target service plan. The quality estimation module is used to provide a quality safety net for the service output process of the service output module.

2. The integrated service information processing system based on big data according to claim 1, characterized in that, The method for generating basic service solutions based on the multi-source heterogeneous data includes: The basic resource allocation vector is calculated based on the data source identifier, data volume, business priority, and initial service window included in the multi-source heterogeneous data; The basic service parameters are set according to the basic resource allocation vector, the preset concurrency level, and the preset response threshold, and the basic service parameters are output as the basic service scheme.

3. The integrated service information processing system based on big data according to claim 2, characterized in that, The step of adapting the basic service plan to the real-time traffic status data to obtain the adapted basic service plan includes: The traffic fluctuation index is calculated based on the current request rate, historical peak rate, first rate weight, and second burst weight included in the real-time traffic status data. The traffic fluctuation index is compared with a preset fluctuation index. The traffic status is judged based on the comparison result. The traffic is adapted to the basic service scheme based on the judgment result to obtain the adapted concurrency. The adapted concurrency is then replaced in the basic service scheme to obtain the adapted basic service scheme.

4. The integrated service information processing system based on big data according to claim 3, characterized in that, The step of performing semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme includes: The semantic drift index is calculated based on the field missing rate and type conflict rate included in the schema semantic index in the schema snapshot data of the data source; The semantic drift index is compared with a preset drift index. The semantic state is judged based on the comparison result. The semantic compensation is performed on the adapted basic service scheme based on the judgment result to obtain the compensated cleaning intensity. The compensated cleaning intensity is then replaced in the adapted basic service scheme to obtain the target service scheme.

5. The integrated service information processing system based on big data according to claim 4, characterized in that, The step of performing semantic compensation on the adapted basic service scheme based on the data source schema snapshot data to obtain the target service scheme further includes: setting the rollback duration Trq according to the schema difference type, setting Trq=300s when the difference is identified as field addition or deletion, setting Trq=60s when the difference is identified as type narrowing, and setting Trq=0s when the difference is identified as encoding change, and supplementing the rollback duration Trq into the target service scheme.

6. The integrated service information processing system based on big data according to claim 1, characterized in that, The quality estimation module provides a quality safety net for the service output process of the service output module, specifically by generating a basic quality estimation strategy based on the target service plan. The target quality estimation strategy is obtained by performing drift correction on the basic quality estimation strategy based on the real-time data quality residual data. The service output process of the service output module is given a quality safety net based on the target quality estimation strategy.

7. The integrated service information processing system based on big data according to claim 6, characterized in that, The step of generating a basic quality estimation strategy based on the target service plan includes: The basic quality level is calculated based on the steady-state response error, backoff time, current throughput, and current default rate included in the target service plan; Based on the aforementioned basic quality level, a data confidence threshold is set to obtain a basic quality estimation strategy that includes the data confidence threshold.

8. The integrated service information processing system based on big data according to claim 7, characterized in that, The step of performing drift correction on the basic quality estimation strategy based on real-time data quality residual data to obtain the target quality estimation strategy includes: The real-time data quality residual data is acquired, and the real-time data quality residual data includes semantic deviation and cleaning rollback rate; The quality residual index is calculated based on the semantic deviation, the cleaning rollback rate, the first deviation weight, and the second rollback weight. The quality residual index is compared with a preset residual index. The data quality status is judged based on the comparison result. The basic quality estimation strategy is then drift-corrected based on the judgment result to obtain the corrected data confidence threshold. The corrected data confidence threshold is then replaced in the basic quality estimation strategy to obtain the target quality estimation strategy.

9. The integrated service information processing system based on big data according to claim 1, characterized in that, The step of providing quality assurance for the service output process of the service output module according to the target quality estimation strategy includes: The corrected data confidence threshold included in the target quality estimation strategy is compared with the data quality score of the current output batch of the service output module, and the service output status is judged based on the comparison result, wherein: When Pq≥J1, the output quality is deemed to meet the standard, and the service output module is allowed to continue performing service output. When Pq < J1, the output quality is determined to be substandard, the current output batch of the service output module is blocked, and the current output batch is transferred to the isolation queue. At the same time, the service orchestration module is triggered to re-execute semantic compensation.

10. The big data-based integrated service information processing system according to claim 9, characterized in that, The execution cycle of the quality fallback is Te=1s, the blocking delay Tdly≤50ms, and the capacity limit of the isolation queue is 10000 records.

Images