Intelligent processing method and system for clinical test data

Abstract

The invention discloses an intelligent processing method and system for clinical test data, and relates to the field of data processing, and the method comprises the steps: recognizing a high flow disturbance stage, and starting an uploading channel adjustment process; executing flow limiting operation in the high flow disturbance stage; obtaining a load pressure evaluation result; and completing data supplementation. According to the method, under the condition of data uploading peak or resource shortage in the clinical test process, the whole-process regulation and control of the disturbance data, such as current-limiting protection, back compensation and abnormal filing, is realized by effectively identifying the high-flow disturbance stage, dynamically adjusting the data uploading behavior and intelligently distinguishing the processing priorities of the key data and the non-key data. The method has the advantages of being high in adaptability, accurate in resource scheduling and safe and reliable in data, solves the problems of system congestion, uploading failure or important data lag in the data uploading stage, and improves the overall operation stability and the data integrity guarantee capability of a clinical test data processing system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to an intelligent processing method and system for clinical trial data. BACKGROUND

[0002] Clinical trial data refers to structured or unstructured data formed by observing, recording and sampling subjects at different trial stages (such as screening period, treatment period, follow-up period, etc.) during the development of drugs, medical devices or treatment plans. The current processing process of clinical trial data usually adopts the centralized collection and timing upload mode. After the uploaded data is transmitted into the central database through the collection terminal or the electronic data collection (EDC) system, it is stored and analyzed subsequently. The existing method mainly focuses on the encryption, desensitization, structuring and consistency processing of the data into the database. Some schemes introduce load balancing or semantic evaluation mechanism to improve the data access and semantic analysis capability.

[0003] The prior art such as Chinese invention patent with publication number CN117573812B is a clinical trial data processing method, device and related equipment. The method comprises: acquiring clinical trial text data for evaluating a clinical trial institution and / or a clinical trial project institution or a clinical trial project; constructing a vector database according to the clinical trial text data, wherein the vector database stores word block vectors of a plurality of word blocks obtained by cutting the clinical trial text data; performing similarity matching between index vectors of a plurality of predetermined evaluation indexes and word block vectors of each word block in the vector database, and acquiring clinical trial text data related to each evaluation index according to the similarity matching result; generating an evaluation result of the clinical trial institution and / or the clinical trial project according to each evaluation index and the clinical trial text data related to each evaluation index.

[0004] The prior art such as Chinese invention patent with publication number CN116244359B is a test data aggregation device, method and equipment, which relates to the field of test data collection. The method comprises receiving different types of JSON objects by using a gateway integrated with multiple different interfaces; the JSON objects include encrypted test data, signed encrypted data digest and encrypted symmetric key; converting the different types of JSON objects into different data aggregation services; distributing the same type of JSON objects to one or more service instances in the data aggregation service according to a load balancing strategy; decrypting the JSON objects in the service instance to determine the original test data; processing the original test data and storing the processed test data into a message queue and importing into a distributed system memory.

[0005] Based on the above, it can be seen that the data processing core of the prior art still stays in the intermediate scheduling between the gateway layer and the service layer, and lacks dynamic sensing and regulation ability for data fluctuation in the test phase. Some technologies introduce load balancing or semantic evaluation mechanism to improve the ability of data access and semantic analysis. However, in the actual central clinical environment, the data generation quantity of different test phases fluctuates greatly, and the traditional uploading strategy is to push at a fixed frequency, which lacks the sensing ability of phase characteristics and data disturbance, and it is difficult to guarantee the real-time and integrity of the uploading process. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides an intelligent processing method and system for clinical trial data. To achieve the above purpose, the present application is implemented by the following technical scheme: an intelligent processing method for clinical trial data, comprising:

[0007] The monitor periodically samples the upload channel state parameters of each clinical trial phase, identifies the high-flow disturbance phase, and starts the upload channel regulation process.

[0008] The flow limiting operation is performed on the high-flow disturbance phase, and the key phase slices and non-key phase slices are identified, the priority upload operation is performed on the key phase slices, and the delayed upload operation is performed on the non-key phase slices.

[0009] The CPU and IO load pressure of the intelligent processing terminal is evaluated to obtain a load pressure evaluation result, and if the load pressure evaluation result is overload, the upload of the non-key phase slices is interrupted.

[0010] The slices that have not completed uploading during the flow limiting operation are marked with a disturbance identifier, after the disturbance ends, the backfill monitoring queue is activated, the disturbance slices are pushed to the speed-limiting independent compensation channel for uploading, and the adjustment parameters of the speed-limiting independent compensation channel regulator are set to complete data backfill.

[0011] In addition, an intelligent processing system for clinical trial data is also provided, comprising:

[0012] The identification module is used for the monitor to periodically sample the upload channel state parameters of each clinical trial phase, identify the high-flow disturbance phase, and start the upload channel regulation process.

[0013] The flow limiting module is used for performing flow limiting operation on the high-flow disturbance phase, and identifying the key phase slices and non-key phase slices, performing priority upload operation on the key phase slices, and performing delayed upload operation on the non-key phase slices.

[0014] The load evaluation module is used for evaluating the CPU and IO load pressure of the intelligent processing terminal to obtain a load pressure evaluation result, and if the load pressure evaluation result is overload, the upload of the non-key phase slices is interrupted.

[0015] The backfill module is used for marking the shards that are not completed uploading during the flow limiting operation with a disturbance identifier, activating a backfill monitoring queue after the disturbance is over, pushing the disturbed shards to a speed-limiting independent compensation channel for uploading, and setting the adjustment parameters of the speed-limiting independent compensation channel regulator to complete data backfilling.

[0016] Compared with the prior art, the embodiments of the present application have at least the following beneficial effects:

[0017] (1) The present application provides an intelligent processing method for clinical trial data. In the case of data uploading peak or system resource shortage in the clinical trial process, the high-flow disturbance stage is effectively identified, the data uploading behavior is dynamically adjusted, and the processing priority of key data and non-key data is intelligently distinguished to realize the whole process regulation and control of disturbance data flow limiting protection, backfill compensation and abnormal archiving. It has the advantages of strong self-adaptability, accurate resource scheduling and reliable data security. It solves the problems of system congestion, uploading failure or important data lag in the data uploading stage in the prior art, and improves the overall operation stability and data integrity protection capability of the clinical trial data processing system.

[0018] (2) The present application accurately identifies the situation of high-flow disturbance in each stage of clinical trials by periodically sampling the state parameters of the uploading channel and combining the preset threshold data set, and matches the dynamic flow limiting control factor based on the resource bearing state to drive the dynamic refreshing of the core scheduling parameters such as uploading period, thread concurrency number, shard granularity and cache write rate, so as to ensure that orderly operation can still be maintained under high load. At the same time, through the key identification mechanism, the shards in the key stage are uploaded preferentially, and the shards in the non-key stage are uploaded with delay, so as to realize accurate resource scheduling and data priority protection.

[0019] (3) The present application senses the load state of key resources such as CPU and IO in real time. If it is determined that the resources are overloaded, the uploading of low-priority shards is interrupted immediately to avoid further aggravating the system pressure. At the same time, the shards that are not completed uploading during the flow limiting period are marked with a disturbance identifier and enter the backfill monitoring queue. After the disturbance is removed, they are divided into compensation task queues or transit channels according to different uploading states. Through the speed-limiting independent compensation channel and the parameter adaptive regulator, the interrupted shards can be backfilled to ensure the complete recovery of the disturbed data.

[0020] (4) For abnormal shards that cannot be processed within the specified survival time window during the backfill uploading process, the present application automatically triggers the archiving blocking mechanism, marks them as archiving exceptions, prevents them from being written into the database and reported to the data management terminal, and applies for degradation processing. This mechanism improves the identification and response ability of the system to abnormal data, prevents damaged or incomplete data from affecting subsequent analysis, and effectively guarantees the data processing quality and audit traceability.

[0021] Of course, implementing any of the products of the application does not necessarily require that all of the above advantages be achieved simultaneously. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flow chart of the method of the present application;

[0023] Figure 2 A schematic diagram of the system modules of the present application;

[0024] Figure 3 A schematic diagram of the logic flow of the present application;

[0025] Figure 4 An overview interface diagram of the clinical trial data management center involved in the embodiment of the present application;

[0026] Figure 5 A data interruption management interface diagram of the clinical trial data management center involved in the embodiment of the present application;

[0027] Figure 6 A data interruption management interface diagram of the clinical trial data management center involved in the embodiment of the present application (continued). DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0029] In the description of the present application, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "periphery" and the like indicate the orientation or positional relationship, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated component or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0030] Referring to Figure 1 and Figure 3 The embodiment of the present application provides an intelligent processing method for clinical trial data, which comprises:

[0031] In the embodiment of the present application, as Figure 4An overview interface of a clinical trial data management center involved in an embodiment of the present application is shown, which is used to visually present the data upload status of each stage of the current clinical trial, so as to facilitate the real-time monitoring and decision intervention of the test management personnel on the whole-process data transmission. The overview interface mainly includes a stage distribution overview area, a data upload status overview area and an upload log, wherein the stage distribution overview area is used to display the names of each clinical trial stage (such as screening period, treatment period, follow-up period, clinical phase I, phase II and phase III, etc.); the data upload status overview area is used to dynamically display the current upload channel state parameter index of each stage; and the upload log is used to display the recent upload data.

[0032] The monitor periodically samples the upload channel state parameters of each clinical trial stage, identifies the high-flow disturbance stage, and starts the upload channel adjustment process.

[0033] The monitor periodically samples the upload channel state parameters of each clinical trial stage, identifies the high-flow disturbance stage, and the specific process is as follows:

[0034] The upload channel state parameters include the slice write rate, the data volume offset rate, the channel queue length, the data ingestion delay and the I / O write queue length.

[0035] The slice write rate (bytes / min) refers to the total amount of slices successfully written to the database or cache system within a unit of time, reflecting the activity level of current data writing. The data volume offset rate (ΔData Volume Ratio) refers to the deviation ratio between the total upload data amount in the current period and the historical average upload amount of the stage, which is used to identify abnormal data fluctuations. If the offset rate is too large, it means that the current upload data amount deviates significantly from the normal level, which may be caused by stage misplacement, repeated upload or system false triggering, etc. The channel queue length (Channel Queue Length) represents the number of data tasks (or cache slices) waiting for processing in the current upload channel. A too long channel queue length indicates that the channel is congested, and data is accumulated before being transmitted to the cache or written to the database, which will affect the real-time and consistency of subsequent data if the growth continues. The data ingestion delay (Data Ingestion Delay) refers to the time difference between data generation (or slice generation) and successful writing to the database, reflecting the delay of the data processing link. The I / O write queue length (I / O Queue Length) refers to the number of data requests currently waiting to be written to the data storage, which directly reflects the load pressure of the underlying storage I / O.

[0036] The uploading channel state parameter of each clinical trial stage is compared with the preset uploading channel state parameter abnormality judgment data set in the database. If any parameter in the uploading channel state parameter of a certain clinical trial stage is greater than the corresponding comparison parameter in the uploading channel state parameter abnormality judgment data set, the clinical trial stage is identified as a high-flow disturbance stage. If all parameters in the uploading channel state parameter of a certain clinical trial stage are less than or equal to the corresponding comparison parameters in the uploading channel state parameter abnormality judgment data set, the clinical trial stage is identified as a normal-flow uploading stage.

[0037] The uploading channel state parameter abnormality judgment data set includes a shard write rate threshold, a data volume offset rate threshold, a channel queue length threshold, a data warehousing lag time threshold, and an I / O write queue length threshold.

[0038] The thresholds in the uploading channel state parameter abnormality judgment data set, including the shard write rate threshold, the data volume offset rate threshold, the channel queue length threshold, the data warehousing lag time threshold, and the I / O write queue length threshold, are mainly determined by historical statistical analysis and dynamic adjustment modeling. Specifically, for each clinical trial stage, historical data behavior in the past period (30 sampling periods in the embodiment of the present application) is collected, the mean and standard deviation are calculated, and a multiple interval is set as the initial threshold. In addition, for test data with strong stage difference, a time series prediction model (the time series prediction model is SARIMA in the embodiment of the present application) is used to fit the future short-time behavior, and the threshold is dynamically adjusted to adapt to the rhythmic fluctuation. The main role of the uploading channel state parameter abnormality judgment data set is as a judgment condition. When a parameter exceeds its set threshold range, it is considered as a high-flow disturbance stage.

[0039] As Figure 5 The data interruption management interface diagram of the clinical trial data management center involved in the embodiment of the present application is shown in the figure, which is used for real-time monitoring, classification management and compensation strategy configuration of uploading data interruption problems when a high-flow disturbance stage occurs in the clinical trial process. The current resource state, load trend and interruption task are presented.

[0040] The uploading channel adjustment process is started, and the specific process is as follows:

[0041] When a certain clinical trial stage is identified as a high-flow disturbance stage, the uploading channel adjustment process trigger channel is automatically written.

[0042] The uploading channel adjustment module uses the psutil interface to collect the current intelligent processing terminal resource usage state parameters, including the current thread number of the processing terminal, the IO load and the cache write frequency.

[0043] It should be noted that the number of threads currently running on the intelligent processing terminal reflects the number of tasks being processed in parallel by the system. The number of threads is a direct measure of the current busyness of the processor. The more threads, the more concurrent processing of upload, calculation or acquisition tasks.

[0044] The IO load reflects the current read-write operation frequency and data throughput of input / output devices such as disks or networks. IO load is one of the most critical bottleneck parameters in the upload process. Upload tasks usually involve data reading (such as local cache) and data writing (such as network sending). When the IO load is too high, the disk / network bandwidth resources are heavily occupied, and the upload task may be congested or delayed.

[0045] The cache write frequency represents the frequency or byte amount of data written to the disk or temporary storage area per unit time (e.g., per second) in the cache area. It can be indirectly measured by the disk write rate. The cache write frequency reflects the intensity of data "flushing" from the local buffer (such as file cache, memory write buffer) to the backend (such as disk, network interface).

[0046] A set of preset intelligent processing terminal resource usage state parameter carrying thresholds is extracted from the database, including a processing terminal current thread number carrying threshold, an IO load carrying threshold, and a cache write frequency carrying threshold.

[0047] Based on the current intelligent processing terminal resource usage state parameters, the resource carrying characteristic value is analyzed and processed, specifically including:

[0048]

[0049] wherein ZY is the resource carrying characteristic value, Num is the processing terminal current thread number, IOL is the IO load, BWF is the cache write frequency, Num0 is the processing terminal current thread number carrying threshold, IOL0 is the IO load carrying threshold, BWF0 is the cache write frequency carrying threshold, is the processing terminal current thread number weighting factor, is the IO load weighting factor, is the cache write frequency weighting factor.

[0050] It should be noted that the processing terminal current thread number weighting factor, the IO load weighting factor and the cache write frequency weighting factor are used for unified dimension conversion, thereby realizing comparability and additivity across resource dimensions. The processing terminal current thread number weighting factor is used to measure the proportion of concurrent scheduling pressure. The IO load weighting factor is used to reflect the proportion of read-write intensity of the disk and network interface. The cache write frequency weighting factor is used to reflect the proportion of cache flushing frequency.

[0051] It also needs to be explained that there is a certain correlation between the current number of terminal threads, IO load and cache write frequency. Although the above three parameters belong to different resource dimensions, they show significant correlation and dynamic coupling characteristics in the high flow disturbance stage. Specifically, the number of threads reflects the concurrency of the processing terminal in executing the upload related tasks, which is the active load at the task scheduling level; the IO load reflects the use intensity of disk reading and writing and network transmission in the upload process, which is the direct resource consumption performance caused by thread scheduling; when the IO interface processing capacity cannot meet the high concurrency upload request, the data will be temporarily stored in the local cache area, resulting in a significant increase in cache write frequency. Therefore, the three parameters show a progressive coupling relationship of "thread concurrency rising → IO load intensifying → cache writing frequently" in the running chain. By jointly analyzing the dynamic linkage characteristics of the three types of parameters, the upload channel adjustment module can realize real-time optimization of the control decision strategy, significantly improving the data upload efficiency and stability in the high flow disturbance stage.

[0052] The resource bearing characteristic value is mapped into the mapping set of the pre-stored resource bearing characteristic value-dynamic throttling control factor in the database to obtain the dynamic throttling control factor.

[0053] The dynamic throttling control factor is used for dynamically adjusting the data flow in the high flow disturbance stage.

[0054] The throttling operation is performed on the high flow disturbance stage, and the key stage shard and the non-key stage shard are identified, the priority upload operation is performed on the key stage shard, and the delayed upload operation is performed on the non-key stage shard.

[0055] The throttling operation is performed on the high flow disturbance stage, and the key stage shard and the non-key stage shard are identified, the priority upload operation is performed on the key stage shard, and the delayed upload operation is performed on the non-key stage shard.

[0056] The dynamic throttling control factor is extracted, and the dynamic throttling control factor is used as the adjustment weight of the flow scheduler, and is used in the core scheduling parameters of the flow scheduler, including the upload cycle interval, the number of single shards, the upload thread concurrency and the cache storage rate.

[0057] The specific action process of the dynamic throttling control factor includes:

[0058] The dynamic throttling control factor is input into the adjustment weight mapping set of the core scheduling parameter, including the upload cycle interval adjustment weight mapping set, the single shard number adjustment weight mapping set, the upload thread concurrency adjustment weight mapping set and the cache storage rate adjustment weight mapping set, to obtain the adjustment weight of the core scheduling parameter, including the upload cycle interval adjustment weight, the single shard number adjustment weight, the upload thread concurrency adjustment weight and the cache storage rate adjustment weight.

[0059] It should be noted that, in order to realize accurate and controllable dynamic adjustment of the data uploading behavior in the high-flow disturbance stage, four types of core scheduling parameter adjustment weight mapping sets are designed and maintained, which are: uploading period interval adjustment weight mapping set, single fragment number adjustment weight mapping set, uploading thread concurrency number adjustment weight mapping set and cache storage rate adjustment weight mapping set. The construction method, parameter content and acquisition mechanism of these mapping sets are as follows:

[0060] The basic form of the adjustment weight mapping set is a kind of multi-value mapping relationship table, the input is the dynamic flow limiting control factor, and the output is the adjustment weight value of the corresponding scheduling parameter. The mapping relationship is constructed by static interval mapping. For example, when the dynamic flow limiting control factor value is high, the corresponding uploading period interval adjustment weight will tend to increase the interval; and when the factor is low, the adjustment weight will tend to shorten the interval to improve the real-time processing capability.

[0061] The definition of these mapping sets is mainly based on three aspects of historical running data analysis, load performance evaluation experiment and artificial experience rule design. Specifically, in the initial deployment stage, by collecting the uploading behavior performance under different resource load conditions (such as response time, cache hit rate, write failure rate, etc.), a set of mapping reference samples is constructed. Subsequently, interpolation, regression or interval setting methods are used to set one or a group of optimal adjustment weights for each dynamic flow limiting control factor value segment. For example, when the control factor is 0.8 (i.e. high flow disturbance), the uploading period interval adjustment weight is set to 1.6, indicating that the uploading period is enlarged to 1.6 times of the original setting; when the control factor is 0.2, the weight is 0.7, indicating that the period is shortened.

[0062] Based on the adjustment weight of the core scheduling parameter, the core scheduling parameter in the flow scheduler is refreshed in real time, so as to perform flow limiting operation in the high-flow disturbance stage, which specifically includes:

[0063] The task trigger period of the flow scheduler is changed from the current value to the dynamic period driven by the uploading period interval adjustment weight, the number of fragments of the uploading task is set to the dynamic value driven by the single fragment number adjustment weight, the uploading thread concurrency number of the task thread pool is set to the dynamic value driven by the uploading thread concurrency number adjustment weight, and the rate limit value of the cache write interface is set to the dynamic value driven by the cache storage rate adjustment weight.

[0064] Adjust the upload scheduling period from a fixed interval mode (such as scheduling every 5 seconds) to a dynamic period driven by the upload scheduling period adjustment weight, which includes: when detecting that the resource load is high (such as CPU or IO occupation is too high), the adjustment weight will automatically extend the scheduling period, thereby reducing the scheduling frequency and relieving system pressure; if the system load is low, the scheduling period is automatically shortened, the data processing timeliness is improved, and the flexible control of the scheduling rhythm is realized.

[0065] Adjust the number of single upload shards from a fixed value (such as 10 shards per upload) to a dynamic value controlled by the single shard number adjustment weight. Specifically, it adjusts according to the current cache pressure and processing capacity to prevent bulk uploading from causing system congestion when resources are tight. At the same time, when resources are abundant, the number of uploaded shards can be increased to improve overall throughput efficiency, thereby achieving flexible regulation and control of data upload at the shard level.

[0066] Adjust the number of upload threads from a static thread pool configuration (such as the default 4 upload threads) to a dynamic number of concurrent threads controlled by the upload thread concurrency adjustment weight. Specifically, when in a high load state, the adjustment weight can reduce the number of concurrent threads to avoid resource contention; when the intelligent processing terminal is running stably and responsively, the number of threads can be appropriately increased to improve the upload processing capacity, ensuring that the intelligent processing terminal has sufficient adaptability to peak traffic.

[0067] Adjust the cache-to-database rate limit from a fixed write rate mode (such as 1000 per second) to a dynamic limit value driven by the cache-to-database rate adjustment weight. Specifically, limit the write rate when the load is high to control the data-to-database pressure; when the database response capability is good, relax the rate limit to speed up the data archiving efficiency, and ensure the data consistency and stability of the overall intelligent processing terminal.

[0068] Identify key phase shards and non-key phase shards, perform priority upload operations on key phase shards, and perform delayed upload operations on non-key phase shards, which includes:

[0069] Obtain the data state parameters of each shard in the high-traffic disturbance phase, including the number of data dimensions, field filling density, and number of data structures of each shard.

[0070] Number of Data Dimensions: This parameter represents the number of valid fields contained in a shard, i.e. the number of field dimensions actually filled in the data structure of the shard. For example, a clinical trial shard defines 100 fields, and if the current shard fills 80 of them, the number of data dimensions is 80. This parameter can be used to evaluate the data complexity of the current shard. The more dimensions, the more complex and complete the data structure, and the higher the required resources for processing.

[0071] Field Fill Density refers to the ratio of the number of data fields actually filled in a shard to the total number of optional fields, also known as the field fill rate. For example, if a shard defines 100 fields and 75 of them are filled with non-null data, the field fill density is 75%. Field fill density is used to assess the information saturation of the shard content. Shards with high fill densities are more likely to contain critical and valid data and should be uploaded first or stored quickly.

[0072] The number of data structures (Data Structures) refers to the number of distinct logical data substructures within a shard. For example, a shard for a treatment phase might contain multiple substructures, such as questionnaire data, laboratory test results, and visit records. This parameter reflects the structural complexity and diversity of a shard. A high number of structures indicates a high degree of logical coupling within the shard and a greater number of dependent processing flows. Therefore, these shards require prioritization or decomposition in flow control and cache scheduling.

[0073] The data state parameters of each shard are processed across shards to obtain the average data state parameters in the high-traffic disturbance stage, including the average number of data dimensions, the average field filling density, and the average number of data structures.

[0074] The data state parameters of each shard are compared with the average state parameters of the data during the high-traffic disturbance phase, and then weighted coupling is performed to obtain the relative criticality evaluation value of each shard, including:

[0075]

[0076] Among them, KEY i is the relative criticality evaluation value of the i-th shard, NDD i is the number of data dimensions of the i-th shard, FFD i The field filling density of the i-th shard, NDS i is the number of data structures of the i-th shard, is the average number of data dimensions, Fill the field with the average density, is the average number of data structures, α1 is the weighting factor for the number of data dimensions, α2 is the weighting factor for the field filling density, α3 is the weighting factor for the number of data structures, i is the shard number, i = 1, 2, 3, ..., n, and n is the total number of shards.

[0077] It should be noted that the data dimension quantity weighting factor, the field filling density weighting factor and the data structure quantity weighting factor are used together to evaluate the relative criticality evaluation value of each shard, dynamically adjust and proportionally control the influence of different data state parameters, and reasonably allocate the contribution value of various parameters to the final upload priority score. For example, the data dimension quantity may originally have a wide value range, and the field filling density value range is relatively narrow. If no weighting control is performed in the upload priority calculation, the data dimension quantity will occupy an disproportionate influence in the comprehensive score, thereby masking the scheduling value of other parameters. Therefore, by introducing the data dimension quantity weighting factor, the actual weight of the data dimension quantity in the score can be effectively compressed or amplified, realizing the dimension unification and influence balance among parameters.

[0078] It should also be noted that there is a clear structural and processing complexity correlation between the data dimension quantity, the field filling density and the data structure quantity. From the perspective of structural coupling, the data structure quantity reflects the number of logical modules contained in a shard, such as subject information, medical record, follow-up data, etc., and the existence of these structures often requires a certain number of data fields to support, so the data structure quantity and the data dimension quantity present a positive correlation trend. The more structures, the more complex the field dimension, and the resources required in the parsing and storage process also increase accordingly.

[0079] The field filling density reflects the proportion of actual content filled in the fields that should be filled. It has a cross-dependent relationship with the data dimension quantity. When the field dimension quantity is large but the filling density is low, it indicates that the shard is in an "incomplete filling" or "to be supplemented" state, and the upload value is low. When the field dimension is moderate but the filling density is high, the data is effective, and it is more meaningful to upload first. Therefore, both dimensions need to be considered to determine whether the data has the integrity of upload processing.

[0080] The three parameters together determine the complexity of data processing and the system load situation. An increase in the number of structures means that more cache or database write threads need to be created; an increase in the field dimension leads to an increase in memory and index matching overhead; and an increase in the filling density means that data write operations are more intensive.

[0081] The relative criticality evaluation value of each shard is compared with the pre-stored relative criticality evaluation threshold in the database. If the relative criticality evaluation value of a shard is greater than or equal to the relative criticality evaluation threshold, the shard is recorded as a critical stage shard.

[0082] If the relative criticality evaluation value of a shard is less than the relative criticality evaluation threshold, the shard is recorded as a non-critical stage shard.

[0083] For each key stage shard, perform the priority upload operation, including configuring the database write thread quota and cache queue allocation ratio, the specific configuration process is:

[0084] Extract the relative criticality evaluation value of each key stage shard, and the difference between the relative criticality evaluation threshold value and the relative criticality evaluation positive difference value is obtained. The mapping matching is carried out in the mapping set of the pre-stored relative criticality evaluation positive difference value-priority upload parameter configuration factor in the database to obtain the priority upload parameter configuration factor of each key stage shard. The priority upload parameter configuration factor is a standardized adjustment amount, and the priority upload parameter configuration factor is applied to the execution module of the priority upload operation to configure the database write thread quota and cache queue allocation ratio.

[0085] It should be noted that in terms of database write thread quota, for example, if the database allows concurrent execution of 20 write threads, the priority upload parameter configuration factor will determine how many threads are allocated to the key stage shard. Assuming the configuration factor is 0.6, 60% of the threads (i.e. 12 threads) are allocated to handle the data write task of the key stage shard.

[0086] In terms of cache queue allocation ratio, the priority upload parameter configuration factor is used to determine the proportion of the key stage shard in the overall cache space. For example, the total cache space is 10GB, and if the current key stage configuration factor is 0.5, 5GB of cache space will be allocated for data storage, scheduling and preprocessing of the key stage shard.

[0087] For each non-key stage shard, perform the delayed upload operation, including setting the delay duration and downsampling rate, and the relative criticality evaluation value of each non-key stage shard is subtracted from the relative criticality evaluation threshold value to obtain the relative criticality evaluation negative difference value of each key stage shard. The specific configuration process is: extract the relative criticality evaluation negative difference value and input it into the mapping set of the pre-stored relative criticality evaluation negative difference value-delayed upload parameter configuration factor in the database to obtain the delayed upload parameter configuration factor of each non-key stage shard, and apply the delayed upload parameter configuration factor to the execution module of the delayed upload operation to configure the delay duration and the downsampling rate.

[0088] It should be noted that in terms of delay duration configuration, the minimum and maximum delay duration interval (e.g. 5 minutes to 30 minutes) is preset, and the delay upload parameter configuration factor is used as a weight value to interpolate the interval to dynamically generate the final delay time. For example, if the delayed upload parameter configuration factor of a non-key stage shard is 0.6, the delayed upload of the non-key stage shard will be set to about 20 minutes.

[0089] In terms of downsampling rate configuration, the upload proportion of each shard data is dynamically determined by a configuration factor. For example, the system sets the maximum reserved proportion as 100%, the minimum reserved proportion as 10%, and when the configuration factor is 0.8, the final reserved proportion will be 28%. The reserved proportion controls the granularity of the uploaded data, such as reserving core fields, uploading data at interval time points, or compressing data structures, thereby effectively reducing the upload load.

[0090] The CPU and IO load pressure of the intelligent processing terminal is evaluated to obtain a load pressure evaluation result. If the load pressure evaluation result is overload, the non-critical stage shard interrupt upload is performed, which specifically includes:

[0091] The CPU and IO states are periodically monitored by the load state sensor to collect load cycle state parameters, including CPU average usage, IO average waiting time, memory average swap frequency, and cache average swap frequency.

[0092] The load cycle state parameter threshold set is extracted from the database, including the CPU average usage threshold, the IO average waiting time threshold, the memory average swap frequency threshold, and the cache average swap frequency threshold.

[0093] The load cycle state parameters are compared with the corresponding load cycle state parameter thresholds one by one. If any load cycle state parameter exceeds the corresponding load cycle state parameter threshold, the load pressure evaluation result is overload. If all load cycle state parameters are less than or equal to the corresponding load cycle state parameter thresholds, the load pressure evaluation result is not overload.

[0094] When the load pressure evaluation result is overload, the non-critical stage shard interrupt upload is performed.

[0095] The shards that have not been completed during the flow limiting operation are marked with a disturbance identifier. After the disturbance ends, the backfill monitoring queue is activated, the disturbance shards are pushed to the speed limit independent compensation channel for uploading, and the adjustment parameters of the speed limit independent compensation channel regulator are set to complete data backfill.

[0096] As shown in Figure 6 is a data interruption management interface diagram of the clinical trial data management center involved in the embodiment of the application. The interface is a functional extension of Figure 5 , mainly used for in-depth display of disturbance shard state information generated in the high-flow disturbance stage, and provides a real-time adjustment operation interface for the upload queue of the speed limit independent compensation channel, further supporting the fine management of the data compensation and archiving stage and the execution of system recovery strategies.

[0097] The shards that have not been completed during the flow limiting operation are marked with a disturbance identifier. After the disturbance ends, the backfill monitoring queue is activated, specifically including:

[0098] The shards that have not been uploaded during the rate limiting operation are marked with a disturbance flag and recorded as disturbed shards. The disturbance flag includes the shard ID, disturbance stage ID, rate limiting occurrence timestamp, remaining unuploaded data amount, and upload status.

[0099] Upload status includes partially written, write pending, and cache in transit.

[0100] Add each disturbed shard to the replenishment monitoring queue.

[0101] After executing the flow limiting operation, the upload channel status parameters are collected again to analyze and determine whether it is still in the high flow disturbance stage. If it is still in the high flow disturbance stage, an early warning is sent to the data management terminal.

[0102] If it is not in the high flow disturbance stage, a disturbance end signal is sent to release the flow limiting operation and obtain the flow limiting release buffer time.

[0103] The throttling buffer period is a gradual transition period after a high-traffic disturbance ends, to avoid the instantaneous resource impact of immediately restoring full data traffic. Based on the type of disturbance (e.g., sudden, continuous, or delayed), the throttling buffer period is mapped from a pre-set disturbance type-buffer period mapping rule base and applied to the upload scheduler.

[0104] After the disturbance ends, the backfill monitoring queue is activated and the upload status of each disturbed shard is queried. If the upload status of a disturbed shard is write pending, the disturbance mark is removed. If the upload status of a disturbed shard is partially written, it is pushed to the compensation upload task queue. If the upload status of a disturbed shard is cache transfer, it is pushed to the transfer queue and handed over to other data processors for processing.

[0105] Push the disturbance shards to the speed limit independent compensation channel for upload, set the adjustment parameters of the speed limit independent compensation channel regulator, and complete data replenishment. The specific processing conditions are as follows:

[0106] The disturbed shards in the compensation upload task queue are pushed to the speed-limited independent compensation channel. The speed-limited independent compensation channel is an isolated and controlled upload path used to compensate the previously interrupted shards in an orderly and speed-limited manner.

[0107] Use message middleware (such as Kafka and RabbitMQ) to obtain the queue length of the compensation upload task queue, and at the same time extract the current intelligent processing terminal resource usage status parameters, process them again to obtain the resource carrying characteristic value, and put the resource carrying characteristic value into the mapping set of resource carrying characteristic value-buffer adjustment control factor preset in the database for mapping and matching to obtain the buffer adjustment control factor of the speed-limited independent compensation channel.

[0108] The buffer adjustment control factor is used as an adjustment basis of the speed-limit independent compensation channel regulator, and acts on the adjustment parameters, including the maximum compensation concurrent upload number and the compensation upload task queue survival time window.

[0109] The specific action process of the buffer adjustment control factor includes:

[0110] The buffer adjustment control factor is input into a set value mapping set of the adjustment parameters, including a maximum compensation concurrent upload number mapping set and a compensation upload task queue survival time window mapping set, to obtain the set values of the adjustment parameters respectively, and refresh the adjustment parameters in the speed-limit independent compensation channel regulator.

[0111] The specific refresh process includes refreshing the adjustment parameters in the speed-limit independent compensation channel regulator by using a hot update mechanism.

[0112] When the compensation upload task queue is completed, it is determined that the data backfill is completed.

[0113] It also includes archiving blocking processing when there is an unfinished upload task in the compensation upload task queue, which specifically includes:

[0114] When a disturbance slice in the compensation upload task queue still has a disturbance identifier after the compensation upload task queue survival time window ends, it is determined that the upload is not completed, the write operation of the disturbance slice is intercepted, the upload state in the disturbance identifier is set to archiving exception, and is recorded in the archiving exception log table.

[0115] A data backfill failure notification is sent to the data management terminal to apply for starting archiving degradation, and the archiving degradation is used to simplify the upload data of the disturbance slice.

[0116] The archiving degradation is an emergency processing mechanism introduced in the compensation upload task process to deal with the situation that part of the disturbance slice cannot complete the upload within the limited time window due to abnormal conditions. The core purpose is to save necessary meta information as much as possible under the premise of ensuring the stable operation of the system, avoid the blockage of data processing flow and waste of resources caused by the continuous failure of individual slices, and improve the robustness and recoverability of the overall system.

[0117] Specifically, the archiving degradation is manifested as the simplification of the data processing target. When the disturbance slice cannot be successfully uploaded within the set survival time window, it is no longer continued to retry or forced to complete the complete data writing, but selectively extracts the key metadata of the slice for recording, such as slice ID, disturbance occurrence stage, upload failure state, timestamp and basic digest, etc. These core fields are used for subsequent abnormal audit and manual intervention to avoid complete data loss while reducing storage and processing burden.

[0118] In the embodiment, please refer to Figure 2As shown, the present application provides an intelligent processing system for clinical trial data, comprising:

[0119] A recognition module is configured to monitor the upload channel state parameters of each clinical trial stage in a periodic sampling manner, recognize a high-flow disturbance stage, and start an upload channel adjustment process.

[0120] A flow limiting module is configured to perform a flow limiting operation on the high-flow disturbance stage, identify critical stage fragments and non-critical stage fragments, perform a priority upload operation on the critical stage fragments, and perform a delayed upload operation on the non-critical stage fragments.

[0121] A load evaluation module is configured to evaluate the CPU and IO load pressure of the intelligent processing terminal, obtain a load pressure evaluation result, and interrupt the upload of the non-critical stage fragments if the load pressure evaluation result is overload.

[0122] A backfill module is configured to mark the fragments that have not completed upload during the flow limiting operation with a disturbance identifier, activate a backfill monitoring queue after the disturbance ends, push the disturbed fragments to a speed-limiting independent compensation channel for upload, and set the adjustment parameters of the speed-limiting independent compensation channel adjuster to complete data backfill.

[0123] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0124] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details of the application, nor limit the application to the specific embodiments. Obviously, many modifications and variations can be made to the application based on the content of the specification. The embodiments are selected and described in the specification to better explain the principles and practical applications of the application, so that those skilled in the art can better understand and utilize the application. Any modifications and variations that do not deviate from the structure of the application or exceed the scope defined by the application should be within the protection scope of the application.

Claims

1. An intelligent processing method for clinical trial data, characterized in that: include: The monitor periodically samples the upload channel status parameters at each clinical trial stage, identifies high-flow disturbance stages, and initiates the upload channel adjustment process; Perform flow control during high-traffic disturbance phases, identify key phase shards and non-key phase shards, prioritize uploading key phase shards, and delay uploading non-key phase shards; Evaluate the CPU and IO load pressure of the intelligent processing terminal and obtain the load pressure assessment result. If the load pressure assessment result is overload, the upload of non-critical fragments will be interrupted; The fragments that were not uploaded during the rate limiting operation are marked with a disturbance mark. After the disturbance ends, the replenishment monitoring queue is activated, and the disturbed fragments are pushed to the independent compensation channel for speed limit to upload. The adjustment parameters of the independent compensation channel regulator are set to complete data replenishment.

2. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The monitor periodically samples the upload channel status parameters of each clinical trial stage to identify the high flow disturbance stage. The specific process is as follows: Upload channel status parameters include shard write rate, data volume offset rate, channel queue length, data storage latency, and I / O write queue length; Compare the upload channel status parameters of each clinical trial stage with the upload channel status parameter abnormality judgment data set preset in the database. If any parameter among the upload channel status parameters of a clinical trial stage is greater than the corresponding comparison parameter in the upload channel status parameter abnormality judgment data set, then the clinical trial stage is identified as a high traffic disturbance stage; otherwise, it is identified as a normal traffic upload stage; The upload channel status parameter abnormality judgment dataset includes the shard write rate threshold, data volume offset rate threshold, channel queue length threshold, data storage delay time threshold, and I / O write queue length threshold.

3. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The specific process of starting the upload channel adjustment process is as follows: When a clinical trial phase is identified as a high-flow disturbance phase, the upload channel adjustment process trigger channel is automatically written; The upload channel adjustment module obtains the current intelligent processing terminal resource usage status parameters, including the current number of threads of the processing terminal, IO load and cache write frequency; Based on the current intelligent processing terminal resource usage status parameters, the resource carrying characteristic value is analyzed and processed, and the resource carrying characteristic value is put into the mapping set of resource carrying characteristic value-dynamic current limiting control factor pre-stored in the database for mapping matching to obtain the dynamic current limiting control factor; The dynamic flow limiting control factor is used to dynamically adjust the data flow in the high flow disturbance phase.

4. The intelligent processing method for clinical trial data according to claim 3, characterized in that: The performing of the flow limiting operation in the high flow disturbance stage specifically includes: Extract the dynamic flow limiting control factor and use it as the adjustment weight of the traffic scheduler. The dynamic flow limiting control factor acts on the core scheduling parameters of the traffic scheduler, including the upload cycle interval, the number of single shards, the number of concurrent upload threads, and the cache storage rate. The specific action process of the dynamic current limiting control factor includes: Input the dynamic current limiting control factor into the adjustment weight mapping set of the core scheduling parameters, including the upload cycle interval adjustment weight mapping set, the single shard number adjustment weight mapping set, the upload thread concurrency adjustment weight mapping set, and the cache storage rate adjustment weight mapping set, and obtain the adjustment weights of the core scheduling parameters respectively, including the upload cycle interval adjustment weight, the single shard number adjustment weight, the upload thread concurrency adjustment weight, and the cache storage rate adjustment weight; Based on the adjustment weights of the core scheduling parameters, the core scheduling parameters in the traffic scheduler are refreshed in real time, thereby performing flow limiting operations during high traffic disturbance phases. Specifically, the following operations are performed: Change the task trigger period of the traffic scheduler from the current value to a dynamic period driven by the upload period interval adjustment weight, set the number of shards of the upload task from the current value to a dynamic value driven by the single shard number adjustment weight, set the concurrent number of upload threads of the task thread pool from the current value to a dynamic value driven by the concurrent number of upload threads adjustment weight, and set the rate limit of the cache write interface from the current value to a dynamic value driven by the cache storage rate adjustment weight.

5. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The identifying of the critical stage fragments and the non-critical stage fragments, performing a priority upload operation on the critical stage fragments, and performing a delayed upload operation on the non-critical stage fragments specifically includes: Obtain the data status parameters of each shard during the high-traffic disturbance phase, including the number of data dimensions, field filling density, and number of data structures of each shard; The data state parameters of each shard are averaged across shards to obtain the average data state parameters during the high-traffic disturbance phase, including the average number of data dimensions, the average field filling density, and the average number of data structures. The data state parameters of each shard are compared with the average state parameters of the data in the high-traffic disturbance stage, and then weighted coupling is performed to obtain the relative criticality evaluation value of each shard; Compare the relative criticality evaluation value of each shard with the relative criticality evaluation threshold pre-stored in the database. If the relative criticality evaluation value of a shard is greater than or equal to the relative criticality evaluation threshold, the shard is recorded as a critical stage shard; If the relative criticality evaluation value of a shard is less than the relative criticality evaluation threshold, the shard is recorded as a non-critical stage shard; For shards in each key stage, priority upload is performed, including configuring the database write thread quota and cache queue allocation ratio. The specific configuration process is as follows: Extract the relative criticality assessment positive difference value of each key stage shard, input it into the mapping of relative criticality assessment positive difference value-priority upload parameter configuration factor pre-stored in the database, perform mapping matching to obtain the priority upload parameter configuration factor of each key stage shard, and apply the priority upload parameter configuration factor to the execution module of the priority upload operation to configure the database write thread quota and cache queue allocation ratio; For each non-critical stage shard, a delayed upload operation is performed, including setting the delay duration and the downsampling rate, and subtracting the relative criticality assessment value of each non-critical stage shard from the relative criticality assessment threshold to obtain the relative criticality assessment negative difference of each critical stage shard. The specific configuration process is: extracting the relative criticality assessment negative difference and putting it into the mapping set of the relative criticality assessment negative difference-delayed upload parameter configuration factor pre-stored in the database for mapping and matching to obtain the delayed upload parameter configuration factor of each non-critical stage shard, and applying the delayed upload parameter configuration factor to the execution module of the delayed upload operation to configure the delay duration and the downsampling rate.

6. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The CPU and IO load pressure of the intelligent processing terminal is evaluated to obtain a load pressure evaluation result. If the load pressure evaluation result is overload, uploading of non-critical segments is interrupted, specifically including: The load status sensor periodically monitors the CPU and I / O status and collects load cycle status parameters, including average CPU usage, average I / O wait time, average memory swap frequency, and average cache swap frequency. Extracting a load cycle status parameter threshold set from the database, including a CPU average usage threshold, an IO average waiting time threshold, a memory average swap frequency threshold, and a cache average swap frequency threshold; The load cycle state parameters are compared one by one with the corresponding load cycle state parameter thresholds. If any load cycle state parameter exceeds the corresponding load cycle state parameter threshold, the load pressure assessment result is overload; if all load cycle state parameters are less than or equal to the corresponding load cycle state parameter threshold, the load pressure assessment result is not overloaded; When the load pressure assessment result is overload, uploading of non-critical segments is interrupted.

7. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The fragments that have not been uploaded during the flow control operation are marked with a disturbance mark. After the disturbance ends, the replenishment monitoring queue is activated, specifically including: The shards that have not been uploaded during the rate limiting operation are marked with a disturbance flag and recorded as disturbed shards. The disturbance flag includes the shard ID, disturbance stage ID, rate limiting occurrence timestamp, remaining unuploaded data amount, and upload status. Upload status includes partially written, write pending, and cache transfer; Add each disturbed shard to the replenishment monitoring queue; After executing the flow limiting operation, re-collect the upload channel status parameters to determine whether it is still in the high flow disturbance stage. If it is still in the high flow disturbance stage, send an early warning to the data management terminal; If it is not in the high flow disturbance stage, a disturbance end signal is sent to release the flow limiting operation and obtain the flow limiting release buffer time at the same time; After the disturbance ends, the backfill monitoring queue is activated and the upload status of each disturbed shard is queried. If the upload status of a disturbed shard is write pending, the disturbance mark is removed. If the upload status of a disturbed shard is partially written, it is pushed to the compensation upload task queue. If the upload status of a disturbed shard is cache transfer, it is pushed to the transfer queue and handed over to other data processors for processing.

8. The intelligent processing method for clinical trial data according to claim 1, characterized in that: The disturbance slice is pushed to the speed limit independent compensation channel for uploading, and the adjustment parameters of the speed limit independent compensation channel regulator are set to complete the data backfill. The specific processing conditions are: Push the disturbed shards in the compensation upload task queue to a speed-limited independent compensation channel. The speed-limited independent compensation channel is an isolated, controlled upload path used to compensate for previously interrupted uploads in an orderly and speed-limited manner. Obtain the queue length of the compensation upload task queue, extract the current intelligent processing terminal resource usage status parameters, process them again to obtain the resource carrying characteristic value, and then put the resource carrying characteristic value into the mapping set of resource carrying characteristic value-buffering adjustment control factor preset in the database for mapping and matching to obtain the buffering adjustment control factor of the speed-limited independent compensation channel; The buffer adjustment control factor is used as the adjustment basis for the rate-limited independent compensation channel regulator and acts on the adjustment parameters, including the maximum number of concurrent compensation uploads and the compensation upload task queue survival window; The specific action process of buffer regulation control factors includes: Input the buffer adjustment control factor into the setting value mapping set of the adjustment parameter, including the maximum compensation concurrent upload number mapping set and the compensation upload task queue survival window mapping set, obtain the setting values ​​of the adjustment parameters respectively, and refresh the adjustment parameters in the speed limit independent compensation channel regulator; When all the upload tasks in the compensation upload task queue are completed, the data is considered to be backfilled.

9. The intelligent processing method for clinical trial data according to claim 8, characterized in that: It also includes archiving blocking processing when there are unfinished upload tasks in the compensation upload task queue, specifically including: If a disturbance shard in the compensation upload task queue still has a disturbance flag after the compensation upload task queue survival window ends, it is determined that the upload is incomplete, the write operation of the disturbance shard is intercepted, and the upload status in its disturbance flag is set to archive exception, and recorded in the archive exception log table; Send a data replenishment failure notification to the data management terminal and apply to start archiving downgrade, which is used to simplify the upload data of the disturbance shard.

10. A system using the intelligent processing method for clinical trial data according to any one of claims 1 to 9, characterized in that: The identification module is used to monitor the upload channel status parameters of each clinical trial stage during periodic sampling, identify the high-flow disturbance stage, and start the upload channel adjustment process; The flow limiting module is used to perform flow limiting operations during high-traffic disturbance phases, identify key phase shards and non-key phase shards, prioritize uploading key phase shards, and delay uploading non-key phase shards; The load assessment module is used to assess the CPU and IO load pressure of the intelligent processing terminal and obtain the load pressure assessment result. If the load pressure assessment result is overload, the upload of non-critical segments will be interrupted; The backfill module is used to mark the shards that have not been uploaded during the rate limiting operation with a disturbance mark. After the disturbance ends, it activates the backfill monitoring queue, pushes the disturbed shards to the independent rate limit compensation channel for upload, and sets the adjustment parameters of the independent rate limit compensation channel regulator to complete data backfill.

Images