Data processing method and device, electronic equipment, and storage medium

Abstract

The present disclosure provides a data processing method and device, electronic equipment and storage medium, relates to the field of medical big data, and can be applied to the scene of medical data research. The data processing method comprises the following steps: in response to reading an input data of a to-be-tested data set, updating a current valid data amount; in response to the updated valid data amount being less than or equal to a set threshold, obtaining a current valid data set, and calling a set median calculation method based on the valid data set to calculate the median of the to-be-tested data set; in response to the updated valid data amount being greater than the set threshold, calling a streaming median calculation method to calculate the median of the to-be-tested data set; and outputting the median. The technical scheme of the present disclosure can calculate the median of data of any data amount under the condition of limited computing resources, improves the accuracy and efficiency of calculation, and can guarantee the number of concurrent users of online services.

Description

Technical Field

[0001] This disclosure relates to the field of medical big data technology, and more specifically, to a data processing method, a data processing device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the statistical analysis of medical data, descriptive statistical analysis, especially descriptive statistical analysis of numerical data, plays an important role. It can use several key data to describe the overall situation of the dataset, such as the maximum value, average value, and median value. It is the first task in analyzing the dataset.

[0003] In median calculation, descriptive statistical analysis of small-scale data in related technologies loads all data into memory for computation; descriptive statistical analysis of large-scale data uses big data platforms, which also load all data into the memory of distributed nodes for computation. Because all data needs to be loaded into memory, the amount of data that can be computed is limited by memory resources.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium, thereby overcoming, to some extent, the problem that the amount of data that can be calculated is limited by memory resources because all data needs to be loaded into memory during the median calculation process.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0008] In response to receiving input data from the dataset to be tested, update the current number of valid data values.

[0009] In response to the updated effective value data volume being less than or equal to a set threshold, the current effective value dataset is obtained, and the median of the test dataset is calculated based on the effective value dataset by calling a set median calculation method.

[0010] in response to the updated valid value data amount being greater than the set threshold, invoking a stream median calculation method to calculate the median of the to-be-tested data set; the stream median calculation method is configured to perform a median estimation based on one valid value data each time one valid value data is read, until all data in the to-be-tested data set is read;

[0011] outputting the median.

[0012] In some example embodiments of the present disclosure, based on the foregoing scheme, the invoking the first median calculation method to calculate the median of the to-be-tested data set based on the valid value data set comprises:

[0013] calculating the median of the to-be-tested data set based on the first 1 / 2 times the set threshold maximum data of the valid value data set.

[0014] In some example embodiments of the present disclosure, based on the foregoing scheme, the calculating the median of the to-be-tested data set based on the first 1 / 2 times the set threshold maximum data of the valid value data set comprises:

[0015] adopting a binary heap to save the first 1 / 2 times the set threshold maximum data of the valid value data set;

[0016] calculating the median of the to-be-tested data set according to the data in the binary heap.

[0017] In some example embodiments of the present disclosure, based on the foregoing scheme, the adopting the binary heap to save the first times the set threshold maximum data of the updated valid value data set comprises:

[0018] in response to the updated valid value data amount being less than or equal to 1 / 2 times the set threshold, inserting the input data into the binary heap;

[0019] in response to the updated valid value data amount being greater than 1 / 2 times the set threshold and less than or equal to the set threshold, inserting the input data into the binary heap and deleting the minimum data in the binary heap.

[0020] In some example embodiments of the present disclosure, based on the foregoing scheme, the invoking the stream median calculation method to calculate the median of the to-be-tested data set comprises:

[0021] based on the input data, invoking a PSquare class method to calculate the median of the to-be-tested data set.

[0022] In some example embodiments of the present disclosure, based on the foregoing scheme, the data processing method further comprises:

[0023] In response to the input data being valid, the estimated value of the PSquare class method is updated based on the input data.

[0024] In some example embodiments of this disclosure, based on the foregoing scheme, the data processing method further includes, prior to the step of reading input data from the dataset to be tested:

[0025] In response to a configuration operation that detects a configuration setting threshold, the setting threshold is determined.

[0026] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0027] The acquisition module is used to update the current number of valid data points in response to reading input data from the dataset to be tested.

[0028] The first calculation module is used to obtain the current effective value dataset in response to the updated effective value data volume being less than or equal to a set threshold, and to calculate the median of the dataset to be tested by calling a set median calculation method based on the effective value dataset.

[0029] The second calculation module is used to call the streaming median calculation method to calculate the median of the dataset under test when the updated valid value data volume is greater than a set threshold. The streaming median calculation method is used to perform a median estimation based on each valid value data read, until all data in the dataset under test is read.

[0030] The output module is used to output the median.

[0031] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the data processing method described in any one of the preceding embodiments.

[0032] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method according to any one of the preceding claims.

[0033] The data processing method provided by the embodiments of the present disclosure comprises: updating the current valid data amount each time an input data is read; when the updated valid data amount is less than or equal to a set threshold, obtaining the current valid data set and calling a set median calculation method based on the valid data set to calculate the median of the to-be-tested data set; and when the updated valid data amount is greater than the set threshold, calling a streaming median calculation method to calculate the median of the to-be-tested data set. On the one hand, the valid data amount is updated each time an input data of the to-be-tested data set is read, and the median is calculated based on the updated valid data amount, so that the incremental reading of data and the dynamic calculation of the median can be realized, and all data of the to-be-tested data set does not need to be loaded into the memory for calculation, the resources consumed in the median calculation process are irrelevant to the size of the data amount, so that the median of data with any data amount can be calculated under the condition of limited computing resources; on the other hand, for online services, since the resources consumed in the median calculation process are irrelevant to the size of the data amount, the number of concurrent users and the stability of the online services can be ensured under the condition of limited computing resources; in addition, for small data amount data with a valid data amount less than or equal to a set threshold, the median can be calculated based on all data in the valid data set, and for large data amount data with a valid data amount greater than the set threshold, the streaming median calculation method is called to calculate the median, so that the accuracy and efficiency of the median calculation are improved.

[0034] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is clear that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art. In the drawings:

[0036] Figure 1 A schematic diagram of a data processing method according to some embodiments of the present disclosure is schematically shown;

[0037] Figure 2 A schematic diagram of another data processing method according to some embodiments of the present disclosure is schematically shown;

[0038] Figure 3 A schematic diagram of a data processing device according to some embodiments of the present disclosure is schematically shown;

[0039] Figure 4 A structural schematic diagram of a computer system of an electronic device according to some embodiments of the present disclosure is schematically shown;

[0040] Figure 5 A schematic diagram of a computer-readable storage medium according to some embodiments of the disclosure is shown schematically.

[0041] In the drawings, like or corresponding elements shown in different figures are designated with the same or corresponding reference numerals. DETAILED DESCRIPTION

[0042] Example implementations are now described with reference to the drawings; however, these implementations are merely examples of implementations and are not intended to limit the scope of what is described herein. Rather, the scope of the disclosure is to be accorded the broadest interpretation so as to encompass all related art technologies and procedures. Furthermore, description of the example implementations is in the context of a medical device; however, the example implementations are not limited to medical devices.

[0043] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, and operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

[0044] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, and operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

[0045] Descriptive statistical analysis of numerical data, such as maximum, minimum, mean, median, variance, etc., can be used to describe the overall situation of the data, and has an important use in statistical analysis of data.

[0046] In the field of biomedical technology, a disease-specific database can be established, such as a breast cancer-specific database, a kidney disease-specific database, a depression-specific database, and the like, to provide a data basis for scientific research on diagnosis and treatment and prevention of the specific disease. Users can statistically analyze and preview patient data in the disease-specific database of the specific disease being studied, such as statistically analyzing patient data such as patient age, etiology, pathology, clinical characteristics, test indicators, clinical diagnosis and treatment, prognosis, and the like, and performing median statistics on numerical data in the patient data. Based on the statistical analysis results, data support can be provided for subsequent scientific research on diagnosis and treatment and prevention of the specific disease, guiding the direction of scientific research decisions. Sometimes, a scientific research project tracks a large number of patients, which can reach hundreds of thousands of people, which puts high requirements on the ability of online services to perform descriptive statistical analysis.

[0047] In related technologies, various indicators of descriptive statistical analysis, such as the maximum value, minimum value, and average value of numerical data such as patient age and vital sign test indicators in patient data, can be calculated using corresponding calculation formulas, for example, using the numerical data statistical analysis calculation methods provided by R language software packages, the numpy (Numerical Python, file access) library in the programming language Python, and the SPSS (Statistical Product and Service Solutions, statistical product and service solutions) software package. In the case of a small amount of data in the data set, the data set can be loaded into memory, and these software packages can be used directly to complete statistical analysis. However, in the case of a large amount of data, or in the case where the available memory for statistical analysis calculation is limited, such as when a background service needs to handle a large number of user calculation requests, it is not possible to load all the data into memory. For median calculation, all data needs to be sorted, otherwise the median result cannot be accurately given, which poses a challenge to median calculation of large amounts of data. Moreover, since the computing resources required for each calculation are proportional to the size of the data, for multiple users, some users have a large amount of data to calculate, and some users have a small amount of data to calculate. Due to limited resources, the number of concurrent users of the online service will affect each other, making it difficult to guarantee the number of concurrent users of the online service, and if the amount of computing resources consumed by each calculation task is not limited, the overall resource availability cannot be guaranteed.

[0048] Based on this, the scheme of the embodiments of the present disclosure is proposed. In the embodiments of the present disclosure, the to-be-tested data set is incrementally read and the statistical value is dynamically updated in a streaming data loading manner. After reading one input data, the current valid value data amount is updated. When the updated valid value data amount is less than or equal to a set threshold, the current valid value data set is obtained, and a set median calculation method is called based on the updated valid value data set to calculate the median of the to-be-tested data set. When the updated valid value data amount is greater than the set threshold, a streaming median calculation method is called to calculate the median of the to-be-tested data set. The median can reflect the data characteristics of the to-be-tested data set, and provide a basis for subsequent data research. For example, in the medical technology field, by statistically analyzing the median of the patient data in the special disease library, data support can be provided for subsequent scientific research of special disease diagnosis and treatment and prevention, and the direction of scientific research decision-making can be guided.

[0049] In the example embodiments of the present disclosure, first, a data processing method is provided, which can be applied to a terminal device, such as a mobile phone, a computer, or other electronic devices. Figure 1 A schematic diagram of a data processing method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 1 As shown, the data processing method can include the following steps:

[0050] Step S110, in response to reading one input data of the to-be-tested data set, updating the current valid value data amount;

[0051] Step S120, in response to the updated valid value data amount being less than or equal to a set threshold, obtaining the current valid value data set, and calling a set median calculation method based on the valid value data set to calculate the median of the to-be-tested data set;

[0052] Step S130, in response to the updated valid value data amount being greater than the set threshold, calling a streaming median calculation method to calculate the median of the to-be-tested data set; the streaming median calculation method is used to perform a median estimation based on one valid value data at a time until all data in the to-be-tested data set is read;

[0053] Step S140, outputting the median.

[0054] According to the data processing method in the example embodiment, on one hand, the valid value data amount is updated once for each input data of the to-be-tested data set, and the median is calculated based on the updated valid value data amount, so that the incremental reading of data and the dynamic calculation of the median can be realized, and all data of the to-be-tested data set does not need to be loaded into the memory for calculation, the resource consumption of the median calculation process is irrelevant to the size of the data amount, so that the median of data of any data amount can be calculated under the condition of limited computing resources; on the other hand, for online services, since the resource consumption of the median calculation process is irrelevant to the size of the data amount, the number of concurrent users and the stability of online services can be ensured under the condition of limited computing resources; in addition, for small data amount data with a valid value data amount less than or equal to a set threshold, all data in the valid value data set can be loaded to calculate the median, and for large data amount data with a valid value data amount greater than the set threshold, a streaming median calculation method is called to calculate the median, so that the accuracy and efficiency of the median calculation are improved.

[0055] In the following, the data processing method in the example embodiment will be further described.

[0056] In step S110, in response to reading one input data of the to-be-tested data set, the current valid value data amount is updated.

[0057] The to-be-tested data set may be, for example, patient data in a breast cancer special database, a kidney disease special database or a depression special database, and descriptive statistical analysis of numerical data is performed on the patient data, such as median calculation, data is read from the to-be-tested data set in a streaming manner, and the current valid value data amount is updated for each input data. For example, for each input data of the to-be-tested data set, the validity of the input data is detected, and if the input data is valid value data, the current valid value data amount is increased by 1, and the total number of read input data is updated. For example, if it is detected that the input data is a non-empty value, the input data is determined to be valid value data, otherwise it is invalid value data.

[0058] In step S120, in response to the updated valid value data amount being less than or equal to a set threshold, the current valid value data set is obtained, and a set median calculation method is called based on the valid value data set to calculate the median of the to-be-tested data set.

[0059] A setting threshold value can be preset, by which the size of the data quantity is divided, if the updated valid value data quantity is less than or equal to the setting threshold value, it can be determined as a small-scale data quantity, if the updated valid value data quantity is greater than the setting threshold value, it can be determined as a large-scale data quantity. For example, a configuration control for configuring the setting threshold value can be provided, in response to detecting a configuration operation of configuring the setting threshold value through the configuration control, the setting threshold value is determined according to the configuration operation, so that the setting threshold value can be flexibly selected for different application scenarios.

[0060] When the updated valid value data quantity is less than or equal to the setting threshold value, all valid value data currently read from the to-be-tested data set can be obtained to obtain the current valid value data set, and then the set median calculation method can be called to calculate the median of the to-be-tested data set. For example, calling the set median calculation method to calculate the median of the to-be-tested data set can include: sorting the valid value data set, and taking the first 1 / 2 times the setting threshold value of the maximum data to calculate the median.

[0061] In an example embodiment, calculating the median of the to-be-tested data set based on the first 1 / 2 times the setting threshold value of the maximum data of the valid value data set can include: using a binary heap to save the first 1 / 2 times the setting threshold value of the maximum data of the valid value data set; and calculating the median of the to-be-tested data set according to the data in the binary heap. For example, when using a binary heap to save the first 1 / 2 times the setting threshold value of the maximum data of the valid value data set, if the updated valid value data quantity is less than or equal to 1 / 2 times the setting threshold value, the input data is inserted into the binary heap, if the updated valid value data quantity is greater than 1 / 2 times the setting threshold value and less than or equal to the setting threshold value, the input data is inserted into the binary heap and the minimum data in the binary heap is deleted, so that the memory occupation space of the binary heap can be controlled within 1 / 2 times the setting threshold value of elements, and 1 / 2 times the setting threshold value of elements can be dynamically saved.

[0062] For example, the binary heap can be a minimum binary heap or a maximum binary heap, and the present disclosure does not make special limitations thereon.

[0063] In step S130, in response to the updated valid value data quantity being greater than the setting threshold value, a stream median calculation method is called to calculate the median of the to-be-tested data set.

[0064] When the updated valid value data amount is greater than the set threshold, a stream median calculation method can be called to dynamically calculate the median of the to-be-tested data set, that is, each time a valid value data is read, the valid value data can be used to estimate the median once to generate an estimated value of the median, until all data in the to-be-tested data set are read, and the final estimated value is the calculated median. This stream median calculation method can dynamically generate an estimated value of the median based on each read valid value data, and the read valid value data can not be stored, so the memory requirement is very small and fixed, and the amount of data that can be calculated is not limited by the availability of computing resources, and a higher precision can be obtained in the calculation of the median of a large amount of data.

[0065] For example, in an embodiment, the stream median calculation method can include a PSquare class method of Python, which can dynamically calculate a percentile, and for the median, the 50th percentile is calculated, and the observation value does not need to be stored in the calculation of the percentile. In the example embodiment of the present disclosure, each input data read from the to-be-tested data set can be used as an observation value, and the PSquare class method can be called to calculate the median of the to-be-tested data set based on the input data. When the input data is a valid value data, the median estimated value of the PSquare class method can be updated according to the input data, to realize the dynamic calculation of the median. Correspondingly, the statistical variable of the PSquare class method can be initialized as a parameter for calculating the median before the input data of the to-be-tested data set is read, and the PSquare class method can use the percentile to represent each percentile, and for the median, the statistical variable is initialized as PSquare(50).

[0066] In step S140, the median is output.

[0067] The median can be displayed at a set position of a display, can be saved, or can be sent to a specified device to guide the direction of data research.

[0068] Based on Figure 1 The data processing method of the corresponding embodiment is described further below with the set threshold being 2000, the binary heap being a minimum binary heap, and the stream median calculation method being a PSquare class method of Python as an example.

[0069] Figure 2Fig. 6 schematically shows another data processing method flow according to some embodiments of the present disclosure. In this data processing method, if the valid value data amount is less than 2000, a min-heap is used to save the top 1000 largest data, so that the median can be accurately calculated; if the valid value data amount is greater than 2000, a PSquare method is used to calculate the median. Referring to Fig. 6, the data processing method can include the following steps S210-S270. Figure 2 As shown in Fig. 6, the data processing method can include the following steps S210-S270.

[0070] In step S210, the min-heap and the statistical variables of the PSquare method are initialized.

[0071] The min-heap can be saved in the form of an array or a list in the memory. In the initialization stage, the min-heap array or list can be initialized to be empty, and the statistical variables of the PSquare method are initialized to be PSquare(50), where 50 represents that only 50 quantiles, i.e., the median, are calculated.

[0072] In step S220, the current valid value data amount is updated after each input data of the data set under test is read.

[0073] For each input data, it is detected whether the input data is valid, for example, by judging whether it is a null value. If the input data is valid, the current valid value data amount is increased by 1 to obtain an updated valid value data amount; if the input data is invalid, the next input data is read from the data set under test. Exemplarily, the total number of the read data can also be updated, and the invalid value data amount can also be updated.

[0074] In step S230, the state of the PSquare method is updated when the input data is valid.

[0075] The update method can be called to update the median estimate value of the PSquare method.

[0076] In step S240, if the updated valid value data amount is less than or equal to 1000, the input data is inserted into the min-heap, and the valid value data amount is used to find the median from the min-heap.

[0077] If the updated valid value data amount is less than or equal to 1000, the input data can be pushed into the min-heap using the insert element function heappush of the Python standard library, and then the median is found according to the binary heap sorting.

[0078] In step S250, if the updated valid value data quantity is greater than 1000 and less than or equal to 2000, the input data is inserted into the minimum binary heap and the minimum data in the minimum binary heap is deleted, and the median is found from the minimum binary heap using the valid value data quantity.

[0079] If the updated valid value data quantity is greater than 1000 and less than or equal to 2000, the input data can be pushed into the minimum binary heap using the insert element function heappush of the Python standard library, and the minimum data in the minimum binary heap can be popped out of the minimum binary heap, that is, deleted, using the pop minimum element function heappop of the Python standard library, so that the memory occupation of the minimum binary heap can be controlled within 1000 elements.

[0080] In step S260, if the updated valid value data quantity is greater than 2000, the median is calculated by calling the PSquare class method.

[0081] If the updated valid value data quantity is greater than 2000, the minimum binary heap can be abandoned and the space of the minimum binary heap can be released, and only the current estimate value of the PSquare class method can be used, and the estimated median can be obtained by the p estimate method in the PSquare class method.

[0082] In step S270, the median is output.

[0083] After obtaining the median of the data set to be measured, the median can be displayed on the display screen, can be saved, or can be sent to a specified device.

[0084] According to Figure 2 The data processing method of the corresponding embodiment can give the median of all input data currently read at any time. If the valid value data quantity is less than or equal to 1000, it means that all data is in the binary heap list, and the median can be directly found according to the binary heap sorting; if the valid value data quantity is greater than 1000 and less than or equal to 2000, the data in the binary heap is the top 1000 data before sorting, and the median can be found from the sorting using the valid value data quantity; if the valid value data quantity is greater than 2000, the binary heap is released to be empty, and the estimate value can be directly obtained from the p estimate method in the PSquare class method as the median.

[0085] According to the data processing method provided by the embodiment of the present disclosure, for service type software of multiple users, the calculation of descriptive statistical analysis of unlimited data can be efficiently processed under limited resources, the stability and accuracy of the calculation result are improved, various application occasions of descriptive statistical analysis can be met, and the universality is relatively high.

[0086] It is to be understood that even though various steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied in any way as to the order of execution or that all illustrated steps be carried out to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined into a single step, carried out in a different order, and / or split into multiple steps, etc.

[0087] In addition, in the present example embodiment, a data processing apparatus is also provided. Referring to Figure 3 As shown in the figure, the data processing apparatus 300 can include: an acquisition module 310, which can be configured to update a current valid data amount in response to reading one input data of a to-be-tested data set; a first calculation module 320, which can be configured to, in response to the updated valid data amount being less than or equal to a set threshold, acquire a current valid data set, and invoke a set median calculation method based on the valid data set to calculate a median of the to-be-tested data set; a second calculation module 330, which can be configured to, in response to the updated valid data amount being greater than the set threshold, invoke a stream median calculation method to calculate the median of the to-be-tested data set, the stream median calculation method being configured to perform a median estimation based on one valid data each time one valid data is read until all data in the to-be-tested data set is read; and an output module 340, which can be configured to output the median.

[0088] In an example embodiment of the present disclosure, based on the foregoing scheme, the first calculation module 320 can be specifically configured to calculate the median of the to-be-tested data set based on the first 1 / 2 times the set threshold maximum data of the valid data set.

[0089] In an example embodiment of the present disclosure, based on the foregoing scheme, the first calculation module 320 can include a saving unit and a calculation unit. The saving unit can be configured to save the first 1 / 2 times the set threshold maximum data of the valid data set using a binary heap; and the calculation unit can be configured to calculate the median of the to-be-tested data set according to the data in the binary heap.

[0090] In an example embodiment of the present disclosure, based on the foregoing scheme, the saving unit can be specifically configured to: in response to the updated valid data amount being less than or equal to 1 / 2 times the set threshold, insert the input data into the binary heap; and in response to the updated valid data amount being greater than 1 / 2 times the set threshold and less than or equal to the set threshold, insert the input data into the binary heap and delete the minimum data in the binary heap.

[0091] In an example embodiment of the present disclosure, based on the foregoing scheme, the second calculation module 330 can be specifically configured to: based on the input data, invoke a PSquare class method to calculate the median of the to-be-tested data set. In an example embodiment of the present disclosure, based on the foregoing scheme, the second calculation module 330 can be specifically configured to: based on the input data, invoke a PSquare class method to calculate the median of the to-be-tested data set.

[0092] In one exemplary embodiment of this disclosure, based on the foregoing scheme, the second calculation module 330 can also be used to update the estimated value of the PSquare class method according to the input data in response to the input data being valid value data.

[0093] In one exemplary embodiment of this disclosure, based on the foregoing scheme, the data processing apparatus 300 may further include a determining module, which may be used to determine a setting threshold in response to detecting a configuration operation that sets a setting threshold.

[0094] The specific details of each module of the data processing device mentioned above have been described in detail in the corresponding data processing methods, so they will not be repeated here.

[0095] It should be noted that although several modules or units of the data processing apparatus have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0096] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described data processing method is also provided.

[0097] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0098] The following reference Figure 4 To describe an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0099] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), and a display unit 440.

[0100] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 410 can perform actions such as... Figure 1 The steps shown are as follows: Step S110, in response to reading an input data of the dataset to be tested, update the current effective value data volume; Step S120, in response to the updated effective value data volume being less than or equal to a set threshold, obtain the current effective value dataset, and calculate the median of the dataset to be tested by calling the set median calculation method based on the effective value dataset; Step S130, in response to the updated effective value data volume being greater than the set threshold, call the streaming median calculation method to calculate the median of the dataset to be tested; Step S140, output the median.

[0101] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 421 and / or cache memory 422, and may further include a read-only memory (ROM) 423.

[0102] Storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0103] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0104] The electronic device 400 can also communicate with one or more external devices 470 such as a keyboard or pointing devices, a Bluetooth device, or a disk drive. These and other peripherals can be connected to the electronic device 400 by one or more peripheral interfaces 450, such as a USB port. The electronic device 400 can also include one or more buses 430, which can allow data to be exchanged between one or more of the above-described components. The above-described devices and materials will be familiar to those skilled in the art.

[0105] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0106] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores the program product capable of implementing the above-mentioned method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing the terminal device to perform the steps according to various example embodiments of the present disclosure described in the above-mentioned "example method" section of the present specification when the program product is run on the terminal device.

[0107] Reference Figure 5 As shown, the program product 500 for implementing the above-mentioned data processing method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used or combined with an instruction execution system, device, or apparatus.

[0108] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave. The computer-readable signal medium can further be any computer-readable medium that is not a storage medium. The computer-readable signal medium can be a computer-readable storage medium that is a propagated signal on a carrier wave.

[0110] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, radio frequency (RF), and the like, or any suitable combination of the foregoing.

[0111] The program code can be executed by one or more programmable processors, which can be implemented in one or more computer systems. In this context, a computer system generally includes a plurality of these programmable processors working in concert to perform a task. These computer systems can be specially constructed for the required purposes, or it can comprise one or more general-purpose computers. The program code can be stored on any suitable computer readable medium, including a non-transitory computer readable medium, which can be a computer readable storage medium or a computer readable signal medium.

[0112] Furthermore, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended for limiting purposes. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0113] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0114] Other embodiments of the present disclosure will be apparent to those skilled in the art with the accomplishment of the present disclosure as set forth in the specification and practice of the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known or customary practice in the art of the present disclosure not specifically disclosed. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0115] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method applied to a medical special disease library data analysis scene for median calculation of medical big data processing, characterized in that, The method comprises: updating a current valid data amount in response to reading one input data of patient data in at least one of a breast cancer database, a kidney disease database, or a depression database; In response to the updated valid value data being less than or equal to a set threshold, a current valid value data set is obtained, and a set median calculation method is called based on the valid value data set to calculate the median of the patient data, including: based on the first times the set threshold maximum data in the valid value data set, the median of the patient data is calculated, wherein the first times the set threshold maximum data in the valid value data set is specifically stored in a binary heap; the memory space of the binary heap is controlled within times the set threshold elements; and the median of the patient data is calculated according to the data in the binary heap. calculating a median of the patient data by invoking a stream median calculation method in response to the updated valid data amount being greater than a set threshold; the stream median calculation method is used to estimate a median based on the valid data each time one valid data is read, and the read valid data is not stored until all data in the patient data is read; outputting the median to provide data support for subsequent disease diagnosis and prevention research and guide the direction of scientific research decision-making; The first part of the method comprises the following steps: The maximum data is set to be a multiple of the set threshold value. in response to the updated valid value data amount being less than or equal to times the set threshold, inserting the input data into the binary heap; in response to the updated valid value data amount being greater than times the set threshold and less than or equal to the set threshold, inserting the input data into the binary heap and deleting the minimum data in the binary heap.

2. The data processing method according to claim 1, characterized in that, the calculation of the median of the patient data by invoking the stream median calculation method comprises: calculating the median of the patient data by invoking a PSquare class method based on the input data.

3. The data processing method according to claim 2, characterized in that, The method further comprises: updating the estimated value of the PSquare class method according to the input data in response to the input data being valid data.

4. The data processing method of claim 1, wherein, Before the input data of the patient data, the data processing method further comprises: determining the set threshold in response to detecting a configuration operation of configuring the set threshold.

5. A data processing device applied to a medical special disease library data analysis scene, used for median calculation of medical big data processing, characterized in that, The method comprises: an acquisition module configured to update a current valid data amount in response to reading one input data of patient data in at least one of a breast cancer database, a kidney disease database, or a depression database; The first calculation module is configured to, in response to the updated valid value data being less than or equal to a set threshold, acquire a current valid value data set, and calculate a median of the patient data based on a set median calculation method and the valid value data set, including: calculating the median of the patient data based on the first 2 / p×set threshold maximum data of the valid value data set, wherein the valid value data set is specifically stored in a binary heap with a memory space controlled within 2 / p×set threshold elements, and the median of the patient data is calculated based on the data in the binary heap. ​​​ a second calculation module configured to calculate a median of the patient data by invoking a stream median calculation method in response to the updated valid data amount being greater than a set threshold; the stream median calculation method is used to estimate a median based on the valid data each time one valid data is read, and the read valid data is not stored until all data in the patient data is read; an output module configured to output the median to provide data support for subsequent disease diagnosis and prevention research and guide the direction of scientific research decision-making. The first part of the step of adopting a binary heap to save the updated valid value data set The step of inserting the input data into the binary heap includes: in response to the updated valid value data amount being less than or equal to times the set threshold value, inserting the input data into the binary heap; and in response to the updated valid value data amount being greater than times the set threshold value and less than or equal to the set threshold value, inserting the input data into the binary heap and deleting the minimum data in the binary heap.

6. An electronic device, comprising: The method comprises: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the data processing method of any one of claims 1 to 4.

7. A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the data processing method of any one of claims 1 to 4.

Images