Data loading method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of data processing, and relates to a data loading method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining a data access request; acquiring a data snapshot according to the data access request; extracting corresponding node coding data according to the data snapshot; decompressing the node coding data to obtain data fragments; and obtaining data structure information, and performing data assembly on the data fragments according to the data structure information to obtain request data. The method and the device can be applied to the field of digital medical treatment, can effectively realize quick and stable loading of the request data, and improve the access efficiency of the request data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, specifically to the field of digital medicine, and in particular to a data loading method, device, computer equipment and storage medium. Background Art

[0002] In the current business service system, with the in-depth development of business informatization, various system data are growing at an unprecedented rate. These data are not only an important basis for business system services, but also an indispensable resource for business systems to conduct data analysis and service optimization. However, the problem of low efficiency in business system data query is becoming increasingly prominent, becoming one of the key factors restricting the improvement of system service quality and efficiency.

[0003] Traditional data query methods, such as simple traversal queries, are inadequate when faced with such a large database. Traversal queries are not only time-consuming and lengthy, greatly affecting the timeliness of managers' acquisition of system data, but may also cause slow system response when data is intensively accessed, affecting the access experience. More seriously, when multiple users or system modules initiate query requests at the same time, that is, multi-threaded loading, the database server needs to handle a large number of concurrent requests, which not only increases the burden on the database search thread, but may also lead to frequent resource contention and lock waiting, which in turn causes data loading failure or system crash, bringing great inconvenience and security risks to business system access. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data loading method, apparatus, computer equipment and storage medium to solve the problems of low efficiency and unstable loading of medical data.

[0005] In order to solve the above technical problems, the present application embodiment provides a data loading method, which adopts the following technical solution:

[0006] Get data access request;

[0007] Acquire a data snapshot according to the data access request;

[0008] Extracting corresponding node encoding data according to the data snapshot;

[0009] Decompressing the node encoded data to obtain data fragments;

[0010] The data structure information is obtained, and the data fragments are assembled according to the data structure information to obtain the requested data.

[0011] Furthermore, the step of obtaining a data access request specifically includes:

[0012] Get access request transaction table;

[0013] querying the current access request transaction in the access request transaction table according to the current timestamp;

[0014] The data access request is generated according to the current access request transaction.

[0015] Furthermore, the step of obtaining a data snapshot according to the data access request specifically includes:

[0016] Determine the access request user according to the data access information;

[0017] Obtaining user access rights corresponding to the access requesting user;

[0018] Acquire the access information field of the data access information, and query the corresponding access data snapshot in the database according to the access information field;

[0019] The access data snapshot is authenticated according to the user access rights, and the authenticated access data snapshot is used as the data snapshot.

[0020] Furthermore, before the step of obtaining the user access rights corresponding to the access requesting user, the method further includes the following steps:

[0021] Obtaining the access data snapshot, and identifying sensitive information of the access data snapshot;

[0022] Encrypt the identified sensitive information to obtain access to encrypted data;

[0023] Obtain a user authority list, input the user authority list and the access encryption data into a preset role management model, and obtain the user access authority.

[0024] Furthermore, the step of extracting corresponding node encoding data according to the data snapshot specifically includes:

[0025] Parsing the data snapshot to obtain a snapshot information identifier and a snapshot information timestamp;

[0026] Determine the corresponding shared memory node according to the snapshot information identifier;

[0027] The shared memory node is traversed and queried according to the snapshot information timestamp to obtain the node encoding data.

[0028] Furthermore, the step of decompressing the node encoded data to obtain data fragments specifically includes:

[0029] Obtaining a data decompression algorithm corresponding to the node encoding data;

[0030] Decompressing the node coded data according to the data decompression algorithm to obtain columnar data;

[0031] The columnar data is sorted to obtain the data slices.

[0032] Furthermore, the step of obtaining the data structure information, assembling the data slices according to the data structure information, and obtaining the requested data specifically includes:

[0033] Obtaining a data identifier of the data slice, and obtaining the data structure information according to the data identifier;

[0034] Determine a data sharding strategy according to the data structure information;

[0035] Generate a data assembly plan according to the data sharding strategy;

[0036] The data fragments are assembled according to the data assembly scheme to obtain the requested data.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a data loading device, which adopts the following technical solution:

[0038] A request acquisition module is used to obtain data access requests;

[0039] A snapshot acquisition module, used to acquire a data snapshot according to the data access request;

[0040] A data extraction module, used to extract corresponding node encoding data according to the data snapshot;

[0041] A data decompression module, used to decompress the node encoded data to obtain data fragments;

[0042] The data assembly module is used to obtain data structure information, and assemble the data fragments according to the data structure information to obtain the requested data.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any of the above data loading methods when executing the computer-readable instructions.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0046] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data loading method described in any one of the above items are implemented.

[0047] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains the request information from the data access party by obtaining the data access request; obtains the data snapshot according to the data access request, thereby quickly obtaining the corresponding information of the node encoding data; extracts the corresponding node encoding data according to the data snapshot, thereby quickly obtaining the information corresponding to the data slice; decompresses the node encoding data, thereby obtaining the data slice corresponding to the request data; obtains the data structure information, and assembles the data slice according to the data structure information, thereby effectively obtaining the request data. This enables fast and stable loading of the request data and improves the efficiency of request data access. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0050] Figure 2 A flowchart of an embodiment of a data loading method according to the present application;

[0051] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0052] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0053] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S30;

[0054] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0055] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0056] Figure 8 is a structural schematic diagram of an embodiment of a data loading device according to the present application;

[0057] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0063] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0064] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0065] It should be noted that the data loading method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data loading device is generally arranged in the server.

[0066] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0067] Continue to refer Figure 2 , shows a flow chart of an embodiment of the image recognition method according to the present application. The data loading method comprises the following steps:

[0068] Step S10, obtaining a data access request;

[0069] In this embodiment, the data access request may be a medical data access request, which is an access request initiated by a user to medical-related data stored in the system. The data access request may be automatically triggered according to a scheduled task or automated process set by the system to obtain the latest medical data for statistical analysis and quality management, or may be generated by a patient or family member performing a data acquisition operation on a medical system through an online platform or mobile application provided by the hospital in order to obtain medical records, test results, treatment progress and other information, or may be generated by a doctor, nurse or other professional who needs to view a patient's medical history, test results, medication records and other data in order to formulate or adjust a treatment plan during the diagnosis and treatment process, and performs data viewing on a medical system.

[0070] Step S20, obtaining a data snapshot according to the data access request;

[0071] In this embodiment, data snapshot refers to a backup or copy generated by performing a snapshot operation on the node coding data stored in the system database or system at a specific time point, which is used to retain a certain state of the node coding data so that it can be restored or analyzed when needed. The snapshot operation on the node coding data includes: obtaining the initial state of the node coding data of each shared memory node, and performing a data consistency check on each shared memory node to ensure the integrity and accuracy of the data; after the data consistency check is completed, a snapshot operation is performed on each node at the same time to obtain the initial snapshot data; after the snapshot operation is completed, the initial snapshot data generated by each shared memory node is integrated using the network communication protocol to form a complete data snapshot. Among them, the shared memory node refers to the shared memory applied for on the business system. After completing the above steps, the integrity and availability of the data snapshot can be verified through the data recovery mechanism to ensure that the data can be quickly restored when needed.

[0072] Step S30, extracting corresponding node encoding data according to the data snapshot;

[0073] In this embodiment, the node encoding data is a data compression result generated by compressing the data fragments stored in a specific structure in a shared memory node through compression encoding. After the compression encoding is completed, the validity of data storage can be judged according to the data compression result, and it can be determined whether the compression rate reaches the preset standard, so as to perform an adjustment operation. The specific steps of the adjustment operation include: analyzing the data compression rate, obtaining the specific value of the compression result, and comparing it with the preset standard to obtain a judgment on whether the compression rate meets the standard. If the compression rate does not meet the standard, a different compression algorithm is used to recompress the data, obtain a new compression result, and perform a compression rate judgment again. When the compression rate meets the standard, the validity of data storage is evaluated, and the storage meets the requirements by checking the data integrity and storage space utilization. According to the storage validity evaluation result, if the storage does not meet the requirements, the storage strategy is adjusted, the storage space allocation is optimized, and the optimized storage solution is obtained. Through the optimized storage solution, the compression quality is analyzed, and the data integrity is determined through data recovery testing. If the data integrity is maintained, the data storage solution is finally determined, and the key parameters in the compression and storage process are recorded for subsequent optimization. The validity and reliability of the node encoding data are guaranteed by the above steps.

[0074] Step S40, decompressing the node encoded data to obtain data fragments;

[0075] In this embodiment, the decompression of the node coded data adopts the decompression technology corresponding to the above compression coding. In this embodiment, the compression coding adopts the GZ IP technology, which is a widely used lossless compression algorithm. It reduces the file size by finding and replacing repeated string sequences in the node coded data and using Huffman coding without losing any original data. The decompression technology adopts the GZ IP decompression technology corresponding to the above compression coding.

[0076] Step S50, obtaining data structure information, and assembling the data slices according to the data structure information to obtain requested data.

[0077] In this embodiment, the data structure information refers to the overall structure information of the request data, including the type, format, relationship of the data and how they are organized and managed. The data sharding strategy is determined based on the overall structure information, and the request data is segmented according to the data segmentation strategy to generate data shards. Among them, data segmentation refers to segmenting the request data in the database according to the data segmentation strategy and storing the fragmented data on different shared memory nodes.

[0078] In this embodiment, the above method can be applied to a medical service system, and the corresponding medical data can be accessed through online operations in the medical service system. Specifically, in this embodiment, the medical service system can be one or more of a medical insurance system and a disease insurance system, and the data access request is generated by a user or a doctor performing a data query or interaction in the medical service system. The data snapshot and the node encoding data are stored in the medical insurance system and the disease insurance system and obtained from the database of the above system, and the shared memory node is stored in the above system to store the data shards.

[0079] This embodiment obtains the request information from the data access party by obtaining the data access request; obtains the data snapshot according to the data access request, thereby quickly obtaining the corresponding information of the node encoding data; extracts the corresponding node encoding data according to the data snapshot, thereby quickly obtaining the information corresponding to the data slice; decompresses the node encoding data, thereby obtaining the data slice corresponding to the request data; obtains the data structure information, and assembles the data slice according to the data structure information, thereby effectively obtaining the request data. This enables fast and stable loading of the request data and improves the efficiency of request data access.

[0080] Continue to refer Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0081] Step S101, obtaining an access request transaction table;

[0082] In this embodiment, before obtaining the access request transaction table, it is necessary to establish a connection with the database. After the connection with the database is established, the access request transaction table can be extracted from the database through the transaction table extraction identifier.

[0083] Step S102, querying the current access request transaction in the access request transaction table according to the current timestamp;

[0084] In this embodiment, the access request transaction table includes header information such as transaction ID, request type, request time, patient ID, request status, and an index corresponding to the header information. A screening query is performed in the access request transaction table using the current timestamp to query the access request transaction closest to the current time.

[0085] Step S103: Generate the data access request according to the current access request transaction.

[0086] In this embodiment, the data access request is generated by analyzing the request type, requested data range and other information of the current access request transaction and constructing a request for retrieving a data snapshot based on the information.

[0087] This embodiment obtains an access request transaction table; searches the access request transaction table for the current access request transaction according to the current timestamp; generates the data access request according to the current access request transaction, thereby realizing generation of the corresponding data access request according to the access request transaction, so as to facilitate subsequent acquisition of data snapshots.

[0088] Continue to refer Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0089] Step S201, determining the access requesting user according to the data access information;

[0090] In this embodiment, the data access information is parsed to obtain user information and request parameters, and then the user identifier (such as user ID) is extracted from the user information, so as to match the user identity in the cached user status information according to the user identifier to determine the access requesting user.

[0091] Step S202, obtaining the user access rights corresponding to the access requesting user;

[0092] In this embodiment, user access rights correspond to the range of requested data accessible to the access requesting user, and user access rights are stored in a user permission table. The user permission table is extracted and the corresponding user access rights are searched in the user permission table based on the user identifier (such as user ID) corresponding to the access requesting user.

[0093] Step S203, obtaining the access information field of the data access information, and searching the database for a corresponding access data snapshot according to the access information field;

[0094] In this embodiment, key access information fields, such as data ID, data type, operation type (read, write, delete), etc., are extracted from the data access information, and then a matching query is performed in the database based on the extracted access information fields to find the access data snapshot corresponding to the information contained in the access information fields. The above matching query can be implemented through an SQL query statement.

[0095] Step S204: performing permission verification on the access data snapshot according to the user access permission, and taking the access data snapshot that passes the verification as the data snapshot.

[0096] In this embodiment, permission verification is achieved by comparing user access rights and permission requirements corresponding to access data snapshots. For example, the currently acquired access data snapshot requires users or permissions of specific roles to access. Users of specific roles can be doctors, nurses, patients, etc.

[0097] This embodiment determines the access request user according to the data access information; obtains the user access rights corresponding to the access request user; obtains the access information field of the data access information, searches the database for the corresponding access data snapshot according to the access information field; verifies the access data snapshot according to the user access rights, and uses the access data snapshot that passes the verification as the data snapshot. This effectively realizes the rapid acquisition of the corresponding data snapshot according to the data access information, so as to facilitate the subsequent extraction of node encoding data.

[0098] In some optional implementations of this embodiment, before step S202, the following steps are further included:

[0099] Obtaining the access data snapshot, and identifying sensitive information of the access data snapshot;

[0100] In this embodiment, the access data snapshot refers to a data snapshot of the node encoding data stored in the shared memory node, wherein the data snapshot refers to a backup or copy generated by a snapshot operation on the node encoding data stored in the medical database or system at a specific point in time, and is used to retain a certain state of the node encoding data for recovery or analysis when needed. The difference between the access data snapshot and the data snapshot is that the data snapshot refers to the access snapshot data that meets the user access rights of the current access request user, while the access snapshot data is a backup or copy generated by a snapshot operation based on the node encoding data. In this embodiment, the sensitive fields or content in the access data snapshot can be identified by regular expressions, natural language processing (NLP) technology, or a predefined sensitive information list, and the identified sensitive fields or content are used as sensitive information.

[0101] Encrypt the identified sensitive information to obtain access to encrypted data;

[0102] In this embodiment, the identified sensitive information can be encrypted by an encryption algorithm (such as AES, RSA, etc.). After the encryption is completed, the encrypted sensitive information and encryption key (to ensure the security and accessibility of the key) are stored to generate access encrypted data.

[0103] Obtain a user authority list, input the user authority list and the access encryption data into a preset role management model, and obtain the user access authority.

[0104] In this embodiment, user access rights correspond to access to encrypted data and non-access to encrypted data, and non-access to encrypted data corresponds to user access rights. The preset role management model adopts the RBAC (Role-Based Access Control) model. In the RBAC model, roles and permissions corresponding to these roles need to be defined. Permissions usually refer to the access and operation capabilities of resources (such as data, functions, etc.) in the system. Roles are a collection of these permissions, which are used to group users and assign specific permissions to each group of users. In this embodiment, the RBAC model assigns roles through user access lists, and assigns access rights corresponding to roles based on user permission lists and access to encrypted data, specifically including: comparing the user permission list with the request to access encrypted data, checking whether the user role has the permission to access the data, and whether any additional access conditions (such as time, location, etc.) are met. If the user permission list contains the permission to access encrypted data, the user is allowed to access the data and recorded in the user access rights; if the user permission list does not contain the permission to access encrypted data, the user is not allowed to access the encrypted data.

[0105] In this embodiment, after the user access rights are allocated, the user access rights can be updated according to the abnormal access log. The update specifically includes: obtaining abnormal access records, judging whether the access behavior meets the preset security standards, and if not, marking it as abnormal; extracting the relevant user role permission information according to the marked abnormal access records, judging whether there are security risks in its permission configuration, and if so, recording the risk information; obtaining the current permission configuration of the user role for the recorded risk information, judging its rationality, and if it is unreasonable, generating adjustment suggestions; obtaining the adjustment suggestions, judging the difference between it and the current configuration, and if there is a difference, generating an updated permission configuration; obtaining the updated permission configuration, judging whether it meets the preset rationality standards, and if so, confirming the configuration update; obtaining the confirmed configuration, judging whether it meets the preset security standards, and if so, completing the configuration update; obtaining the final permission configuration, judging whether it is consistent with the preset update standards, and if so, recording the update results.

[0106] This embodiment obtains the access data snapshot, identifies the sensitive information of the access data snapshot; encrypts the identified sensitive information to obtain access encrypted data; obtains a user permission list, and inputs the user permission list and the access encrypted data into a preset role management model, thereby effectively obtaining user access rights that are reasonably allocated based on user access rights and access encrypted data, to facilitate subsequent permission verification processing.

[0107] Continue to refer Figure 5 In some optional implementations of this embodiment, step S30 includes the following steps:

[0108] Step S301, parsing the data snapshot to obtain a snapshot information identifier and a snapshot information timestamp;

[0109] In this embodiment, the snapshot information identifier refers to a unique identifier for distinguishing different data snapshots, which can be a UUID, a timestamp plus a serial number, or other unique codes. The snapshot information timestamp refers to the exact time when the snapshot was created, which is used for accurate matching during tracking or data recovery.

[0110] Step S302, determining a corresponding shared memory node according to the snapshot information identifier;

[0111] In this embodiment, a shared memory node refers to a memory area storing node encoding data, which is created by a system application and can be accessed by multiple processes or threads. In this embodiment, a mapping table of snapshot information identifiers and shared memory nodes is pre-created, and the mapping table can be obtained by querying a database, a configuration file, or a memory, and then the corresponding shared memory node is determined by mapping the snapshot information identifier to a specific shared memory node address.

[0112] Step S303: perform traversal query in the shared memory node according to the snapshot information timestamp to obtain the node encoding data.

[0113] In this embodiment, the node encoding data refers to the business-related data stored in the shared memory node and processed by compression encoding. The data is traversed in the shared memory node at a specific time point corresponding to the snapshot information timestamp, so as to find the data record that matches or is closest to the timestamp, and extract the data record as the node encoding data.

[0114] In this embodiment, access to shared memory nodes can be synchronously controlled and mutually exclusive controlled through distributed locks and concurrency control algorithms, which specifically include the following steps: obtaining current status information of the shared memory node, and determining whether a distributed lock is required for synchronous control; if synchronous control is required, obtaining a distributed lock to obtain a locked state; performing mutually exclusive access processing on the shared memory node using a preset concurrency control algorithm; obtaining priority information of access requests to the shared memory node, and determining whether the priority information meets preset access conditions; if the access conditions are met, allowing access to the shared memory node to obtain an access result.

[0115] This embodiment parses the data snapshot to obtain a snapshot information identifier and a snapshot information timestamp; determines the corresponding shared memory node according to the snapshot information identifier; and performs traversal query in the shared memory node according to the snapshot information timestamp, thereby effectively obtaining the corresponding node encoding data according to the snapshot information identifier and the snapshot information timestamp of the data snapshot, so as to facilitate subsequent decompression processing.

[0116] Continue to refer Figure 6 In some optional implementations of this embodiment, step 40 includes the following steps:

[0117] Step S401, obtaining a data decompression algorithm corresponding to the node encoding data;

[0118] In this embodiment, the data decompression algorithm adopts the GZIP decompression algorithm, which can be obtained according to the metadata attached to the node encoding data, which can be stored in the file header, database field or special file. It can also be obtained according to the system configuration file, which lists the compression and decompression algorithms of different data types.

[0119] Step S502, decompressing the node coded data according to the data decompression algorithm to obtain column data;

[0120] In this embodiment, columnar data refers to data in a specific storage format obtained by storing data slices in columnar form in a shared memory node. The data in this specific storage format is stored in columns rather than rows. The use of this specific storage format is conducive to compression and decompression of data slices, so as to effectively ensure the data integrity and efficiency of the data slices during the compression and decompression process.

[0121] Step S503: sort the columnar data to obtain the data slices.

[0122] In this embodiment, data slicing refers to dividing the requested data into smaller and easier-to-manage data parts, and data sorting for columnar data refers to converting the columnar data into a row format or other formats that are convenient for subsequent processing and analysis.

[0123] This embodiment obtains the data decompression algorithm corresponding to the node encoding data; decompresses the node encoding data according to the data decompression algorithm to obtain columnar data; and organizes the columnar data, thereby effectively decompressing the node encoding data to obtain complete data fragments, so as to facilitate subsequent data assembly processing.

[0124] Continue to refer Figure 7 In some optional implementations of this embodiment, step 50 includes the following steps:

[0125] Step 501, obtaining a data identifier of a data slice, and obtaining the data structure information according to the data identifier;

[0126] In this embodiment, the data identifier of the data slice refers to the unique data identifier of the corresponding data slice, such as the data ID, which can be obtained through the metadata of the data slice. The data structure information includes the definition of the data slice (such as medical data, medical knowledge, diagnosis results, etc.), the data slice type (such as string, number, date, etc.), and the structural relationship between the data slices (such as sequence structure, etc.). The data structure information is generated when the request data is processed by data slicing, and can be stored in the database, and extracted from the database through the data identifier of the data slice.

[0127] Step 502, determining a data sharding strategy according to the data structure information;

[0128] In this embodiment, the data sharding strategy refers to a strategy for deciding how to disperse the request data of a complete data set into data shards. The data sharding strategy is generated when the request data is processed by data sharding. The corresponding data sharding strategy is determined by comparing the data structure information with the logic of the data sharding strategy.

[0129] Step 503, generating a data assembly plan according to the data sharding strategy;

[0130] In this embodiment, the data assembly scheme refers to a scheme for sequentially merging and processing individual data shards to form a request data scheme for a complete data set. The scheme can be generated based on the sharding rules of the data sharding strategy and the reverse logic of the sharding relationship.

[0131] Step 504: assemble the data slices according to the data assembly scheme to obtain the requested data.

[0132] In this embodiment, after the data assembly is completed, the request data can be verified for integrity to verify whether the request data is valid. The integrity verification of the request data includes: checking required fields: ensuring that all necessary fields are filled in without omissions; field format verification: checking whether the format of the data field meets predefined requirements, such as date format, telephone number format, etc.; internal logical consistency: verifying whether the logical relationship within the data is consistent, such as whether the age is consistent with the date of birth, whether the diagnosis result matches the symptom description, etc.; cross-table data consistency: if the data comes from multiple tables or systems, it is necessary to verify the consistency of these data between different tables or systems; data constraint check: use the constraints in the database (such as primary key constraints, foreign key constraints, unique constraints, etc.) to check the integrity of the data; business rule verification: verification according to business rules, such as certain medical items must be accompanied by specific examination or treatment records.

[0133] This embodiment obtains the data identifier of the data shard, obtains the data structure information according to the data identifier; determines the data sharding strategy according to the data structure information; generates a data assembly plan according to the data sharding strategy; assembles the data shards according to the data assembly plan, thereby effectively assembling according to the data shards to obtain complete and valid request data.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0135] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0136] Further references Figure 8 , as a response to the above Figure 1 In order to realize the method shown in the figure, the present application provides an embodiment of a data loading device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0137] like Figure 8 As shown, the data loading device 600 described in this embodiment includes: a request acquisition module 601, a snapshot acquisition module 602, a data extraction module 603, a data decompression module 604, and a data assembly module 605. Among them:

[0138] A request acquisition module 601 is used to acquire a data access request;

[0139] The snapshot acquisition module 602 is used to acquire a data snapshot according to the data access request;

[0140] A data extraction module 603 is used to extract corresponding node encoding data according to the data snapshot;

[0141] A data decompression module 604 is used to decompress the node encoded data to obtain data fragments;

[0142] The data assembly module 605 is used to obtain data structure information, and assemble the data fragments according to the data structure information to obtain the requested data.

[0143] By adopting the above-mentioned device, this embodiment can obtain a data access request, thereby obtaining request information from a data access party; by obtaining a data snapshot according to the data access request, the corresponding information of the node encoding data can be quickly obtained; by extracting the corresponding node encoding data according to the data snapshot, the information corresponding to the data slice can be quickly obtained; by decompressing the node encoding data, the data slice corresponding to the request data can be obtained; by obtaining data structure information, the data slice is assembled according to the data structure information, thereby effectively obtaining the request data. This enables fast and stable loading of the request data and improves the efficiency of request data access.

[0144] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 with components 71-73, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk equipped on the computer device 7, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions of the data loading method, etc. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0148] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run computer-readable instructions stored in the memory 71 or process data, such as computer-readable instructions for running the data loading method.

[0149] The network interface 73 may include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0150] By adopting the above-mentioned computer device, this embodiment can obtain data access requests, thereby obtaining request information from the data access party; by obtaining a data snapshot according to the data access request, the corresponding information of the node encoding data can be quickly obtained; by extracting the corresponding node encoding data according to the data snapshot, the information corresponding to the data slice can be quickly obtained; by decompressing the node encoding data, the data slice corresponding to the request data can be obtained; by obtaining data structure information, the data slice is assembled according to the data structure information, thereby effectively obtaining the request data. This can realize fast and stable loading of the request data and improve the efficiency of request data access.

[0151] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data loading method as described above.

[0152] This embodiment can obtain data access requests by using the above-mentioned computer-readable storage medium, thereby obtaining request information from the data access party; by obtaining a data snapshot according to the data access request, the corresponding information of the node encoding data can be quickly obtained; by extracting the corresponding node encoding data according to the data snapshot, the information corresponding to the data slice can be quickly obtained; by decompressing the node encoding data, the data slice corresponding to the request data can be obtained; by obtaining data structure information, the data slice is assembled according to the data structure information, thereby effectively obtaining the request data. This can realize fast and stable loading of the request data and improve the efficiency of request data access.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data loading method, characterized in that: The steps include: Get data access request; Acquire a data snapshot according to the data access request; Extracting corresponding node encoding data according to the data snapshot; Decompressing the node encoded data to obtain data fragments; The data structure information is obtained, and the data fragments are assembled according to the data structure information to obtain the requested data.

2. The data loading method according to claim 1, characterized in that: The step of obtaining a data access request specifically includes: Get access request transaction table; querying the current access request transaction in the access request transaction table according to the current timestamp; The data access request is generated according to the current access request transaction.

3. The data loading method according to claim 1, characterized in that: The step of obtaining a data snapshot according to the data access request specifically includes: Determine the access request user according to the data access information; Obtaining user access rights corresponding to the access requesting user; Acquire the access information field of the data access information, and query the corresponding access data snapshot in the database according to the access information field; The access data snapshot is authenticated according to the user access rights, and the authenticated access data snapshot is used as the data snapshot.

4. The data loading method according to claim 3, characterized in that: Before the step of obtaining the user access rights corresponding to the access requesting user, the following steps are also included: Obtaining the access data snapshot, and identifying sensitive information of the access data snapshot; Encrypt the identified sensitive information to obtain access to encrypted data; Obtain a user authority list, input the user authority list and the access encryption data into a preset role management model, and obtain the user access authority.

5. The data loading method according to claim 1, characterized in that: The step of extracting corresponding node encoding data according to the data snapshot specifically includes: Parsing the data snapshot to obtain a snapshot information identifier and a snapshot information timestamp; Determine the corresponding shared memory node according to the snapshot information identifier; The shared memory node is traversed and queried according to the snapshot information timestamp to obtain the node encoding data.

6. The data loading method according to claim 1, characterized in that: The step of decompressing the node encoded data to obtain data fragments specifically includes: Obtaining a data decompression algorithm corresponding to the node encoding data; Decompressing the node coded data according to the data decompression algorithm to obtain columnar data; The columnar data is sorted to obtain the data slices.

7. The data loading method according to claim 1, characterized in that: The step of obtaining data structure information, and assembling the data slices according to the data structure information to obtain the requested data specifically includes: Obtaining a data identifier of the data slice, and obtaining the data structure information according to the data identifier; Determine a data sharding strategy according to the data structure information; Generate a data assembly plan according to the data sharding strategy; The data fragments are assembled according to the data assembly scheme to obtain the requested data.

8. A data loading device, characterized in that: include: A request acquisition module is used to obtain data access requests; A snapshot acquisition module, used to acquire a data snapshot according to the data access request; A data extraction module, used to extract corresponding node encoding data according to the data snapshot; A data decompression module, used to decompress the node encoded data to obtain data fragments; The data assembly module is used to obtain data structure information, and assemble the data fragments according to the data structure information to obtain the requested data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data loading method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data loading method according to any one of claims 1 to 7 are implemented.

Images