Data processing method and related device, equipment, system and storage medium

Abstract

The present application discloses a data processing method and related devices, equipment, system and storage medium, wherein the data processing method comprises: receiving a data acquisition request; wherein the data acquisition request is used to request to acquire target image data; in response to the data acquisition request, selecting at least one data cache node as a first candidate node; wherein the data cache node is used to cache image data; based on the load information of the first candidate node, selecting a target node for responding to the data acquisition request from the first candidate node; wherein the target node caches the target image data. The above scheme can be specifically used to process medical image data, and can improve data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related devices, equipment, systems and storage media. Background Art

[0002] Image data is usually of great significance in clinical practice. For example, medical staff can use medical imaging data such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) to analyze the patient's condition, plan surgery, etc.

[0003] At present, image data is generally stored in a unified manner on a server so that users can access the image data by accessing the server. However, when multiple users access the data concurrently, this method will cause a sudden increase in server pressure due to the concentrated load, which may slow down the server response speed and further affect the data processing efficiency. In view of this, how to improve data processing efficiency has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data processing method and related devices, equipment, system and storage medium.

[0005] The first aspect of the present application provides a data processing method, including: receiving a data acquisition request; wherein the data acquisition request is used to request to acquire target image data; in response to the data acquisition request, selecting at least one data cache node as a first candidate node; wherein the data cache node is used to cache image data; based on the load information of the first candidate node, selecting a target node from the first candidate node for responding to the data acquisition request; wherein the target node caches the target image data.

[0006] Therefore, a data acquisition request is received, and the data acquisition request is used to request to acquire target image data, and then in response to the data acquisition request, at least one data cache node is selected as a first candidate node, and the data cache node is used to cache image data, and then based on the load information of the first candidate node, a target node for responding to the data acquisition request is selected from the first candidate node, and the target node caches the target image data. Therefore, on the one hand, caching image data by different data cache nodes can be beneficial to dispersing the load, and on the other hand, when the data acquisition request is received, the target node is finally selected based on the load information of the screened first candidate node, which can further help balance the load. Therefore, by dispersing and balancing the load, it can help reduce the possibility of slowing down the response speed of the data cache node, and thus improve data processing efficiency.

[0007] Among them, the first candidate nodes all cache the target image data, and the load information includes the currently remaining computing resources and the total computing resources of the first candidate nodes; based on the load information of the first candidate nodes, select the target node for responding to the data acquisition request from the first candidate nodes, including: for each first candidate node, obtain the proportion of the currently remaining computing resources in the total computing resources as the selection weight of the corresponding first candidate node; select the first candidate node whose weight meets the first preset condition as the target node.

[0008] Therefore, the first candidate nodes are all cached with the target image data, and the load information includes the currently remaining computing resources and the total computing resources of the first candidate nodes, so that for each first candidate node, the proportion of the currently remaining computing resources in the total computing resources is obtained as the selection weight of the corresponding first candidate node, and the first candidate node whose selection weight meets the first preset condition is selected as the target node. Therefore, since the first candidate nodes are all cached with the target image data, it is only necessary to select the target node based on the proportion of the currently remaining computing resources in the total computing resources in the subsequent selection, which can help narrow the scope of the subsequent selection of the target node, and then can help improve the speed of selecting the target node, improve the response speed of the data acquisition request, and help improve the data processing efficiency.

[0009] Among them, the data processing method also includes: in response to acquiring new image data, based on a consistent hashing method, selecting at least one data cache node as a second candidate node, and caching the new image data to the second candidate nodes respectively; in response to a data acquisition request, selecting at least one data cache node as a first candidate node, including: in response to a data acquisition request, based on a consistent hashing method, selecting at least one data cache node as a first candidate node; wherein the first candidate nodes all cache the target image data.

[0010] Therefore, in response to acquiring new image data, based on a consistent hashing method, at least one data cache node is selected as a second candidate node, and the new image data is cached to the second candidate nodes respectively, and in response to a data acquisition request, based on a consistent hashing method, at least one data cache node is selected as a first candidate node; wherein the first candidate nodes all cache the target image data, which can facilitate the image data storage and retrieval process to be based on a consistent hashing method, thereby helping to reduce the impact on normal business such as image data storage or reading when the data storage nodes are expanded or reduced.

[0011] Among them, based on the consistent hashing method, at least one data cache node is selected as the second candidate node, including: determining the data source of the new image data, and obtaining the storage weight corresponding to the data source, and the storage weight represents the number of data cache nodes to be cached for the image data originating from the data source, and repeatedly executing based on the consistent hashing method, selecting a data cache node as the second cache node until the number of nodes as the second candidate nodes are selected.

[0012] Therefore, by determining the data source of the new image data and obtaining the storage weight corresponding to the data source, and the storage weight represents the number of data cache nodes to be cached for the image data originating from the data source, the consistent hashing method is repeatedly executed to select a data cache node as the second cache node until the number of second candidate nodes is selected. In this way, image data originating from different data sources can be cached in second candidate nodes with different numbers of nodes, which can help improve the stability of the data cache.

[0013] Among them, the first candidate nodes all have image data cached, and the load information includes the currently remaining computing resources and the total computing resources of the first candidate nodes; based on the load information of the first candidate nodes, a target node for responding to the data acquisition request is selected from the first candidate nodes, including: for each first candidate node, obtaining the proportion of the currently remaining computing resources in the total computing resources, and based on the proportion and whether the first candidate node has the target image data cached, obtaining the selection weight of the corresponding first candidate node; and selecting the first candidate node whose weight meets the first preset condition as the target node.

[0014] Therefore, the first candidate nodes all have image data cached, and the load information includes the currently remaining computing resources and the total computing resources of the first candidate nodes, so that for each first candidate node, the proportion of the currently remaining computing resources in the total computing resources is obtained, and based on the proportion and whether the first candidate node has the target image data cached, the selection weight of the corresponding first candidate node is obtained, and then the first candidate node whose selection weight meets the first preset condition is selected as the target node. Therefore, since the first candidate nodes all have image data cached, the first candidate nodes may have the target image data cached, or may not have the target image data cached, so that the subsequent decision based on the proportion of the load information in the total computing resources and whether the first candidate node has the target image data cached is used to select the target node, which can help improve the robustness of the target node selection.

[0015] Among them, based on the proportion and whether the first candidate node has cached target image data, the selection weight corresponding to the first candidate node is obtained, including: when the first candidate node has cached target image data, the sum of the proportion and the preset weight is used as the selection weight corresponding to the first candidate node; when the first candidate node does not have cached target image data, the proportion is used as the selection weight corresponding to the first candidate node.

[0016] Therefore, when the first candidate node has the target image data cached, the sum of the proportion and the preset weight is used as the selection weight corresponding to the first candidate node. When the first candidate node does not have the target image data cached, the proportion is used as the selection weight corresponding to the first candidate node. This can improve the selection weight of the first candidate node that has the target image data cached, and can help improve the robustness of the subsequent selection of the target node based on the selection weight.

[0017] Among them, the first preset condition includes: selecting the largest weight; and / or the computing resources include at least one of the following: memory, processor.

[0018] Therefore, the first preset condition is set to include: the selection weight is the largest, and the first candidate node corresponding to the largest selection weight can be used as the target node, which can help improve the robustness of selecting the target node; and the computing resources are set to include at least one of the memory and the processor, which can help improve the accuracy of the computing resource evaluation of the first candidate node, which can help improve the balance of the data cache node load.

[0019] The data processing method further includes: when it is detected that the current remaining storage resources of the data cache node meet the second preset condition, deleting part of the image data based on the requested status of each image data in the data cache node.

[0020] Therefore, when it is detected that the current remaining storage resources of the data cache node meet the second preset condition, part of the image data is deleted based on the requested situation of each image data in the data cache node. Therefore, by deleting part of the image data based on the requested situation of each image data in the data cache node, it can help reduce the possibility of being unable to cache image data due to insufficient remaining storage resources of the data cache node, and at the same time reduce the possibility of affecting user browsing due to the deletion of frequently requested image data.

[0021] The image data corresponds to a data identifier, which is used to distinguish different image data; and / or the image data includes medical image data; and / or the storage resources of the data cache node include at least one of the following: a solid state drive, a disk array.

[0022] Therefore, a data identifier is set corresponding to the image data, and the data identifier is used to distinguish different image data, which can help improve the orderliness of image data storage and reading; and by setting the image data to include medical image data, it can help to achieve distributed storage and reading of medical image data; by setting the storage resources of the data cache node to include at least one of the following: a solid-state hard drive, a disk array, it can help to improve the reading speed of the image data.

[0023] The second aspect of the present application provides a data processing device, including: a request receiving module, a node screening module and a target selection module, the request receiving module is used to receive a data acquisition request; wherein the data acquisition request is used to request to obtain target image data; the node screening module is used to select at least one data cache node as a first candidate node in response to the data acquisition request; wherein the data cache node is used to cache image data; the target selection module is used to select a target node for responding to the data acquisition request from the first candidate node based on the load information of the first candidate node; wherein the target node caches the target image data.

[0024] A third aspect of the present application provides a data processing device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the data processing method in the first aspect.

[0025] A fourth aspect of the present application provides a data scheduling system, a plurality of data caching nodes and a data processing device as in the third aspect, and the plurality of data caching nodes are respectively communicatively connected with the data processing device.

[0026] The fifth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the data processing method in the first aspect.

[0027] The above scheme receives a data acquisition request, and the data acquisition request is used to request to acquire target image data, and then in response to the data acquisition request, selects at least one data cache node as a first candidate node, and the data cache node is used to cache image data, and then based on the load information of the first candidate node, selects a target node from the first candidate node to respond to the data acquisition request, and the target node caches the target image data. Therefore, on the one hand, caching image data by different data cache nodes can be beneficial to dispersing the load, and on the other hand, when receiving the data acquisition request, ultimately selecting the target node based on the load information of the screened first candidate node can further help balance the load, so that by dispersing and balancing the load, it can help reduce the possibility of slowing down the response speed of the data cache node, and thus can improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the framework of an embodiment of the data scheduling system of the present application;

[0029] Figure 2 It is a flowchart of an embodiment of the data processing method of the present application;

[0030] Figure 3 It is a schematic diagram of the framework of an embodiment of a data processing device of the present application;

[0031] Figure 4 It is a schematic diagram of the framework of an embodiment of a data processing device of the present application;

[0032] Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0035] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0036] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of the framework of an embodiment of the data scheduling system 10 of the present application. Figure 1 As shown, in the embodiment of the present disclosure, the data scheduling system 10 may include multiple data cache nodes 11 and a data processing device 12, and the multiple data cache nodes 11 are respectively connected to the data processing device 12 in communication. Specifically, the number of data cache nodes 11 can be set according to actual application needs. For example, when the amount of cached data is large, the number of data cache nodes 11 can be set slightly larger, and when the amount of cached data is small, the number of data cache nodes 11 can be set slightly smaller, which is not limited here.

[0037] In one implementation scenario, the data cache node 11 can be connected to the data processing device 12 through wired methods such as optical fiber and network cables; in addition, the data cache node 11 can also be connected to the data processing device through wireless methods such as WiFi (Wireless Fidelity) or 4G, 5G, etc., which is not limited here.

[0038] In another implementation scenario, the data processing device 12 may include a processor and a memory coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the steps in any data processing method embodiment of the present application. For details, please refer to the following embodiments of the present application, which will not be described here. In addition, the data processing device 12 may specifically include but is not limited to: network devices such as switches.

[0039] In another implementation scenario, the data cache node 11 may be provided with computing resources such as a processor and a memory, and may also be provided with storage resources such as an SSD (Solid State Disk), a disk array (Redundant Arrays of Independent Disks, RAID), etc. The disk array may specifically include, but is not limited to: RAID 0, RAID 1, RAID 5, RAID 6, etc., which are not limited here.

[0040] In another implementation scenario, a user may request the data scheduling system 10 to obtain data cached by the data cache node 11 through a terminal such as a desktop computer, a laptop computer, a mobile phone, or a tablet computer. Specifically, the data cached by the data cache node 11 may include, but is not limited to, image data, video data, text data, etc., which are not limited here.

[0041] In a specific implementation scenario, taking the data cache node 11 caching image data as an example, medical image data such as CT and MR can be cached, and pathological image data such as microscopic images of pathological sections can also be cached, which is not limited here. Since the medical image data or pathological image data of a patient is often large, usually between 100MB and 1GB (e.g., 200MB), the load can be effectively dispersed by caching the image data in the data cache node 11.

[0042] In another specific implementation scenario, the data scheduling system 10 in the embodiment of the present disclosure can be deployed in an IDC (Internet Data Center), etc., so that it can serve one medical institution or multiple medical institutions, which is not limited here.

[0043] In another specific implementation scenario, in order to improve the robustness of data scheduling, the data scheduling system 10 may also include a database. In addition to being cached in a distributed manner in the data cache node 11, the image data can also be stored in a unified manner in the database. Therefore, when the image data is deleted by the data cache node 11 due to timeout or other reasons, the image data can still be obtained from the database, thereby improving the robustness of data scheduling.

[0044] The above scheme can greatly improve the data service capability through the data scheduling system 10 including multiple data caching nodes 11, which is conducive to supporting a large number of users to use it online at the same time. For example, each data caching node 11 can support 20 users to use it online at the same time. If the data scheduling system 10 deploys 100 data caching nodes 11, the data scheduling system 10 can support 2000 users to use it online at the same time.

[0045] See also Figure 2 , Figure 2 It is a flow chart of an embodiment of the data processing method of the present application.

[0046] Specifically, the following steps may be included:

[0047] Step S21: receiving a data acquisition request.

[0048] In the disclosed embodiment, the data acquisition request is used to request to acquire target image data. As described in the aforementioned disclosed embodiment, the image data may specifically include medical image data, such as CT image data, MR image data, etc., which are not limited here; in addition, the image data may also include pathological image data such as microscopic images of pathological sections, which are not limited here.

[0049] In an implementation scenario, in order to distinguish different image data, the image data may also be provided with a data identifier, and different image data may have different data identifiers. For example, if the image data is DICOM (Digital Imaging and Communications in Medicine), series_uid may be used as the data identifier.

[0050] In one implementation scenario, the target image data may include image data of a certain patient during a certain examination, such as CT image data of patient A in the morning; or, the target image data may also include image data of a certain patient during multiple examinations, such as CT image data of patient A in the morning and MR image data in the afternoon; or, the target image data may also include image data of multiple patients, which is not limited here. It should be noted that, regardless of whether the target image data includes image data of a certain patient during a certain examination, image data of a certain patient during multiple examinations, or image data of multiple patients, the data identifiers (such as series_id) of any two image data are different.

[0051] In one implementation scenario, a user may issue a data acquisition request through a terminal such as a desktop computer, a laptop computer, a mobile phone, or a tablet computer. For example, a doctor may issue a data acquisition request through any of the above terminals during a consultation with a patient to request to obtain the target image data of the patient. Other situations may be deduced in the same way, and examples are not given one by one here.

[0052] Step S22: In response to the data acquisition request, select at least one data cache node as a first candidate node.

[0053] As described in the aforementioned disclosed embodiments, the data cache node is used to cache image data. In order to improve the access efficiency of image data, the data cache node may be provided with storage resources, and the storage resources may include at least one of the following: a solid state drive, a disk array, and the details may refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0054] In one implementation scenario, in order to improve the randomness in the image data caching process, in response to obtaining new image data, at least one data caching node can be selected as a second candidate node based on a consistent hashing method, and the new image data can be cached to the second candidate node respectively. Correspondingly, when selecting the first candidate node, in response to the data acquisition request, at least one data caching node can be selected as a first candidate node based on a consistent hashing method, and the first candidate nodes all cache the target image data. The above method, which implements the access of image data based on a consistent hashing method, can not only improve the randomness in the image data caching process, thereby being conducive to further realizing the load balancing of data caching nodes, but also, thanks to the consistent hashing method, when the number of data caching nodes changes (such as expanding data caching nodes, or reducing data caching nodes), the impact on other data caching nodes can be reduced as much as possible, thereby being conducive to improving the robustness of the data scheduling system. For example, the data scheduling system currently includes 100 data cache nodes. During the use of these 100 data cache nodes, 20 data cache nodes can be further expanded, so that the data scheduling system can be expanded to 120 data cache nodes without affecting the normal business of these 100 data cache nodes as much as possible.

[0055] In a specific implementation scenario, the new image data may be obtained by scanning with a CT machine, an MRI machine, or other equipment, which is not limited here.

[0056] In another specific implementation scenario, the data sources of different image data may be different. For example, the image data may come from medical institution A or medical institution B, which is not limited here. In addition, the number of users and the amount of data required to be cached in different medical institutions are also different. For example, medical institution A is large in scale, and its number of users is usually large, and the amount of data required to be cached is usually large. Therefore, in order to improve the stability of data caching, more data caching nodes can be selected to cache the image data from medical institution A, or medical institution B is small in scale, and its users are usually small, and the amount of data required to be cached is usually small. Therefore, in order to improve the stability of data caching, fewer data caching nodes can be selected to cache the image data from medical institution B. Therefore, different storage weights can be set for different data sources, and the storage weight represents the number of data caching nodes to be cached for the image data from the data source. On this basis, when new image data is obtained, the data source of the new image data can be determined, and the storage weight corresponding to the data source can be obtained, so that the step of selecting a data caching node as the second candidate node based on the consistent hashing method can be repeated until a second candidate node with a number of nodes is selected. For example, for image data with a file name of "image data A", and the data source is medical institution A, the storage weight corresponding to the data source is 5, that is, the number of data cache nodes to be cached by the image data with a file name of "image data A" is 5. On this basis, different suffixes can be given to the file names of the image data, for example: "image data A_1", "image data A_2", "image data A_3", "image data A_4", "image data A_5", "image data A_6", etc., and based on the consistent hashing method, each of the above file names is used to determine the data cache node used to cache the image data. For example, for "image data A_1", based on the consistent hashing method, data cache node 1 can be determined as the second candidate node, and for "image data A_2", based on the consistent hashing method, data cache node 2 can be determined as the second candidate node, and so on, until 5 second candidate nodes are selected. Other situations can be deduced by analogy, and examples are not given one by one here.

[0057] In another specific implementation scenario, the consistent hashing method organizes the hash space into a virtual ring, and the value space of the hash function is 0 to 2. 32-1, the entire ring is organized in a clockwise direction, the point directly above the ring represents 0, the first point to the right of 0 represents 1, and so on. On this basis, each data cache node is hashed using its IP (Internet Protocol) address or host name and other keywords, so that each data cache node can be mapped to a certain position on the hash ring, and then the key value (key) of the image data can be hashed to determine its position on the hash ring, and the first candidate node or the second candidate node is selected based on the position. The specific process will not be repeated here.

[0058] In another specific implementation scenario, taking the data scheduling system including 5 data cache nodes as an example, for the convenience of description, these 5 data cache nodes can be respectively recorded as data cache node 1, data cache node 2, data cache node 3, data cache node 4, and data cache node 5, and the new image data can be recorded as o. Then, when the new image data o is obtained, based on the consistent hashing method, data cache node 4 and data cache node 5 can be used as the second candidate nodes, and the new image data o can be cached to data cache node 4 and data cache node 5. Without loss of generality, the image data cached by each data cache node at this time are: data cache node 1 {a, b, c}, data cache node 2 {c, d, e}, data cache node 3 {a, d, f}, data cache node 4 {f, g, o}, data cache node 5 {g, a, o}. It should be noted that the number of the above data cache nodes and the image data cached by each data cache node are only one possible situation in the actual application process, and other possible situations are not limited thereto.

[0059] In another specific implementation scenario, still taking the above-mentioned five data cache nodes as an example, when the target image data requested by the data acquisition request is image data d, data cache node 2 and data cache node 3 that cache the target image data d can be selected as first candidate nodes based on a consistent hashing method.

[0060] In one implementation scenario, during the caching process, the remaining storage resources of the data cache node can also be detected, and if the remaining storage resources meet the preset conditions, based on the requested status of each image data in the data cache node, some image data can be deleted. The above method can help reduce the possibility of being unable to cache image data due to insufficient remaining storage resources of the data cache node, and at the same time reduce the possibility of affecting user browsing due to the deletion of frequently requested image data.

[0061] In a specific implementation scenario, the preset condition may include: the current remaining storage resources are lower than the preset data volume. The preset data volume may be set according to the actual application situation. For example, if the image data is generally large, the preset data volume may be set larger accordingly. For example, if the image data is generally around 800MB, the preset data volume may be set to 5GB; or, for example, if the image data is generally small, the preset data volume may be set smaller accordingly. For example, if the image data is generally around 100MB, the preset data volume may be set to 600MB. This is not limited here.

[0062] In another specific implementation scenario, the request status of each image data may include: the number of times each image data is requested in a preset time period before the current time. For example, the number of times each image data is requested within the 5 days before the current time is not limited here. In addition, the deleted part of the image data may include any of the following: image data whose number of requests within the preset time period before the current time is less than the preset number; image data whose number of requests within the preset time period before the current time is ranked in the last several places according to the number of times sorted from most to least. For example, if the current storage resources of the data cache node 1 meet the aforementioned preset conditions, the request status of each image data in the data cache node 1 can be obtained: the number of times image data a is requested within the 5 days before the current time is 0 times, the number of times image data b is requested within the 5 days before the current time is 1 time, and the number of times image data c is requested within the 5 days before the current time is 2 times, then the image data whose number of requests is less than the preset number (such as 1 time) can be deleted, or, sorted from most to least according to the number of times, the image data ranked last can be deleted, which is not limited here.

[0063] In another specific implementation scenario, the LRU (Least Recently Used) method can also be directly used to detect data cache nodes during the caching process, and at least part of the image data in the data cache nodes can be deleted. The LRU method eliminates data based on the historical access records of the data. Its core idea is that "if the data has been accessed recently, it is more likely to be accessed in the future." The LRU method can be implemented by any of LRU-K, Two-queues, and Multi-Queue, which will not be described in detail here.

[0064] Step S23: Based on the load information of the first candidate node, a target node for responding to the data acquisition request is selected from the first candidate nodes.

[0065] In the disclosed embodiment, the target node caches the target image data, so that the target node can respond to the data acquisition request based on the cached target image data.

[0066] In one implementation scenario, the load information may specifically include the total computing resources currently remaining in the first candidate node. As described in the aforementioned disclosed embodiment, the computing resources may include at least one of a memory and a processor. The amount of computing resources can affect the response speed of the data storage node to the data acquisition request. For example, if the memory of the data cache node is large, more image data can be loaded; or, for example, if the processor of the data cache node has a large number of cores and threads, more data acquisition requests can be responded to simultaneously.

[0067] In one implementation scenario, the first candidate nodes are all cached with the target image data. For example, the aforementioned consistent hashing method can be used to select the first candidate nodes that are all cached with the target image data, which is not limited here. On this basis, for each first candidate node, the proportion of the currently remaining computing resources in the total computing resources can be obtained as the selection weight of the corresponding first candidate node, so that the first candidate node whose selection weight meets the preset conditions can be used as the target node. In the above method, since the first candidate nodes are all cached with the target image data, it is only necessary to select the target node based on the proportion of the currently remaining computing resources in the total computing resources in the subsequent selection, which can help to narrow the scope of the subsequent selection of the target node, and then can help to increase the speed of selecting the target node, increase the response speed of the data acquisition request, and help to improve the data processing efficiency.

[0068] In a specific implementation scenario, as described in the aforementioned disclosed embodiment, the computing resources may include at least one of the following: memory, processor. For example, the total memory size of the data cache node 1 is 16G, and the current remaining available memory size is 8G, then the current remaining computing resources of the data cache node 1 account for 50% of the total computing resources. In addition, in different operating systems, the calculation method of the proportion of the currently remaining available processor resources in the total processor resources is different. For example, each core of the processor can record the current system clock and other information when switching tasks, and the processor resource occupancy rate of each task can be counted through this information, so that the current remaining available processor resources can be calculated. The proportion of the total processor resources. In the case where the computing resources include both memory and processors, the first proportion of the currently remaining available memory resources in the total memory resources can be counted, and the second proportion of the currently remaining available processor resources in the total processor resources can be counted, so that the first proportion and the second proportion can be weighted and summed as the selection weight. For example, the first proportion is 50%, the second proportion is also 40%, the weight of the first proportion is 0.5, and the weight of the second proportion is also 0.5. After weighted summation, the selection weight can be obtained as 0.45. Other situations can be deduced by analogy, and examples are not given one by one here.

[0069] In another specific implementation scenario, the above-mentioned preset conditions may specifically include selecting the node with the largest selection weight, that is, the first candidate node with the largest selection weight may be selected as the target node. Still taking the above-mentioned five data cache nodes as an example, when the target image data is image data d, the selection weight of one of the first candidate nodes (i.e., data cache node 2) is 0.2, and the selection weight of another first candidate node (i.e., data cache node 3) is 0.1, then data cache node 2 may be selected as the target node. Other situations can be deduced by analogy, and no examples are given here one by one.

[0070] In another implementation scenario, the first candidate nodes all have image data cached, that is, the first candidate nodes may have target image data cached or may not have target image data cached. On this basis, the proportion of the currently remaining computing resources in the total computing resources can be obtained for each first candidate node, and based on the proportion and whether the first candidate node has target image data cached, the selection weight of the corresponding first candidate node is obtained, so that the first candidate node whose selection weight meets the first preset condition is selected as the target node. In the above method, since the first candidate nodes all have image data cached, the first candidate nodes may have target image data cached or may not have target image data cached, so that the target node is selected based on the proportion of the currently remaining computing resources in the total computing resources and whether the first candidate node has target image data cached, which can help improve the robustness of selecting the target node.

[0071] In a specific implementation scenario, when the first candidate node caches the target image data, the sum of the proportion and the preset weight can be used as the selection weight corresponding to the first candidate node, and when the first candidate node does not cache the target image data, the proportion can be used as the selection weight corresponding to the first candidate node. Still taking the above-mentioned five data cache nodes as an example, since the first candidate nodes all cache image data, the above-mentioned five data cache nodes can all be used as first candidate nodes, and the proportion of one of the first candidate nodes (i.e., data cache node 1) is 0.5, the proportion of one of the first candidate nodes (i.e., data cache node 2) is 0.2, the proportion of one of the first candidate nodes (i.e., data cache node 3) is 0.1, the proportion of one of the first candidate nodes (i.e., data cache node 4) is 0.6, and the proportion of the last first candidate node (i.e., data cache node 5) is 0.3. When the target image data is image data d, the proportion of one of the first candidate nodes (i.e., data cache node 2) of 0.2 can be added to the preset weight of 0.5 to obtain the selection weight of the first candidate node (i.e., data cache node 2) of 0.7, and the proportion of one of the first candidate nodes (i.e., data cache node 3) of 0.1 can be added to the preset weight of 0.5 to obtain the selection weight of the first candidate node (i.e., data cache node 3) of 0.6, and the proportions of other first candidate nodes can be used as the selection weights of the corresponding first candidate nodes, so the first candidate node with the largest selection weight, i.e., data cache node 2, can be screened out as the target node. Other situations can be deduced by analogy, and examples are not given here one by one. It should be noted that the size of the above-mentioned preset weights can be unlimited, such as it can also be set to 0.6, 0.7, 0.8, etc. In addition, in order to minimize the influence of data cache nodes that do not cache target image data on the selection of target nodes, the preset weights can be set to not less than 1, such as the preset weights can be set to 1, 1.5, 2, etc., which are not limited here.

[0072] In another implementation scenario, as described in the aforementioned disclosed embodiment, the data scheduling system may further include a database. After selecting the target node, it may further determine whether the target node's currently remaining computing resources are sufficient to respond to the data acquisition request. If not, the database may respond to the data acquisition request instead. Conversely, if sufficient, the target node may be determined to respond to the data acquisition request. The above method, after selecting the target node, further determines whether the target node's currently remaining computing resources are sufficient to respond to the data acquisition request. If not, the database may respond to the data acquisition request instead. This can help improve the robustness of data scheduling.

[0073] In a specific implementation scenario, the computing resources required to respond to the data acquisition request can be pre-set. For example, the computing resources required to respond to the data acquisition request can be set to include 200MB of memory. Then, by judging whether the remaining memory resources of the target node are not less than 200MB, it can be determined whether the target node still responds to the data acquisition request or the database responds to the data acquisition request instead. Other situations can be deduced in the same way, and examples are not given one by one here.

[0074] In another specific implementation scenario, the storage resources of the database may include an ordinary mechanical hard disk. In this case, if the database responds to the data acquisition request instead, it can also output a prompt message, which is used to prompt the user to wait patiently for the target image data to load. For example, the prompt message may include "The current data recording is slow, please wait patiently" and so on, which is not limited here.

[0075] The above scheme receives a data acquisition request, and the data acquisition request is used to request to acquire target image data, and then in response to the data acquisition request, selects at least one data cache node as a first candidate node, and the data cache node is used to cache image data, and then based on the load information of the first candidate node, selects a target node from the first candidate node to respond to the data acquisition request, and the target node caches the target image data. Therefore, on the one hand, caching image data by different data cache nodes can be beneficial to dispersing the load, and on the other hand, when receiving the data acquisition request, ultimately selecting the target node based on the load information of the screened first candidate node can further help balance the load, so that by dispersing and balancing the load, it can help reduce the possibility of slowing down the response speed of the data cache node, and thus can improve data processing efficiency.

[0076] See also Figure 3 , Figure 3 : is a schematic diagram of a framework of an embodiment of a data processing device 30 of the present application. The data processing device 30 includes a request receiving module 31, a node screening module 32 and a target selection module 33. The request receiving module 31 is used to receive a data acquisition request; wherein the data acquisition request is used to request to acquire target image data; the node screening module 32 is used to select at least one data cache node as a first candidate node in response to the data acquisition request; wherein the data cache node is used to cache image data; the target selection module 33 is used to select a target node for responding to the data acquisition request from the first candidate node based on the load information of the first candidate node; wherein the target node caches the target image data.

[0077] The above scheme receives a data acquisition request, and the data acquisition request is used to request to acquire target image data, and then in response to the data acquisition request, selects at least one data cache node as a first candidate node, and the data cache node is used to cache image data, and then based on the load information of the first candidate node, selects a target node from the first candidate node to respond to the data acquisition request, and the target node caches the target image data. Therefore, on the one hand, caching image data by different data cache nodes can be beneficial to dispersing the load, and on the other hand, when receiving the data acquisition request, ultimately selecting the target node based on the load information of the screened first candidate node can further help balance the load, so that by dispersing and balancing the load, it can help reduce the possibility of slowing down the response speed of the data cache node, and thus can improve data processing efficiency.

[0078] In some disclosed embodiments, the first candidate nodes all cache target image data, and the load information includes the currently remaining computing resources and the total computing resources of the first candidate nodes. The target selection module 33 includes a weight acquisition submodule and a target selection submodule. The weight acquisition submodule is used to obtain the proportion of the currently remaining computing resources in the total computing resources for each first candidate node as the selection weight of the corresponding first candidate node. The target selection submodule is used to select the first candidate node whose selection weight meets the first preset condition as the target node.

[0079] Different from the above-mentioned embodiment, the first candidate nodes are all cached with the target image data, so that for each first candidate node, the proportion of the currently remaining computing resources in the total computing resources is obtained as the selection weight of the corresponding first candidate node, and the first candidate node whose selection weight meets the first preset condition is selected as the target node. Therefore, since the first candidate nodes are all cached with the target image data, it is only necessary to select the target node based on the proportion of the currently remaining computing resources in the total computing resources in the subsequent selection, which can help narrow the scope of the subsequent selection of the target node, and then can help improve the speed of selecting the target node, improve the response speed of the data acquisition request, and help improve the data scheduling efficiency.

[0080] In some disclosed embodiments, the data processing device 30 also includes a data caching module for selecting at least one data caching node as a second candidate node based on a consistent hashing method in response to acquiring new image data, and caching the new image data to the second candidate nodes respectively. The node screening module is specifically used for selecting at least one data caching node as a first candidate node based on a consistent hashing method in response to a data acquisition request; wherein the first candidate nodes all cache the target image data.

[0081] Different from the aforementioned embodiment, therefore, in response to acquiring new image data, based on a consistent hashing method, at least one data cache node is selected as a second candidate node, and the new image data is cached to the second candidate nodes respectively, and in response to a data acquisition request, based on a consistent hashing method, at least one data cache node is selected as a first candidate node; wherein the first candidate nodes all cache the target image data, which can be beneficial in that the image data storage and retrieval process are based on a consistent hashing method, thereby being beneficial in that when the data storage nodes are expanded or reduced, the impact on normal business such as image data storage or reading can be reduced.

[0082] In some disclosed embodiments, the data cache module includes a weight determination submodule, which is used to determine the data source of new image data and obtain the storage weight corresponding to the data source, and the storage weight represents the number of data cache nodes to be cached for the image data originating from the data source. The data cache module includes a weight determination submodule and a node selection submodule, which is used to repeatedly execute based on a consistent hashing method to select a data cache node as a second cache node until the number of second candidate nodes is selected.

[0083] Different from the aforementioned embodiment, by determining the data source of new image data and obtaining the storage weight corresponding to the data source, and the storage weight represents the number of data cache nodes to be cached for the image data originating from the data source, a consistent hashing method is repeatedly performed to select a data cache node as the second cache node until a number of second candidate nodes are selected. Image data originating from different data sources can be cached in second candidate nodes with different numbers of nodes, which can help improve the stability of data caching.

[0084] In some disclosed embodiments, the first candidate nodes all have image data cached, and the load information includes the computing resources of the first candidate nodes in the current order and the total computing resources. The weight acquisition submodule is used to obtain the proportion of the currently remaining computing resources in the total computing resources for each first candidate node, and based on the proportion and whether the first candidate node has target image data cached, obtain the selection weight of the corresponding first candidate node. The target selection submodule is used to select the first candidate node whose selection weight meets the first preset condition as the target node.

[0085] Different from the above-mentioned embodiment, the first candidate nodes are all cached with image data, and the load information includes the currently remaining computing resources and the total computing resources, so that for each first candidate node, the proportion of the currently remaining computing resources in the total computing resources is obtained, and based on the proportion and whether the first candidate node has the target image data cached, the selection weight of the corresponding first candidate node is obtained, and then the first candidate node whose selection weight meets the first preset condition is selected as the target node. Therefore, since the first candidate nodes are all cached with image data, the first candidate nodes may have the target image data cached, or may not have the target image data cached, so that the target node is selected based on the proportion of the currently remaining computing resources in the total computing resources and whether the first candidate node has the target image data cached, which can help improve the robustness of the target node selection.

[0086] In some disclosed embodiments, the weight acquisition submodule includes a first acquisition unit, which is used to use the sum of the proportion and the preset weight as the selection weight corresponding to the first candidate node when the first candidate node has target image data cached, and the weight acquisition submodule includes a second acquisition unit, which is used to use the proportion as the selection weight corresponding to the first candidate node when the first candidate node does not have target image data cached.

[0087] Different from the aforementioned embodiment, when the first candidate node has the target image data cached, the sum of the proportion and the preset weight is used as the selection weight corresponding to the first candidate node; and when the first candidate node does not have the target image data cached, the proportion is used as the selection weight corresponding to the first candidate node, thereby improving the selection weight of the first candidate node that has the target image data cached, which in turn can help improve the robustness of the subsequent selection of the target node based on the selection weight.

[0088] In some disclosed embodiments, the first preset condition includes: selecting the largest weight; and / or the computing resources include at least one of the following: memory, processor.

[0089] Different from the aforementioned embodiment, the first preset condition is set to include: the selection weight is the largest, and the first candidate node corresponding to the largest selection weight can be used as the target node, which can help improve the robustness of selecting the target node; and the computing resources are set to include at least one of the memory and the processor, which can help improve the accuracy of the computing resource evaluation of the first candidate node, which can help improve the balance of the data cache node load.

[0090] In some disclosed embodiments, the data processing device 30 also includes a data deletion module for deleting part of the image data based on the request status of each image data in the data cache node when it is detected that the current remaining storage resources of the data cache node meet the second preset condition.

[0091] Different from the aforementioned embodiment, when it is detected that the current remaining storage resources of the data cache node meet the second preset condition, part of the image data is deleted based on the requested situation of each image data in the data cache node. Therefore, by deleting part of the image data based on the requested situation of each image data in the data cache node, it can help reduce the possibility of being unable to cache image data due to insufficient remaining storage resources of the data cache node, and at the same time reduce the possibility of affecting user browsing due to the deletion of frequently requested image data.

[0092] In some disclosed embodiments, the image data corresponds to a data identifier, which is used to distinguish different image data; and / or the image data includes medical image data; and / or the storage resources of the data cache node include at least one of the following: a solid state drive, a disk array.

[0093] Different from the aforementioned embodiment, a data identifier is set corresponding to the image data, and the data identifier is used to distinguish different image data, which can help improve the orderliness of image data storage and reading; and by setting the image data to include medical image data, it can help to achieve distributed storage and reading of medical image data; by setting the storage resources of the data cache node to include at least one of the following: a solid-state hard drive, a disk array, it can help to improve the reading speed of the image data.

[0094] See also Figure 4 , Figure 4 4 is a schematic diagram of a data processing device 40 of the present application. The data processing device 40 includes a memory 41 and a processor 42 coupled to each other, and the processor 42 is used to execute program instructions stored in the memory 41 to implement the steps of any of the above data processing method embodiments.

[0095] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps of any of the above data processing method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0096] The above solution can improve data processing efficiency.

[0097] See also Figure 5 , Figure 5 The schematic diagram of the framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 501 that can be executed by a processor, and the program instructions 501 are used to implement the steps of any of the above data processing method embodiments.

[0098] The above solution can improve data processing efficiency.

[0099] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0100] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A data processing method, characterized in that: include: receiving a data acquisition request, wherein the data acquisition request is used to request acquisition of target image data; In response to the data acquisition request, selecting at least one data cache node as a first candidate node, wherein the data cache node is used to cache image data, and the first candidate nodes all cache the image data; and For each of the first candidate nodes, obtaining a proportion of the currently remaining computing resources of the first candidate node in the total computing resources of the first candidate node, and obtaining a selection weight corresponding to the first candidate node based on the proportion and whether the first candidate node has the target image data cached; The first candidate node whose selection weight satisfies a first preset condition is used as a target node for responding to the data acquisition request, wherein the target node caches the target image data.

2. The method according to claim 1, characterized in that The method further comprises: In response to acquiring new image data, selecting at least one of the data cache nodes as a second candidate node based on a consistent hashing method, and caching the new image data to the second candidate node respectively; The step of selecting at least one data cache node as a first candidate node in response to the data acquisition request includes: In response to the data acquisition request, at least one data cache node is selected as the first candidate node based on the consistent hashing method; wherein the first candidate nodes all cache the target image data.

3. The method according to claim 2, characterized in that The selecting at least one of the data cache nodes as the second candidate node based on the consistent hashing method includes: Determine the data source of the new image data, and obtain a storage weight corresponding to the data source, wherein the storage weight represents the number of data cache nodes to be cached for the image data originating from the data source; Repeat the step of selecting a data cache node as a second candidate node based on a consistent hashing method until the number of second candidate nodes is selected.

4. The method according to claim 1, characterized in that: The obtaining a selection weight corresponding to the first candidate node based on the proportion and whether the first candidate node has the target image data cached therein comprises: In the case where the first candidate node has the target image data cached, taking the sum of the proportion and a preset weight as the selection weight corresponding to the first candidate node; In the case that the first candidate node does not cache the target image data, the proportion is used as a selection weight corresponding to the first candidate node.

5. The method according to any one of claims 1 to 4, characterized in that: The first preset condition includes: the selection weight is the largest; And / or, the computing resources include at least one of the following: memory, processor.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When it is detected that the current remaining storage resources of the data cache node meet the second preset condition, part of the image data is deleted based on the requested status of each image data in the data cache node.

7. The method according to any one of claims 1 to 4, characterized in that: The image data corresponds to a data identifier, and the data identifier is used to distinguish different image data; And / or, the image data includes medical image data; And / or, the storage resource of the data cache node includes at least one of the following: a solid state drive, a disk array.

8. A data processing device, characterized in that: include: A request receiving module, used for receiving a data acquisition request, wherein the data acquisition request is used for requesting to acquire target image data; A node screening module, configured to select at least one data cache node as a first candidate node in response to the data acquisition request; wherein the data cache node is used to cache image data, and the first candidate nodes all cache the image data; A target selection module is used to obtain, for each of the first candidate nodes, the proportion of the currently remaining computing resources of the first candidate node in the total computing resources of the first candidate node, and obtain the selection weight corresponding to the first candidate node based on the proportion and whether the first candidate node has cached the target image data, and use the first candidate node whose selection weight meets the first preset condition as the target node for responding to the data acquisition request, wherein the target node has cached the target image data.

9. A data processing device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A data scheduling system, characterized in that: include: A plurality of data caching nodes and a data processing device as claimed in claim 9; wherein the plurality of data caching nodes are respectively communicatively connected to the data processing device.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images