Data processing method, apparatus, device, and medium

Abstract

The embodiment of the application provides a data processing method, device, equipment and medium, which are applied to the technical field of data processing, the target external data source and the target model client are determined from a routing table according to a data processing notification by receiving the data processing notification; a pre-defined reconstruction interface is called to reconstruct an index model of the target model client; first data is acquired from the target external data source; the first data is converted to obtain second data corresponding to the index model; the second data is inserted into the index model by calling a pre-defined adding interface; a plurality of data sources and models can be compatible, and the conversion and transmission of data can be quickly completed; the unification of different data acquisition processes is realized through the pre-defined interface; the equipment can be integrated into a lightweight toolkit, and has the advantages of low hardware resource demand and low maintenance cost.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to data processing methods, apparatus, equipment and media. Background Technology

[0002] Medical image symptom prediction algorithms require acquiring medical image data from data sources for offline training and online prediction. This process presents several challenges: diverse data source interfaces, varied data acquisition processes, and data format compatibility between the data source and the model. Current solutions include the Flume log collection system and the Flink distributed processing framework. While Flume addresses the issue of diverse data source interfaces by allowing configuration to access different data sources, it cannot be directly used for data transmission from the data source to the model platform, and its data source extension interfaces are complex and difficult to maintain. Flink, on the other hand, suffers from large size and high deployment costs. Summary of the Invention

[0003] The purpose of this application is to at least partially solve one of the technical problems existing in the related art.

[0004] Therefore, one object of the embodiments of this application is to provide a data processing method, apparatus, device and medium.

[0005] To achieve the above objectives, a first aspect of this application provides a data processing method, comprising:

[0006] Receive a data processing notification, and determine the target external data source and target model client from the routing table based on the data processing notification;

[0007] The predefined reconstruction interface is invoked to reconstruct the index model of the target model client;

[0008] Obtain first data from the target external data source;

[0009] The first data is transformed to obtain the second data corresponding to the index model;

[0010] The second data is added to the index model by calling the predefined add interface.

[0011] In some embodiments, the step of performing data transformation on the first data to obtain second data corresponding to the index model includes:

[0012] Determine the target data converter from among multiple data converters that corresponds to the target external data source and the target model client;

[0013] The first data is transformed using the target data converter to obtain the second data corresponding to the index model.

[0014] In some embodiments, receiving a data processing notification and determining the target external data source and the target model client from the routing table based on the data processing notification includes:

[0015] Receive data processing notifications from external management modules, the data processing notifications including business identifiers and multi-model segmentation identifiers within the business;

[0016] Based on the service identifier and the multi-model segmentation identifier within the service, the target external data source and the target model client are determined from the routes pre-configured by the external management module in the routing table.

[0017] In some embodiments, obtaining the first data from the target external data source includes:

[0018] Determine the data source interface adapter corresponding to the target external data source;

[0019] Send a data acquisition command to the target external data source;

[0020] The system receives the first data from the target external data source through a data source interface adapter corresponding to the target external data source.

[0021] In some embodiments, receiving a data processing notification and determining the target external data source and the target model client from the routing table based on the data processing notification includes:

[0022] Receive data processing notifications from external data sources through the data source interface adapter corresponding to the external data source;

[0023] Based on the data processing notification, the external data source is identified as the target external data source, and the target model client is determined from the routing table.

[0024] In some embodiments, obtaining the first data from the target external data source includes:

[0025] The system receives the first data from the target external data source through a data source interface adapter corresponding to the external data source.

[0026] In some embodiments, the data processing method further includes:

[0027] Receive data analysis requests from external applications, the data analysis requests including business identifiers, business multi-model segmentation identifiers, target external data sources, and target model clients;

[0028] According to the data analysis request, the target data is obtained from the index model;

[0029] The target data is sent to the corresponding target model through the target model client, so that the target model can obtain analysis results based on the target data.

[0030] To achieve the above objectives, a second aspect of this application provides a data processing apparatus, comprising:

[0031] The target determination module is used to receive data processing notifications and determine the target external data source and target model client from the routing table based on the data processing notifications.

[0032] The model reconstruction module is used to call a predefined reconstruction interface to reconstruct the index model of the target model client;

[0033] The data acquisition module is used to acquire first data from the target external data source;

[0034] The data conversion module is used to convert the first data to obtain the second data corresponding to the index model;

[0035] The data addition module is used to call a predefined addition interface to add the second data to the index model.

[0036] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the data processing method described above.

[0037] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions for causing a computer to perform the data processing method described above.

[0038] The data processing method, apparatus, device, and medium disclosed in this application receive a data processing notification, determine the target external data source and target model client from the routing table according to the data processing notification; call a predefined reconstruction interface to reconstruct the index model of the target model client; obtain first data from the target external data source; perform data transformation on the first data to obtain second data corresponding to the index model; and call a predefined add interface to insert the second data into the index model. It is compatible with multiple data sources and models, and can quickly complete data transformation and transmission. It unifies different data acquisition processes through predefined interfaces and can be integrated into a lightweight SDK toolkit, offering advantages such as low hardware resource requirements and low maintenance costs. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0040] Figure 1 This is a schematic diagram illustrating the connection between the middleware provided in this application embodiment and the external model platform, external upstream business applications, external management modules, and external data sources;

[0041] Figure 2 This is a flowchart illustrating the steps of the data processing method provided in the embodiments of this application;

[0042] Figure 3 This is a sub-step diagram of step S100 when the middleware pulls data from the data warehouse;

[0043] Figure 4 This is a sub-step diagram of step S300 when the middleware pulls data from the data warehouse;

[0044] Figure 5 This is a sub-step diagram of step S400;

[0045] Figure 6 This is a sub-step diagram of step S100 when the data source control system pushes data to the middleware.

[0046] Figure 7 This is a sub-step diagram of step S300 when the data source control system pushes data to the middleware.

[0047] Figure 8 This is a flowchart of the online prediction process;

[0048] Figure 9This is a structural diagram of the data processing apparatus provided in the embodiments of this application;

[0049] Figure 10 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0053] In the description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0054] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0055] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Middleware: Middleware is a type of computer software that connects software components and applications, comprising a set of services. Middleware is a major category of foundational software, belonging to the realm of reusable software. It is an independent system software or service program that allows distributed applications to share resources across different technologies. Middleware resides above the client / server operating system, managing computing resources and network communication. This architecture is typically used to support and simplify complex distributed applications, including web servers, transaction monitors, and message queuing software. Middleware sits above the operating system, network, and database, below application software, its overall role being to provide a runtime and development environment for the application software above it, helping users to flexibly and efficiently develop and integrate complex applications.

[0059] To achieve the above objectives, embodiments of this application provide a data processing method, apparatus, device, and medium. This method involves receiving a data processing notification, determining the target external data source and target model client from a routing table based on the notification, reconstructing the index model of the target model client by calling a predefined reconstruction interface, obtaining first data from the target external data source, converting the first data to obtain second data corresponding to the index model, and inserting the second data into the index model by calling a predefined add interface. This method is compatible with multiple data sources and models, quickly completing data conversion and transmission. It can be integrated into a lightweight SDK toolkit, offering advantages such as low hardware resource requirements and low maintenance costs.

[0060] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0061] The embodiments of this application provide a data transmission method applied to middleware.

[0062] Reference Figure 1 The middleware includes a central control unit, multiple data source interface adapters, a protocol unit, multiple data converters, multiple model clients, and multiple online request adapters.

[0063] The central control unit is used to control the allocation and scheduling of middleware resources.

[0064] The data source interface adapter can interact with the corresponding data source, retrieve data from the data warehouse of the corresponding data source located outside the middleware, and interact with the protocol unit to call the various interfaces of the protocol unit.

[0065] The protocol unit predefines multiple interfaces, including a rebuild interface, an add interface, and a terminate interface, as well as predefine interfaces for adding, deleting, querying, and modifying data.

[0066] Data converters are used to convert data from one format to another. When there are N external data sources and M model clients, there are N*M data converters.

[0067] The model client is a client for the model located outside the middleware.

[0068] Different online request adapters correspond to different services.

[0069] In this embodiment, the middleware has the advantages of small size and high integration. Its minimum form can be an SDK package, which can quickly enable the system to have training and prediction capabilities compatible with various data sources and various models.

[0070] Furthermore, this middleware boasts strong scalability. Based on existing data sources and model clients, the middleware only requires routing configuration according to the existing data sources and model clients. When new data sources and model platforms need to be added, corresponding new routes are added to the middleware based on the new data sources and model platforms.

[0071] In addition, it can be integrated into a lightweight SDK toolkit, which has the advantages of low hardware resource requirements and low maintenance costs.

[0072] In addition, the middleware can interact with external management modules, external data sources, external model platforms, and external upstream business applications.

[0073] The external management module is a module located outside the middleware that can be used to manage and control the middleware. Administrators can configure and manage the middleware through the external management module.

[0074] External data sources are data sources set outside the middleware. The data source includes a data warehouse and a data source control system. The data warehouse is used to store data, and the data source control system is used to configure and control the data warehouse, as well as to interact with other modules.

[0075] The external model platform includes multiple models set outside the middleware. These models can be machine learning models, deep learning models, indexing platforms, Elasticsearch clusters, etc.

[0076] For example, a machine learning model is a model used for machine learning. Machine learning specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0077] An Elasticsearch (ES) cluster has multiple nodes, one of which is the master node. This master node is elected. The master-slave relationship is internal to the cluster. Externally, the ES cluster is logically a unified whole; it is decentralized, meaning there is no central node. ES clusters can shard a single index across multiple different distributed physical machines, thus achieving high availability and fault tolerance. Indexes in an ES cluster consist of multiple shards, and each shard can have multiple replicas. By dividing a single index into multiple shards, large indexes that cannot run on a single server can be handled. Each shard can have multiple replicas; by distributing replicas across multiple servers, query load capacity is improved.

[0078] External upstream business applications are those set up outside the middleware. Upstream business applications are those that can configure business processes with the middleware.

[0079] Reference Figure 2 The data processing method provided in the embodiments of this application includes, but is not limited to, the following steps:

[0080] Step S100: Receive data processing notification; based on the data processing notification, determine the target external data source and target model client from the routing table.

[0081] Step S200: Call the predefined reconstruction interface to reconstruct the index model of the target model client;

[0082] Step S300: Obtain the first data from the target external data source;

[0083] Step S400: Transform the first data to obtain the second data corresponding to the index model;

[0084] Step S500: Call the predefined add interface to add the second data to the index model.

[0085] The above data processing method is compatible with multiple data sources and models, and can quickly complete data conversion and transmission; it unifies different data acquisition processes through predefined interfaces; and it can be integrated into a lightweight SDK toolkit, which has the advantages of low hardware resource requirements and low maintenance costs.

[0086] In this embodiment, the middleware can obtain data from the data source in two ways: the first way is for the middleware to pull data from the data warehouse, and the second way is for the data source to control the system to push data to the middleware.

[0087] Among them, the middleware pulls data from the data warehouse, which can be used for offline training of models in external model platforms. The specific processing procedure of the data processing method is as follows.

[0088] Reference Figure 3 For step S100, a data processing notification is received. Based on the data processing notification, the target external data source and the target model client are determined from the routing table, including but not limited to the following steps:

[0089] Step S111: Receive a data processing notification from an external management module. The data processing notification includes a business identifier and a multi-model segmentation identifier within the business.

[0090] Step S112: Based on the business identifier and the multi-model segmentation identifier within the business, determine the target external data source and the target model client from the routes pre-configured by the external management module in the routing table.

[0091] Before step S111, the external management module assigns routes to the middleware. For example, one route might be: business identifier appId = appId1, multi-model segmentation identifier within the business subId = subId1, data source = first data source, model = Elasticsearch cluster. The middleware records the routes assigned by the external management module in its routing table.

[0092] In step S111, the external management module sends a data processing notification to the middleware. This notification includes a business identifier and a multi-model segmentation identifier within the business. For example, the notification might include the business identifier appId = appId1 and the multi-model segmentation identifier subId = subId1. The middleware's central control unit receives the data processing notification from the external management module.

[0093] For step S112, the middleware's central control unit, based on the business identifier appId=appId1 and the business multi-model segmentation identifier subId=subId1 in the data processing notification, looks up the route pre-configured by the external management module in the routing table, i.e., the route: business identifier appId=appId1, business multi-model segmentation identifier subId=subId1, data source=first data source, model=ES cluster; and then determines that the target external data source is the first data source and the target model client is the ES cluster.

[0094] In step S200, a data reconstruction instruction is sent to the middleware through the data source interface adapter corresponding to the first data source. Upon receiving the data reconstruction instruction from the data source interface adapter corresponding to the first data source, the middleware calls its predefined reconstruction interface to reconstruct the index model of the target model client corresponding to the Elasticsearch cluster.

[0095] Reference Figure 4 For step S300, the first data is obtained from the target external data source, including but not limited to the following steps:

[0096] Step S311: Determine the data source interface adapter corresponding to the target external data source;

[0097] Step S312: Send a data acquisition command to the target external data source;

[0098] Step S313: Receive the first data from the target external data source through the data source interface adapter corresponding to the target external data source.

[0099] For step S311, the data source interface adapter corresponding to the target external data source—the first data source—is determined to be the first data source interface adapter.

[0100] For step S312, a data acquisition command is sent to the first data source through the first data source interface adapter.

[0101] For step S313, the first data source receives a data acquisition instruction from the first data source interface adapter, retrieves the data mapped in the data acquisition instruction from the data warehouse, and sends it to the middleware; and receives medical image data from the first data source, i.e., the first data, through the first data source interface adapter.

[0102] Reference Figure 5 For step S400, the first data is transformed to obtain the second data corresponding to the index model, including but not limited to the following steps:

[0103] Step S410: Determine the target data converter from multiple data converters that corresponds to the target external data source and the target model client;

[0104] Step S420: The first data is converted using a target data converter to obtain the second data corresponding to the index model.

[0105] In this embodiment, specifically, the target external data source is the first data source, and the target model client is the model client corresponding to the Elasticsearch cluster. Therefore, the first data converter can be determined as the target data converter from among M*N data converters. This target data converter can convert the data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster. By using this target data converter to transform the first data from the first data source—that is, to convert the first data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster—the second data is obtained.

[0106] For step S500, specifically, the predefined add interface is called from the protocol module through the first data source interface adapter, and the second data after data format conversion is added to the index model of the model client corresponding to the Elasticsearch cluster via the route from the first data source to the model client corresponding to the Elasticsearch cluster.

[0107] Understandably, when new data is continuously transmitted from the first data source, the data converter first converts the data format of the new data into the data format of the index model of the corresponding model client of the Elasticsearch cluster. Then, the predefined add interface is continuously called from the protocol module through the first data source interface adapter. The new data, after data format conversion, is added to the index model of the corresponding model client of the Elasticsearch cluster via the route from the first data source to the corresponding model client of the Elasticsearch cluster.

[0108] The process continues until the first data source returns null data, indicating that no new data has been transmitted from the first data source. Then, the termination interface is called from the protocol module via the first data source interface adapter to terminate the data addition operation for the indexed model of the target model client.

[0109] The specific processing steps for the data processing method, which corresponds to the way the data source control system pushes data to the middleware, are as follows.

[0110] Reference Figure 6 For step S100, a data processing notification is received. Based on the data processing notification, the target external data source and the target model client are determined from the routing table, including but not limited to the following steps:

[0111] Step S121: Receive data processing notification from the external data source through the data source interface adapter corresponding to the external data source;

[0112] Step S122: Based on the data processing notification, the external data source is identified as the target external data source, and the target model client is determined from the routing table.

[0113] In step S121, the data source control system of the external data source and the corresponding data source interface adapter of the external data source maintain a continuous communication connection. For example, the data source interface adapter corresponding to the first data source subscribes to the notification information of the first data source. When the first data source updates its data, the data source control system of the first data source triggers a connection with the corresponding data source interface adapter of the first data source and sends a data processing notification to the data source interface adapter. The data source interface adapter corresponding to the first data source receives the data processing notification from the first data source.

[0114] For step S112, since the data processing notification is received through the data source interface adapter corresponding to the first data source, it can be determined that the first data source is the target external data source. Furthermore, based on the information that the target external data source is the first data source, the middleware's control unit can look up the route in the routing table: business identifier appId = appId1, business multi-model segmentation identifier subId = subId1, data source = first data source, model = Elasticsearch cluster; thus, it is determined that the target external data source is the first data source, and the target model client is the Elasticsearch cluster.

[0115] In step S200, a data reconstruction instruction is sent to the middleware through the data source interface adapter corresponding to the first data source. Upon receiving the data reconstruction instruction from the data source interface adapter corresponding to the first data source, the middleware calls its predefined reconstruction interface to reconstruct the index model of the target model client corresponding to the Elasticsearch cluster.

[0116] Reference Figure 7 For step S300, the first data is obtained from the target external data source, including but not limited to the following steps:

[0117] Step S321: Receive the first data from the target external data source through the data source interface adapter corresponding to the external data source.

[0118] Specifically, the first data source actively sends data to the middleware, and the middleware receives the medical image data from the first data source, i.e., the first data, through the data source interface adapter corresponding to the first data source.

[0119] Reference Figure 5For step S400, the first data is transformed to obtain the second data corresponding to the index model, including but not limited to the following steps:

[0120] Step S410: Determine the target data converter from multiple data converters that corresponds to the target external data source and the target model client;

[0121] Step S420: The first data is converted using a target data converter to obtain the second data corresponding to the index model.

[0122] In this embodiment, specifically, the target external data source is the first data source, and the target model client is the model client corresponding to the Elasticsearch cluster. Therefore, the first data converter can be determined as the target data converter from among M*N data converters. This target data converter can convert the data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster. By using this target data converter to transform the first data from the first data source—that is, to convert the first data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster—the second data is obtained.

[0123] For step S500, specifically, the predefined add interface is called from the protocol module through the first data source interface adapter, and the second data after data format conversion is added to the index model of the model client corresponding to the Elasticsearch cluster via the route from the first data source to the model client corresponding to the Elasticsearch cluster.

[0124] Understandably, when new data is continuously transmitted from the first data source, the data converter first converts the data format of the new data into the data format of the index model of the corresponding model client of the Elasticsearch cluster. Then, the predefined add interface is continuously called from the protocol module through the first data source interface adapter. The new data, after data format conversion, is added to the index model of the corresponding model client of the Elasticsearch cluster via the route from the first data source to the corresponding model client of the Elasticsearch cluster.

[0125] The process continues until the first data source returns null data, indicating that no new data has been transmitted from the first data source. Then, the termination interface is called from the protocol module via the first data source interface adapter to terminate the data addition operation for the indexed model of the target model client.

[0126] Reference Figure 8 Building upon this, the data processing method also includes a model-oriented online prediction process. This online prediction process includes, but is not limited to, the following steps:

[0127] Step S610: Receive a data analysis request from an external application. The data analysis request includes a business identifier, a multi-model segmentation identifier within the business, a target external data source, and a target model client.

[0128] Step S620: Obtain target data from the index model according to the data analysis request;

[0129] Step S630: Send the target data to the corresponding target model through the target model client so that the target model can obtain the analysis results based on the target data.

[0130] In step S610, the external upstream business application sends a data analysis request to the middleware to analyze medical images and obtain disease results. This data analysis request includes a business identifier, a multi-model segmentation identifier within the business, a target external data source, and a target model client. For example, the business identifier appID = appID1, the multi-model segmentation identifier subID = subID1, the target external data source is the first data source, and the target model client is the model client of the corresponding Elasticsearch cluster. The middleware receives the data analysis request from the external application.

[0131] For step S620, the central control unit calls an online request adapter corresponding to the business identifier appID1 to interpret the data analysis request, maps the data address to the address of the target data of the index model of the model client of the corresponding Elasticsearch cluster, and then calls the query interface predefined by the protocol unit to query the target data of the corresponding medical image from the index model of the model client of the corresponding Elasticsearch cluster according to the address.

[0132] In step S630, the target data is sent to the Elasticsearch cluster via the model client corresponding to the Elasticsearch cluster. The Elasticsearch cluster receives the target data from the model client and performs online analysis based on the target data of the corresponding medical image. The analysis results are obtained by performing disease analysis on the medical image, such as image recognition on a lung CT image to obtain analysis results characterizing whether a tumor exists in the lung and the type of tumor.

[0133] An embodiment of this application provides a data processing apparatus that uses middleware.

[0134] Reference Figure 9 The data processing device includes a target determination module 110, a model reconstruction module 120, a data acquisition module 130, a data conversion module 140, and a data addition module 150.

[0135] The target determination module 110 is used to execute step S100, which is to receive a data processing notification and determine the target external data source and the target model client from the routing table according to the data processing notification; the model reconstruction module 120 is used to execute step S200, which is to call a predefined reconstruction interface to reconstruct the index model of the target model client; the data acquisition module 130 is used to execute step S300, which is to acquire the first data from the target external data source; the data conversion module 140 is used to execute step S400, which is to convert the first data to obtain the second data corresponding to the index model; and the data addition module 150 is used to execute step S500, which is to call a predefined addition interface to add the second data to the index model.

[0136] In this embodiment, the data processing device can obtain data from the data source in two ways using middleware: the first way is for the middleware to pull data from the data warehouse, and the second way is for the data source control system to push data to the middleware.

[0137] The specific data processing procedure for the middleware that pulls data from the data warehouse is as follows.

[0138] For the target determination module 110, it receives a data processing notification and, based on the notification, determines the target external data source and the target model client from the routing table. Specifically, it receives a data processing notification from the external management module, which includes a business identifier and a multi-model segmentation identifier within the business. Based on the business identifier and the multi-model segmentation identifier within the business, it determines the target external data source and the target model client from the routes pre-configured by the external management module in the routing table.

[0139] Furthermore, the external management module assigns routes to the middleware. For example, one route might be: business identifier appId = appId1, multi-model segmentation identifier within the business subId = subId1, data source = first data source, model = Elasticsearch cluster. The middleware records the routes assigned by the external management module in its routing table.

[0140] The external management module sends a data processing notification to the middleware. This notification includes a business identifier and a multi-model segmentation identifier within the business. For example, the notification might include the business identifier appId = appId1 and the multi-model segmentation identifier subId = subId1. The middleware's central control unit receives the data processing notification from the external management module.

[0141] The middleware's central control unit retrieves the pre-configured route from the routing table based on the business identifier appId=appId1 and the business multi-model segmentation identifier subId=subId1 in the data processing notification. The route is: business identifier appId=appId1, business multi-model segmentation identifier subId=subId1, data source=first data source, model=ES cluster. It then determines that the target external data source is the first data source and the target model client is the ES cluster.

[0142] For model reconstruction module 120, a data reconstruction command is sent to the middleware through the data source interface adapter corresponding to the first data source. Upon receiving the data reconstruction command from the data source interface adapter corresponding to the first data source, the middleware calls its predefined reconstruction interface to reconstruct the index model of the target model client corresponding to the Elasticsearch cluster.

[0143] For the data acquisition module 130, it acquires first data from the target external data source. Specifically, it determines the data source interface adapter corresponding to the target external data source; sends a data acquisition instruction to the target external data source; and receives the first data from the target external data source through the data source interface adapter corresponding to the target external data source.

[0144] Furthermore, the data source interface adapter corresponding to the target external data source—the first data source—is identified as the first data source interface adapter. A data retrieval command is sent to the first data source through the first data source interface adapter. Upon receiving the data retrieval command from the first data source interface adapter, the first data source retrieves the data mapped in the data retrieval command from the data warehouse and sends it to the middleware; the medical image data from the first data source, i.e., the first data, is received through the first data source interface adapter.

[0145] For the data conversion module 140, the first data is converted to obtain the second data corresponding to the index model. Specifically, a target data converter corresponding to the target external data source and the target model client is determined from multiple data converters; the first data is converted using the target data converter to obtain the second data corresponding to the index model.

[0146] In this embodiment, specifically, the target external data source is the first data source, and the target model client is the model client corresponding to the Elasticsearch cluster. Therefore, the first data converter can be determined as the target data converter from among M*N data converters. This target data converter can convert the data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster. By using this target data converter to transform the first data from the first data source—that is, to convert the first data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster—the second data is obtained.

[0147] For the data addition module 150, the predefined addition interface is called from the protocol module through the first data source interface adapter. The second data, after data format conversion, is added to the index model of the model client corresponding to the Elasticsearch cluster via the routing from the first data source to the model client corresponding to the Elasticsearch cluster.

[0148] Understandably, when new data is continuously transmitted from the first data source, the data converter first converts the data format of the new data into the data format of the index model of the corresponding model client of the Elasticsearch cluster. Then, the predefined add interface is continuously called from the protocol module through the first data source interface adapter. The new data, after data format conversion, is added to the index model of the corresponding model client of the Elasticsearch cluster via the route from the first data source to the corresponding model client of the Elasticsearch cluster.

[0149] The process continues until the first data source returns null data, indicating that no new data has been transmitted from the first data source. Then, the termination interface is called from the protocol module via the first data source interface adapter to terminate the data addition operation for the indexed model of the target model client.

[0150] The specific processing steps for the data processing method, which corresponds to the way the data source control system pushes data to the middleware, are as follows.

[0151] For the target determination module 110, it receives a data processing notification and, based on the notification, determines the target external data source and the target model client from the routing table. Specifically, it receives the data processing notification from the external data source through the data source interface adapter corresponding to the external data source; based on the notification, it determines the external data source as the target external data source and determines the target model client from the routing table.

[0152] Furthermore, the data source control system of the external data source and the corresponding data source interface adapter maintain a continuous communication connection. For example, the data source interface adapter corresponding to the first data source subscribes to the notification information of the first data source. When the first data source updates its data, the data source control system of the first data source triggers a connection with the corresponding data source interface adapter and sends a data processing notification to that data source interface adapter. The data source interface adapter corresponding to the first data source receives the data processing notification from the first data source. Since the data processing notification is received through the data source interface adapter corresponding to the first data source, the first data source can be identified as the target external data source. Furthermore, based on the information that the target external data source is the first data source, the middleware's central control unit can retrieve the route from the routing table: business identifier appId = appId1, business multi-model segmentation identifier subId = subId1, data source = first data source, model = Elasticsearch cluster; thus, it determines that the target external data source is the first data source and the target model client is the Elasticsearch cluster.

[0153] For model reconstruction module 120, a data reconstruction command is sent to the middleware through the data source interface adapter corresponding to the first data source. Upon receiving the data reconstruction command from the data source interface adapter corresponding to the first data source, the middleware calls its predefined reconstruction interface to reconstruct the index model of the target model client corresponding to the Elasticsearch cluster.

[0154] For the data acquisition module 130, first data is acquired from the target external data source. Specifically, the first data is received from the target external data source through the data source interface adapter corresponding to the external data source. Further, the first data source actively sends data to the middleware, and the middleware receives the medical image data, i.e., the first data, from the first data source through the data source interface adapter corresponding to the first data source.

[0155] For the data conversion module 140, the first data is converted to obtain the second data corresponding to the index model. Specifically, a target data converter corresponding to the target external data source and the target model client is determined from multiple data converters; the first data is converted using the target data converter to obtain the second data corresponding to the index model.

[0156] Furthermore, if the target external data source is the first data source and the target model client is the model client corresponding to the Elasticsearch cluster, then the first data converter can be determined as the target data converter from among M*N data converters. This target data converter can convert the data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster. By using this target data converter to transform the first data from the first data source, that is, to convert the first data from the data format corresponding to the first data source to the data format of the index model of the model client corresponding to the Elasticsearch cluster, the second data is obtained.

[0157] Specifically, for the data addition module 150, the predefined addition interface is called from the protocol module through the first data source interface adapter. The second data, after data format conversion, is added to the index model of the model client corresponding to the Elasticsearch cluster via the routing from the first data source to the model client corresponding to the Elasticsearch cluster.

[0158] Understandably, when new data is continuously transmitted from the first data source, the data converter first converts the data format of the new data into the data format of the index model of the corresponding model client of the Elasticsearch cluster. Then, the predefined add interface is continuously called from the protocol module through the first data source interface adapter. The new data, after data format conversion, is added to the index model of the corresponding model client of the Elasticsearch cluster via the route from the first data source to the corresponding model client of the Elasticsearch cluster.

[0159] The process continues until the first data source returns null data, indicating that no new data has been transmitted from the first data source. Then, the termination interface is called from the protocol module via the first data source interface adapter to terminate the data addition operation for the indexed model of the target model client.

[0160] Building upon this, the data processing device also executes a model-oriented online prediction process. This online prediction process is as follows: It receives data analysis requests from external applications, including a business identifier, a multi-model segmentation identifier within the business, a target external data source, and a target model client; based on the data analysis request, it retrieves target data from the index model; and through the target model client, it sends the target data to the corresponding target model, enabling the target model to obtain analysis results based on the target data.

[0161] Furthermore, external upstream business applications send data analysis requests to the middleware. These requests include a business identifier, a multi-model segmentation identifier within the business, a target external data source, and a target model client. For example, the business identifier appID = appID1, the multi-model segmentation identifier subID = subID1, the target external data source is the first data source, and the target model client is the model client of the corresponding Elasticsearch cluster. The middleware receives these data analysis requests from external applications.

[0162] The central control unit calls an online request adapter corresponding to the business identifier appID1 to interpret the data analysis request, maps the data address to the address of the target data in the index model of the corresponding Elasticsearch cluster model client, and then calls the query interface predefined by the protocol unit to retrieve the target data from the index model of the corresponding Elasticsearch cluster model client according to the address.

[0163] The target data is sent to the Elasticsearch (ES) cluster via the model client corresponding to the ES cluster. The ES cluster receives the target data from the model client and performs online analysis based on the target data of the corresponding medical images. The analysis results are obtained by performing disease analysis on the medical images. For example, image recognition is performed on lung CT images to obtain analysis results indicating whether there is a tumor in the lungs and the type of tumor.

[0164] In this embodiment, the target determination module 110 receives a data processing notification and determines the target external data source and target model client from the routing table based on the notification. The model reconstruction module 120 reconstructs the index model of the target model client by calling a predefined reconstruction interface. The data acquisition module 130 acquires first data from the target external data source. The data conversion module 140 converts the first data to obtain second data corresponding to the index model. The data addition module 150 inserts the second data into the index model by calling a predefined addition interface. This method is compatible with multiple data sources and models, and can quickly complete data conversion and transmission. It unifies different data acquisition processes through predefined interfaces and can be integrated into a lightweight SDK toolkit, which has the advantages of low hardware resource requirements and low maintenance costs.

[0165] Embodiments of this application also provide an electronic device. (Refer to...) Figure 10 The electronic device includes a memory 220, a processor 210, a program stored in the memory 220 and executable on the processor 210, and a data bus 230 for establishing communication between the processor 210 and the memory 220. When the program is executed by the processor 210, it implements the data processing method described above.

[0166] The aforementioned electronic device receives data processing notifications and, based on these notifications, determines the target external data source and the target model client from the routing table; it then calls a predefined reconstruction interface to reconstruct the index model of the target model client; it obtains first data from the target external data source; it performs data transformation on the first data to obtain second data corresponding to the index model; and it calls a predefined add interface to insert the second data into the index model. It is compatible with multiple data sources and models, quickly completing data transformation and transmission; it unifies different data acquisition processes through predefined interfaces; and it can be integrated into a lightweight toolkit, offering advantages such as low hardware resource requirements and low maintenance costs.

[0167] This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0168] In general, for the hardware structure of electronic devices, the processor 210 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solutions provided in the embodiments of this application.

[0169] The memory 220 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 210 to execute the data monitoring method of the embodiments of this application.

[0170] Input / output interfaces are used to implement information input and output.

[0171] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0172] Bus 230 transmits information between various components of the device (e.g., processor 210, memory 220, input / output interface, and communication interface). Processor 210, memory 220, input / output interface, and communication interface are interconnected within the device via bus 230.

[0173] An embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the data processing method described above.

[0174] The aforementioned computer-readable storage medium receives data processing notifications, determines the target external data source and target model client from the routing table based on the data processing notifications; calls a predefined reconstruction interface to reconstruct the index model of the target model client; obtains first data from the target external data source; performs data transformation on the first data to obtain second data corresponding to the index model; and calls a predefined add interface to insert the second data into the index model. It is compatible with multiple data sources and models, and can quickly complete data transformation and transmission. It unifies different data acquisition processes through predefined interfaces. It can be integrated into a lightweight toolkit, with the advantages of low hardware resource requirements and low maintenance costs.

[0175] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0176] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0177] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0181] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0182] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A data processing method, characterized in that, Applied to middleware, the method includes: Receive a data processing notification, and determine the target external data source and target model client from the routing table based on the data processing notification; The predefined reconstruction interface is invoked to reconstruct the index model of the target model client; Obtain first data from the target external data source; The first data is transformed to obtain the second data corresponding to the index model; The second data is added to the index model by calling the predefined add interface; Receive data analysis requests from external applications, the data analysis requests including business identifiers, business multi-model segmentation identifiers, target external data sources, and target model clients; According to the data analysis request, the target data is obtained from the index model; The target data is sent to the corresponding target model through the target model client, so that the target model can obtain analysis results based on the target data.

2. The data processing method according to claim 1, characterized in that, The step of transforming the first data to obtain the second data corresponding to the index model includes: Determine the target data converter from among multiple data converters that corresponds to the target external data source and the target model client; The first data is transformed using the target data converter to obtain the second data corresponding to the index model.

3. The data processing method according to claim 1 or 2, characterized in that, The receiving of the data processing notification, and determining the target external data source and target model client from the routing table based on the data processing notification, includes: Receive data processing notifications from external management modules, the data processing notifications including business identifiers and multi-model segmentation identifiers within the business; Based on the service identifier and the multi-model segmentation identifier within the service, the target external data source and the target model client are determined from the routes pre-configured by the external management module in the routing table.

4. The data processing method according to claim 3, characterized in that, The step of obtaining the first data from the target external data source includes: Determine the data source interface adapter corresponding to the target external data source; Send a data acquisition command to the target external data source; The system receives the first data from the target external data source through a data source interface adapter corresponding to the target external data source.

5. The data processing method according to claim 1 or 2, characterized in that, The receiving of the data processing notification, and determining the target external data source and target model client from the routing table based on the data processing notification, includes: Receive data processing notifications from external data sources through the data source interface adapter corresponding to the external data source; Based on the data processing notification, the external data source is identified as the target external data source, and the target model client is determined from the routing table.

6. The data processing method according to claim 5, characterized in that, The step of obtaining the first data from the target external data source includes: The system receives the first data from the target external data source through a data source interface adapter corresponding to the external data source.

7. A data processing apparatus, characterized in that, Applied to middleware, the device includes: The target determination module is used to receive data processing notifications and determine the target external data source and target model client from the routing table based on the data processing notifications. The model reconstruction module is used to call a predefined reconstruction interface to reconstruct the index model of the target model client; The data acquisition module is used to acquire first data from the target external data source; The data conversion module is used to convert the first data to obtain the second data corresponding to the index model; The data addition module is used to call a predefined addition interface to add the second data to the index model; The data analysis module is used to receive data analysis requests from external applications. The data analysis request includes a business identifier, a multi-model segmentation identifier within the business, a target external data source, and a target model client. Based on the data analysis request, the module obtains target data from the index model. The module then sends the target data to the corresponding target model through the target model client, so that the target model obtains analysis results based on the target data.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the data processing method as described in any one of claims 1 to 6.

Images