Data asset management method and system based on micro-service architecture and application

Abstract

The invention provides a data asset management method and system based on a micro-service architecture and application, and the method comprises the steps: carrying out the visual modeling of a user demand through a business modeling tool, and recognizing and sorting a core business process; performing product scheme and prototype design by using a design tool so as to clarify a system function module and a user interaction interface, and designing and reviewing according to product requirements; the method comprises the following steps of: dividing a business field into a plurality of independent micro-service modules by adopting a micro-service architecture based on field-driven design according to business requirements; determining an interface function and a protocol of the micro-service, and designing a data model and a data interaction mode of the micro-service interface; and deploying the completed micro-service to a Kubernetes cluster in a container mirror image form, and performing service management and communication by using a micro-service registration center and a gateway. According to the invention, an enterprise whole-process management system is constructed, enterprise data asset entry is promoted, and data value metering, data application and product circulation are realized.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a data asset management method, system and application based on a microservice architecture. Background Art

[0002] Provinces and cities across the country have actively responded to the central government's call, and have successively formulated and issued a series of local policies and regulations such as data regulations, management methods, implementation rules, and implementation plans, and made detailed requirements for the management, openness, authorized operation and application of public data. However, data assets, as emerging assets in the digital transformation of the economy and society, still face many difficulties and shortcomings, which are mainly reflected in the obvious shortage of high-quality data assets, unclear paths for the compliant use of data assets, insufficient value-added empowerment of data asset applications, and difficulties in the circulation and trading of data assets.

[0003] The data element industry as a whole is in the exploratory stage, and there is currently no mature data asset management and operation platform. Summary of the invention

[0004] In view of this, the present application aims to propose a data asset management method, system and application based on microservice architecture to solve the above-mentioned deficiencies.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows: In a first aspect, the present application provides a data asset management method based on a microservice architecture, comprising: Use business modeling tools to visually model user needs, identify and organize core business processes, including data element registration process, data asset maintenance process, data asset evaluation process, and data asset entry process; Use design tools to design product solutions and prototypes to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements; Adopt a microservice architecture based on domain-driven design according to business needs, and divide the business domain into multiple independent microservice modules; Determine the interface functions and protocols of microservices, and design the data model and data interaction method of the microservice interface; Deploy the completed microservices to the Kubernetes cluster in the form of container images, and use the microservice registry and gateway for service management and communication.

[0006] In the second aspect, based on the same inventive concept, the present application also provides a data asset management system based on a microservice architecture, including: An identification module is configured to use a business modeling tool to perform visual modeling of user requirements, identify and organize core business processes, wherein the business processes include a data element registration process, a data asset maintenance process, a data asset evaluation process, and a data asset entry process; The design module is configured to use design tools to conduct product solutions and prototype design to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements; The microservice architecture module is configured to adopt a microservice architecture based on domain-driven design according to business needs, dividing the business domain into multiple independent microservice modules; The interface building module is configured to determine the interface functions and protocols of the microservices and design the data model and data interaction mode of the microservice interface; The deployment module is configured to deploy the completed microservices into the Kubernetes cluster in the form of container images, and use the microservice registry and gateway for service management and communication.

[0007] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0008] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0009] Compared with the prior art, the data asset management method, system and application based on microservice architecture described in this application have the following beneficial effects: The data asset management method, system and application based on microservice architecture described in this application realize the full system management and operation capabilities of enterprise data asset inventory, management, entry, valuation and circulation, and realize the rapid implementation and launch of data asset management and operation platform through different deployment methods to provide data-related services; at the same time, by building a full-process management system for data asset compliance management, asset identification, cost management, and usage monitoring, it promotes the entry of enterprise data assets into the table, realizes the measurable value of data, the applicability of data, and the circulation of products. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1This is a flow chart of a data asset management method based on a microservice architecture described in an embodiment of the present application; Figure 2 This is a data element registration flow chart described in an embodiment of the present application; Figure 3 This is a data asset maintenance flow chart described in the embodiment of this application; Figure 4 This is a data asset evaluation flow chart described in an embodiment of the present application; Figure 5 This is a flowchart of data asset entry into a table as described in an embodiment of the present application; Figure 6 This is a schematic diagram of the microservice architecture described in an embodiment of the present application; Figure 7 This is a schematic diagram of the structure of a data asset management system based on a microservice architecture according to an embodiment of the present application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device described in the embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0014] See also Figure 1 As shown, this embodiment provides a data asset management method based on a microservice architecture, which specifically includes the following steps: Step S101: Use business modeling tools to perform visual modeling of user needs, identify and organize core business processes, where the business processes include data element registration process, data asset maintenance process, data asset evaluation process and data asset entry process.

[0015] Specifically, in this embodiment, Figure 2 As shown, the data element registration process includes: Identify the data elements of the registered data assets to register the data elements, review and approve the registration of the data elements, and process the data elements after approval, including data element cancellation, data element inventory and data element change; The processed data elements are subject to approval for each project.

[0016] The data element registration process provides data asset registration management functions, supports filling in basic data asset information, including data asset name, source, acquisition method, industry, coverage area, data scale, time span, applicable scenarios, etc., and associates data resources to clarify data asset call information. It also provides data asset registration approval functions, supports the approval of data asset registration applications, and generates a unique data asset certificate after approval.

[0017] like Figure 3 As shown in the figure, the data asset maintenance process includes: Carry out data asset identification for registered data resources, and approve the registration of identified data assets. After approval, carry out asset processing, including asset cancellation, asset verification and asset change; Conduct multi-project approval for processed assets.

[0018] The data asset maintenance process provides a data asset change application function, which supports changes and modifications to registered data asset information, and an approval function, which supports the approval of data asset change applications. After approval, the change content is updated to the registered data asset information.

[0019] like Figure 4 As shown in the figure, the data asset assessment process includes: Conduct data asset cost assessment, data asset compliance assessment, data asset quality assessment, data asset security assessment and data asset value assessment respectively; Evaluate and approve each evaluation project and generate a corresponding evaluation report. For the data asset cost evaluation that has passed the evaluation, generate total cost data; Managed separately according to each assessment report and total cost data.

[0020] Further, cost assessment: Provides cost accounting function, supports adding direct and indirect costs of data assets, and associates registered data assets, including acquisition cost, processing cost, storage cost, security cost, maintenance and update cost. Provides cost management function, supports accounting of data asset costs, allocates indirect costs according to expected income or working hours, and combines the direct cost of data assets to obtain the total cost of data assets; Compliance assessment: Provides compliance indicator management functions and supports the configuration of compliance reports for data assets, including subject compliance, data content compliance, data source compliance, and data maintenance compliance throughout the entire life cycle. Provides compliance management functions and supports the assessment of data asset compliance, generates compliance reports through compliance commitments, and can link to third-party compliance assessment reports; Quality assessment: Provides quality indicator management functions and supports the configuration of the indicator system for data quality assessment, including accuracy indicators, completeness indicators, timeliness indicators, consistency indicators, accessibility indicators, and standardization indicators. Provides quality management functions and supports data quality assessment according to data quality assessment indicators, generates quality scoring coefficients, and can be associated with third-party quality assessment reports; Security assessment: Provides security index management functions and supports the configuration of the index system for data security assessment, including data collection security, data transmission security, data storage security, data processing security, data exchange security, data destruction security, general security, etc. Provides security management functions and supports data security assessment according to data security assessment indicators, generates security scoring coefficients, and can be associated with third-party security assessment reports; Value assessment: Provides value parameter configuration function, supports configuration of data asset value parameters according to multi-factor cost correction model, and configures industry average return on investment. Provides value management function, supports evaluation of data asset value according to multi-factor cost correction model, substitutes data asset cost, quality, security, and value correlation coefficient for calculation, and obtains estimated value of data assets, which can be linked to third-party value assessment reports.

[0021] like Figure 5 As shown in the figure, the process of entering data assets into the table includes: Create a data asset entry plan according to the cycle or create entry tasks according to the items to clarify the asset information to be entered into the table. After the entry is completed, update the task status and entry result information.

[0022] The data asset entry process provides entry task management functions, supports the creation of data asset entry plans by period or entry tasks by matter, clarifies the scope, period, subject and other information of the assets to be entered into the table, and after the entry is completed, the task status and entry result information can be updated.

[0023] Step S102: Use design tools to design product solutions and prototypes to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements.

[0024] Specifically, in this embodiment, after clearly defining the business process, use tools such as Moknife to start product solution planning and prototype construction, and organize a rigorous product demand design review. Use tools such as class diagrams and ER diagrams to accurately identify entities in the business field and their internal relationships, build entity models, and rely on database management systems (DBMS) to implement data storage and control.

[0025] Step S103: Adopt a microservice architecture based on domain-driven design according to business needs, and divide the business domain into multiple independent microservice modules.

[0026] Specifically, in this embodiment, a microservice architecture based on domain-driven design (DDD) is adopted according to business needs, and the business domain is divided into multiple independent microservice modules, including data registration service, data evaluation service, data transaction service, privacy computing service, etc.

[0027] Among them, Figure 6 As shown, the microserver architecture includes: Data persistence layer: This application uses MySQL as the data persistence layer, which can be deployed in master and backup or cluster according to actual conditions to ensure high availability of the system. Redis is used as a cache database to store hot data and improve the system impact speed; Service provision layer: It is divided into basic service layer and business service layer, both of which adopt the microservice Java Spring Cloud architecture. The backend service uses Java as the development language and has the ability to compile and run across platforms. The registration center uses Nacos to implement service discovery, registration, and configuration file management. Network layer: It is divided into traffic gateway and business gateway. The traffic gateway uses nginx proxy, and the business gateway uses gateway to perform request distribution, fuse, current limiting and other functions to ensure system availability; Public capability layer: Provides log services to facilitate users to view system logs, provides full-link tracking and monitoring services, and promptly discovers system problems and provides feedback. Object storage services are used to save user files, and provide message queue services and system message queue management functions.

[0028] This embodiment uses the Java Spring Cloud framework to design a microservice architecture, uses MySQL as a database to store business data, uses Redis as a cache database to store hot data, and uses minio as a file server to store file data.

[0029] Step S104: determine the interface function and protocol of the microservice, and design the data model and data interaction method of the microservice interface.

[0030] Specifically, in this embodiment, the interface functions and protocols are determined, and the interface is defined using the RESTful API protocol. At the same time, a data collaboration interface based on federated learning is provided to support cross-departmental data modeling tasks. The data model and data interaction mode of the microservice interface are designed, for example, using JSON, XML and other formats to implement data interaction. The code implementation of the microservice interface is written, for example, the microservice interface code is written using programming languages ​​such as Java.

[0031] This embodiment makes full use of microservice Java Spring Cloud, distributed, and cloud computing technologies to improve the high availability of the data asset management platform, fully considers the application of new technologies in various scenarios, uses mature and stable Internet protocols to achieve standardized data interaction, and uses the RESTful API protocol that complies with unified specifications to implement data models and user input data interaction to complete business functions.

[0032] Step S105: Deploy the completed microservice to the Kubernetes cluster in the form of a container image, and use the microservice registry and gateway for service management and communication.

[0033] Specifically, in this embodiment, containerization technology (such as Docker, Kubernetes, etc.) is used to implement the deployment of microservices. A microservice registration center and a microservice gateway are used to implement the management and communication of microservices. Load balancing and fault tolerance processing of microservices are performed to achieve microservice traffic distribution and ensure the availability and reliability of microservices.

[0034] This embodiment uses containerization-related technologies (such as Docker, Kubernetes, etc.) to simplify the deployment process and optimize the operation and maintenance implementation plan. It can achieve rapid horizontal expansion through elastic scaling of backend services, and achieve vertical expansion through JVM parameter adjustment and container operation parameter adjustment to ensure high availability of the system.

[0035] The method described in this embodiment realizes the full-system management and operation capabilities of enterprise data asset inventory, management, entry, valuation, and circulation, and realizes the rapid implementation and launch of the data asset management and operation platform through different deployment methods to provide data-related services; at the same time, by building a full-process management system for data asset compliance management, asset identification, cost management, and usage monitoring, it promotes the entry of enterprise data assets into the table, realizes the measurable value of data, the applicability of data, and the circulation of products.

[0036] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0037] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the embodiment of the present application also provides a data asset management system based on a microservice architecture.

[0038] like Figure 7 As shown, the data asset management system based on the microservice architecture includes: The identification module 11 is configured to use a business modeling tool to perform visual modeling of user requirements, identify and organize core business processes, wherein the business processes include a data element registration process, a data asset maintenance process, a data asset evaluation process, and a data asset entry process; The design module 12 is configured to use design tools to perform product solutions and prototype design to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements; The microservice architecture module 13 is configured to adopt a microservice architecture based on domain-driven design according to business requirements, and divide the business domain into multiple independent microservice modules; An interface building module 14 is configured to determine the interface functions and protocols of the microservices and provide a data collaboration interface based on federated learning; The deployment module 15 is configured to deploy the completed microservices into the Kubernetes cluster in the form of container images, and use the microservice registry and gateway for service management and communication.

[0039] For the convenience of description, the above system is described by dividing the functions into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0040] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0041] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0042] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0043] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0044] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0045] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0046] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0047] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0048] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0049] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0050] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0051] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0052] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0053] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0054] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device may be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0055] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0056] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A data asset management method based on microservice architecture, characterized in that: include: Use business modeling tools to visually model user needs, identify and organize core business processes, including data element registration process, data asset maintenance process, data asset evaluation process, and data asset entry process; Use design tools to design product solutions and prototypes to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements; Adopt a microservice architecture based on domain-driven design according to business needs, and divide the business domain into multiple independent microservice modules; Determine the interface functions and protocols of microservices, and design the data model and data interaction method of the microservice interface; Deploy the completed microservices to the Kubernetes cluster in the form of container images, and use the microservice registry and gateway for service management and communication.

2. The method according to claim 1, characterized in that The data element registration process includes: Identify the data elements of the registered data assets to register the data elements, review and approve the registration of the data elements, and process the data elements after approval, including data element cancellation, data element inventory and data element change; The processed data elements are subject to approval for each project.

3. The method according to claim 1, characterized in that The data asset maintenance process includes: Carry out data asset identification for registered data resources, and approve the registration of identified data assets. After approval, carry out asset processing, including asset cancellation, asset verification and asset change; Conduct multi-project approval for processed assets.

4. The method according to claim 1, characterized in that: The data asset assessment process includes: Conduct data asset cost assessment, data asset compliance assessment, data asset quality assessment, data asset security assessment and data asset value assessment respectively; Evaluate and approve each evaluation project and generate a corresponding evaluation report. For the data asset cost evaluation that has passed the evaluation, generate total cost data; Managed separately according to each assessment report and total cost data.

5. The method according to claim 1, characterized in that The data asset entry process includes: Create a data asset entry plan according to the cycle or create entry tasks according to the items to clarify the asset information to be entered into the table. After the entry is completed, update the task status and entry result information.

6. The method according to claim 1, characterized in that: The microservice architecture includes a data persistence layer, a service provision layer, a network layer and a public capability layer, wherein the service provision layer includes a basic service layer and a business service layer, and the network layer is divided into a traffic gateway and a business gateway.

7. The method according to claim 1, characterized in that Also includes: Distribute microservice traffic through load balancing tools or container orchestration tools.

8. A data asset management system based on microservice architecture, characterized in that: include: An identification module is configured to use a business modeling tool to perform visual modeling of user requirements, identify and organize core business processes, wherein the business processes include a data element registration process, a data asset maintenance process, a data asset evaluation process, and a data asset entry process; The design module is configured to use design tools to conduct product solutions and prototype design to clarify system functional modules and user interaction interfaces, and conduct design reviews based on product requirements; The microservice architecture module is configured to adopt a microservice architecture based on domain-driven design according to business needs, dividing the business domain into multiple independent microservice modules; The interface building module is configured to determine the interface functions and protocols of the microservices and design the data model and data interaction mode of the microservice interface; The deployment module is configured to deploy the completed microservices into the Kubernetes cluster in the form of container images, and use the microservice registry and gateway for service management and communication.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: in, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images