Data processing method, device, storage medium and electronic device

Abstract

The present invention discloses a data processing method, device, storage medium and electronic device. The method comprises: reading data from each of a plurality of databases; writing the read data into a distributed file system; upon receiving a write request to write data to any one of the plurality of databases, writing the data requested to be written by the write request into the distributed file system; registering a target service for processing data in the distributed file system; upon receiving a request to process data in the distributed file system, calling the target service to process the data in the distributed file system. The present invention solves the technical problem that data barriers exist between units and data cannot be accessed mutually.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data processing method, device, storage medium and electronic equipment. Background Art

[0002] In the prior art, when data is involved from different organizations or units, since the databases storing the data of each organization or unit are not interoperable, if data is to be read and written across units, the account used by unit A to access the database of unit A needs to be assigned to the employees of unit B so that the employees of unit B can use the data of unit A.

[0003] In other words, the methods in the prior art have the problem of data barriers between different units and inability to communicate and access each other. Summary of the Invention

[0004] The embodiments of the present invention provide a data processing method, device, storage medium, and electronic device to at least solve the technical problem that data barriers exist between units and data cannot be accessed interoperably.

[0005] According to one aspect of an embodiment of the present invention, a data processing method is provided, comprising: reading data from each of a plurality of databases; writing the read data into a distributed file system; upon receiving a write request to write data to any one of the plurality of databases, writing the data requested by the write request into the distributed file system; registering a target service for processing data in the distributed file system; and upon receiving a request to process data in the distributed file system, calling the target service to process the data in the distributed file system.

[0006] According to another aspect of an embodiment of the present invention, a data processing device is provided, including: a reading module for reading data from each of a plurality of databases; a first writing module for writing the read data into a distributed file system; a second writing module for, upon receiving a write request for writing data to any one of the plurality of databases, writing the data requested by the write request into the distributed file system; a registration module for registering a target service for processing data in the distributed file system; and a calling module for, upon receiving a request for processing data in the distributed file system, calling the target service to process the data in the distributed file system.

[0007] As an optional example, the above-mentioned second writing module includes: a parsing unit, which is used to parse the data requested to be written by the above-mentioned write request through a parser when a write request to write data to any one of the above-mentioned multiple databases is received; a transmission unit, which is used to transmit the parsed data to the stream computing task through a message bus; and a computing unit, which is used to calculate the parsed data by the above-mentioned stream computing task and write it into the above-mentioned distributed file system.

[0008] As an optional example, the above-mentioned device also includes: a monitoring module, which is used to monitor the service status and life cycle of each service including the above-mentioned target service after registering the target service for processing the data in the above-mentioned distributed file system; a deregistration module, which is used to deregister the above-mentioned service whose life cycle has ended when the above-mentioned life cycle of any of the above-mentioned services ends; and a sending module, which is used to send an alarm message when the above-mentioned service status of any of the above-mentioned services is abnormal.

[0009] As an optional example, the above-mentioned device also includes: a creation module, which is used to create a target sandbox when an application request for applying to use a sandbox is received and no available sandbox currently exists, wherein the above-mentioned target sandbox prohibits downloading of data in the above-mentioned target sandbox; a push module, which is used to periodically push the data to be accessed by the above-mentioned first account to the above-mentioned target sandbox when the first account that generates the above-mentioned application request has the account authority to access the above-mentioned target sandbox; and a control module, which is used to allow the above-mentioned first account to access the data in the above-mentioned target sandbox when the above-mentioned first account logs in to the above-mentioned target sandbox.

[0010] As an optional example, the above-mentioned device also includes: a first detection module, used to periodically detect the status of the above-mentioned target sandbox; a deletion module, used to delete the data in the above-mentioned target sandbox when the above-mentioned target sandbox expires; and a first processing module, used to prohibit the above-mentioned first account from accessing the above-mentioned target sandbox.

[0011] As an optional example, the above-mentioned device also includes: a second detection module, which is used to detect the field type of the data to be written when writing the read data into the distributed file system or when writing the data requested to be written by the above-mentioned write request into the above-mentioned distributed file system, and determine the encryption level of the data to be written according to the above-mentioned field type; a processing module, which is used to, after writing the data to be written into the above-mentioned distributed file system, upon receiving an access request for the first data to be written, and when the access level of the second account that generates the above-mentioned access request is less than the encryption level of the above-mentioned first data, use special characters to at least partially replace the content of the above-mentioned first data, return the replaced above-mentioned first data to the above-mentioned second account, and return the above-mentioned first data to the above-mentioned second account when the access level of the above-mentioned second account is greater than or equal to the encryption level of the above-mentioned first data.

[0012] As an optional example, the above-mentioned device also includes: a determination module, which is used to determine the current access volume of the above-mentioned distributed file system when an access request for accessing data in the above-mentioned distributed file system is received; and a second processing module, which is used to prohibit the third account that generates the above-mentioned access request from accessing the data in the above-mentioned distributed file system when the above-mentioned current access volume exceeds the preset access volume.

[0013] As an optional example, the above-mentioned device also includes: an acquisition module for acquiring a target image, wherein the above-mentioned target image includes text data; an identification module for identifying the above-mentioned text data in the above-mentioned target image to obtain target data; and a storage module for storing the above-mentioned target data in the above-mentioned distributed file system.

[0014] According to another aspect of the embodiments of the present invention, a storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the data processing method is executed.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data processing method through the computer program.

[0016] In an embodiment of the present invention, a method is adopted for reading data from each of a plurality of databases; writing the read data into a distributed file system; upon receiving a write request to write data to any of the plurality of databases, writing the data requested by the write request into the distributed file system; registering a target service for processing data in the distributed file system; upon receiving a processing request for data in the distributed file system, calling the target service to process the data in the distributed file system. In the above method, for a plurality of non-interoperable databases, data in each database can be read and written into the distributed file system, and for new data to be written into any of the plurality of databases, it is written into the distributed file system. Different units can access the data in the distributed file system, thereby achieving the purpose of breaking through data barriers, and further solving the technical problem that data barriers exist between units and data cannot be accessed interoperably. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of an optional data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a system diagram of an optional data processing method according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a sandbox application according to an optional data processing method according to an embodiment of the present invention;

[0021] Figure 4 is a schematic structural diagram of an optional data processing device according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to a first aspect of an embodiment of the present invention, a data processing method is provided, optionally, as follows: Figure 1 As shown, the above method includes:

[0026] S102, reading data from each of the multiple databases; and writing the read data into a distributed file system;

[0027] S104, when receiving a write request to write data to any one of the multiple databases, writing the data requested by the write request to the distributed file system;

[0028] S106, registering a target service for processing data in the distributed file system;

[0029] S108: When a request for processing data in the distributed file system is received, a target service is called to process the data in the distributed file system.

[0030] Optionally, in this embodiment, the multiple databases may be databases of different units or systems, and the data in the multiple databases are not interoperable. Users of unit A access database A, and users of unit B access database B. Users of unit A cannot access the database of unit B.

[0031] Optionally, in this embodiment, for the aforementioned multiple databases, data in each database can be read and copied or migrated to a distributed file system. The distributed file system can be an intranet database or an extranet database, and can receive user access requests and return corresponding data.

[0032] Optionally, in this embodiment, data already existing in multiple databases is copied or migrated to the distributed file system, while data to be written to any one of the multiple databases in real time is written to the distributed file system. Thus, the distributed file system contains data existing in multiple databases and data to be written to multiple databases, and can provide external services through the distributed file system.

[0033] Optionally, because data in a distributed file system comes from multiple sources, such as databases from multiple units, departments, or systems, different access permissions can be assigned to data from different sources. For example, after unit A's data is imported into the distributed file system, employees of unit A can be granted access to data originally belonging to unit A in the distributed file system. However, employees of unit A cannot access data belonging to unit B in the distributed file system. By restricting permissions, data can be accessed securely.

[0034] Optionally, in this embodiment, after copying or migrating data to a distributed file system, a target service for accessing data in the distributed file system can be registered in the system. Different target services can have different functions, and using target services with different functions to access data in the distributed file system can meet different user needs.

[0035] In the above method, data from multiple, non-interconnected databases can be read from each database and written to the distributed file system. New data to be written to any one of the multiple databases is written to the distributed file system. Different units can access data in the distributed file system, thus breaking down data barriers.

[0036] As an optional example, when a write request to write data to any one of the multiple databases is received, writing the data requested by the write request to the distributed file system includes:

[0037] When receiving a write request to write data into any one of the plurality of databases, parsing the data requested to be written by the write request through a parser;

[0038] The parsed data is transmitted to the stream computing task through the message bus;

[0039] The stream computing task calculates the parsed data and writes it to the distributed file system.

[0040] Optionally, in this embodiment, for data written in real time into any one of multiple databases, a parser can be called to parse the data to be written, and then the data to be written is transmitted to the stream computing service through the message bus. The stream computing service performs calculations and writes the calculation results to the distributed file system.

[0041] As an optional example, after registering the target service for processing data in the distributed file system, the method further includes:

[0042] Monitor the service status and lifecycle of each service, including the target service;

[0043] When the lifecycle of any service ends, the service whose lifecycle has ended will be deregistered;

[0044] When the service status of any service is abnormal, an alarm message is sent.

[0045] Optionally, in this embodiment, each registered service is assigned a lifecycle, which is used to monitor the service's lifetime. In this embodiment, a lifecycle can be assigned to a registered service during registration. When the lifecycle expires, the corresponding service is deregistered. In this embodiment, the service status of services whose lifecycles have not yet expired can also be monitored. If the service status is abnormal, an alert can be sent to the administrator.

[0046] As an optional example, the above method further includes:

[0047] Upon receiving an application request for using a sandbox, if no sandbox is currently available, creating a target sandbox, wherein the target sandbox prohibits downloading data in the target sandbox;

[0048] When the first account generating the application request has the account permission to access the target sandbox, periodically push the data to be accessed by the first account to the target sandbox;

[0049] When the first account logs in to the target sandbox, the first account accesses the data in the target sandbox.

[0050] Optionally, in this embodiment, if a user wishes to access data in a distributed file system, they can choose to use the public network or a sandbox. Sandboxes can protect data. For example, if a user chooses to access data using a sandbox, the system searches for available sandboxes. If one exists, it can be allocated to the user. The data the user wishes to access is transferred to the sandbox in batches, and the user can then access the data within the sandbox, ensuring secure access. If no sandbox exists, a sandbox is created and allocated to the user.

[0051] As an optional example, the above method further includes:

[0052] Periodically check the status of the target sandbox;

[0053] When the target sandbox expires, delete the data in the target sandbox;

[0054] Prohibit the first account from accessing the target sandbox.

[0055] Optionally, in this embodiment, a sandbox usage limit can be set. The usage limit for different sandboxes can be set based on user permissions, account level, etc. When the sandbox usage limit expires, the user is prohibited from accessing the sandbox and the data in the sandbox is deleted. The sandbox can then be applied for by other users.

[0056] As an optional example, the above method further includes:

[0057] When writing read data into the distributed file system or when writing data requested by a write request into the distributed file system, detecting a field type of the data to be written and determining an encryption level of the data to be written based on the field type;

[0058] After the data to be written is written into the above-mentioned distributed file system, when an access request for the written first data is received, if the access level of the second account generating the above-mentioned access request is lower than the encryption level of the above-mentioned first data, the content of the above-mentioned first data is at least partially replaced with special characters, and the replaced first data is returned to the above-mentioned second account; if the access level of the above-mentioned second account is greater than or equal to the encryption level of the above-mentioned first data, the above-mentioned first data is returned to the above-mentioned second account.

[0059] Optionally, in this embodiment, sensitive content can be removed from data to be written to the distributed file system before it is written to the distributed file system. Sensitive content removal can be determined based on the data field type. The encryption level of the data is determined based on the field type. For example, if the field type is 11 digits, the data may be a mobile phone number, and the encryption level can be set to medium. For example, if the field type is 18 digits or 17 digits + 1 digit X, the data may be an ID number, and the encryption level can be high. For example, if the field type is 2-4 Chinese characters, the data may be a name, and the encryption level can be low. Different data types have different encryption levels. When a second account requests data from the distributed file system, the second account's access level is checked. If it is low, the second account can view data with a low encryption level. For example, if the second account wants to view Zhang San's name, ID number, and mobile phone number, based on the second account's access level and the data's encryption level, only Zhang San's name will be displayed, and the ID number and mobile phone number can be replaced with special characters and not displayed; alternatively, partial digits in the ID number and mobile phone number can be replaced with special characters and not displayed in full. For example, the special characters may be asterisks, dashes, exclamation marks, question marks, etc. If the access level of the second account is intermediate, then Zhang San's name and phone number may be displayed, and Zhang San's ID number may be replaced by special characters.

[0060] As an optional example, the above method further includes:

[0061] Upon receiving an access request for accessing data in the distributed file system, determining a current access volume of the distributed file system;

[0062] When the current access volume exceeds the preset access volume, the third account that generates the access request is prohibited from accessing the data in the distributed file system.

[0063] Optionally, in this embodiment, access services can be provided externally through a gateway. When users access data in a distributed file system, they can access the gateway. The gateway can provide external functions such as current limiting, black and white list restrictions, unified identity authentication, replay attacks, grayscale publishing, request forwarding, and unified exception handling. For current limiting, when an access request to a distributed file system is obtained, the current access volume of the distributed file system can be queried. If the preset access volume is reached or exceeded, the access request to the distributed file system is denied. For black and white lists, users in the white list do not need to be verified, and users in the black list can be denied access, etc. When the number of times a user accesses the distributed file system reaches a threshold and each access is normal, the user can be added to the white list.

[0064] As an optional example, the above method further includes:

[0065] Acquire a target image, wherein the target image includes text data;

[0066] Recognize text data in the target image to obtain target data;

[0067] Store the target data in a distributed file system.

[0068] Optionally, this embodiment can also take a photo of a paper document and scan the resulting image to obtain the data information in the image and generate an electronic version of the document. This method can improve the data upload speed when breaking down data barriers between different units and systems.

[0069] Optionally, this embodiment aims to address data barriers between multiple parties. A big data sharing bridging subsystem can be used to break down data barriers, aggregate all data, and provide unified data services to the outside world. A comprehensive rights management system can provide data security.

[0070] This embodiment utilizes this technical solution through the Lambda architecture, implementing both stream and batch synchronization based on data criticality. For data with low criticality requirements, such as data residing in multiple databases, data synchronization is performed periodically using a data synchronization tool and synchronized to a distributed file system. For data with high real-time requirements, data synchronization can be performed through a stream computing platform. Data can be parsed using the relational database management system (RDBMS) binary log parser and then transmitted to downstream stream computing tasks via a message bus. The downstream stream computing tasks then receive the corresponding data, parse it, and perform storage operations, writing the data to the distributed file system. To ensure data security, sensitive data can be dynamically desensitized, such as filtering out ID numbers and certain sensitive keywords. A security sandbox is also provided to prevent data leakage.

[0071] In this embodiment, a secure sandbox at the user environment level can be built using the application container engine (Docker). Key fields are dynamically desensitized when data is generated based on the metadata management system. Multiple data sources are connected to the cloud stream computing platform for real-time data synchronization and storage, with data latency in seconds. Image recognition can also be performed based on optical character recognition (OCR), allowing for data upload via photo capture. Excel file processing tools can be integrated to parse Excel spreadsheets and complete data upload and storage. Geospatial, Word, PDF, and other documents are stored in a distributed file system for archiving. A gateway is provided externally, offering a range of application programming interface (API) features, including high performance, traceability, blacklist and whitelist filtering, and rate limiting. Files can be downloaded in formats such as CSV, JSON, XML, and EXCEL, with the ability to push files regularly.

[0072] Figure 2 This is a system framework diagram of this embodiment.

[0073] For the external access side of the system, you can use the Kong gateway, which achieves high performance and multi-plug-in effects. The Kong gateway is responsible for current limiting, black and white list verification, unified identity authentication, replay attacks, grayscale release, request forwarding, unified exception handling, etc.

[0074] On the service side, services can be registered with the Dynamic Naming and Configuration Service (NACOS) service registry. Services can call each other via Feign, and link tracking uses the distributed tracing system SkyWalking. The service status within the container is monitored using the monitoring and alarm system Prometheus + Grafana. The entire container lifecycle is managed by the orchestration management tool Kubernetes (K8s).

[0075] On the data security side, when calling services and using data, it is possible to limit whether the data is exposed to the public network or used in a secure sandbox. If it is a secure sandbox, the user can apply for the sandbox in advance. After approval, the shared exchange platform will periodically push the data into the secure sandbox. Data can only be operated within the sandbox, and download operations are not allowed, ensuring the security of the data. At the same time, the big data platform manages data security for metadata. The data fields are defined densely, and the data will be dynamically desensitized when it is used.

[0076] Figure 3This is a security sandbox sequence diagram for this embodiment. Users can apply for a sandbox from the shared exchange platform. The shared exchange platform allocates the sandbox to the user and returns the container IP address, port, username, and password. When the user uses the sandbox, the platform periodically pushes data to the sandbox, allowing the user to access the data.

[0077] On the big data support side, the transmission tool Sqoop can be used to periodically synchronize data in batches, and unified scheduling tasks can be used to execute Hive SQL job flows to complete the layered construction of data warehouses. For database tables with high real-time requirements, they are parsed using corresponding RDBMS binlog parsers such as Canal and Kafka-Connector, and then transmitted downstream through the message bus. After the downstream stream computing tasks obtain the corresponding data, they parse and store it in the warehouse.

[0078] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0079] According to another aspect of the embodiment of the present application, a data processing device is also provided. Figure 4 Shown, including:

[0080] A reading module 402 is used to read data from each of the multiple databases;

[0081] A first writing module 404 is used to write the read data into the distributed file system;

[0082] The second writing module 406 is configured to write the data requested by the writing request into the distributed file system when receiving a writing request to write data to any one of the multiple databases;

[0083] A registration module 408, configured to register a target service for processing data in a distributed file system;

[0084] The calling module 410 is configured to call a target service to process the data in the distributed file system when a request for processing the data in the distributed file system is received.

[0085] Optionally, in this embodiment, the multiple databases may be databases of different units or systems, and the data in the multiple databases are not interoperable. Users of unit A access database A, and users of unit B access database B. Users of unit A cannot access the database of unit B.

[0086] Optionally, in this embodiment, for the aforementioned multiple databases, data in each database can be read and copied or migrated to a distributed file system. The distributed file system can be an intranet database or an extranet database, and can receive user access requests and return corresponding data.

[0087] Optionally, in this embodiment, data already existing in multiple databases is copied or migrated to the distributed file system, while data to be written to any one of the multiple databases in real time is written to the distributed file system. Thus, the distributed file system contains data existing in multiple databases and data to be written to multiple databases, and can provide external services through the distributed file system.

[0088] Optionally, because data in a distributed file system comes from multiple sources, such as databases from multiple units, departments, or systems, different access permissions can be assigned to data from different sources. For example, after unit A's data is imported into the distributed file system, employees of unit A can be granted access to data originally belonging to unit A in the distributed file system. However, employees of unit A cannot access data belonging to unit B in the distributed file system. By restricting permissions, data can be accessed securely.

[0089] Optionally, in this embodiment, after copying or migrating data to a distributed file system, a target service for accessing data in the distributed file system can be registered in the system. Different target services can have different functions, and using target services with different functions to access data in the distributed file system can meet different user needs.

[0090] In the above method, data from multiple, non-interconnected databases can be read from each database and written to the distributed file system. New data to be written to any one of the multiple databases is written to the distributed file system. Different units can access data in the distributed file system, thus breaking down data barriers.

[0091] As an optional example, the second writing module includes:

[0092] A parsing unit, configured to, upon receiving a write request to write data into any one of the plurality of databases, parse the data requested to be written by the write request through a parser;

[0093] The transmission unit is used to transmit the parsed data to the stream computing task through the message bus;

[0094] The computing unit is used to calculate the parsed data by the stream computing task and write it to the distributed file system.

[0095] Optionally, in this embodiment, for data written in real time into any one of multiple databases, a parser can be called to parse the data to be written, and then the data to be written is transmitted to the stream computing service through the message bus. The stream computing service performs calculations and writes the calculation results to the distributed file system.

[0096] As an optional example, the above device further includes:

[0097] A monitoring module, configured to monitor the service status and life cycle of each service including the target service after registering the target service for processing data in the distributed file system;

[0098] The deregistration module is used to deregister any service when its lifecycle ends.

[0099] The sending module is used to send an alarm message when the service status of any service is abnormal.

[0100] Optionally, in this embodiment, each registered service is assigned a lifecycle, which is used to monitor the service's lifetime. In this embodiment, a lifecycle can be assigned to a registered service during registration. When the lifecycle expires, the corresponding service is deregistered. In this embodiment, the service status of services whose lifecycles have not yet expired can also be monitored. If the service status is abnormal, an alert can be sent to the administrator.

[0101] As an optional example, the above device further includes:

[0102] a creation module, configured to, upon receiving an application request for using a sandbox, create a target sandbox if no sandbox is currently available, wherein the target sandbox prohibits downloading data in the target sandbox;

[0103] A push module, configured to periodically push data to be accessed by the first account to the target sandbox when the first account generating the application request has account authority to access the target sandbox;

[0104] The control module is configured to enable the first account to access data in the target sandbox when the first account logs into the target sandbox.

[0105] Optionally, in this embodiment, if a user wishes to access data in a distributed file system, they can choose to use the public network or a sandbox. Sandboxes can protect data. For example, if a user chooses to access data using a sandbox, the system searches for available sandboxes. If one exists, it can be allocated to the user. The data the user wishes to access is transferred to the sandbox in batches, and the user can then access the data within the sandbox, ensuring secure access. If no sandbox exists, a sandbox is created and allocated to the user.

[0106] As an optional example, the above device further includes:

[0107] A first detection module is used to periodically detect the status of the target sandbox;

[0108] The deletion module is used to delete the data in the target sandbox when the target sandbox expires;

[0109] The first processing module is configured to prohibit the first account from accessing the target sandbox.

[0110] Optionally, in this embodiment, a sandbox usage limit can be set. The usage limit for different sandboxes can be set based on user permissions, account level, etc. When the sandbox usage limit expires, the user is prohibited from accessing the sandbox and the data in the sandbox is deleted. The sandbox can then be applied for by other users.

[0111] As an optional example, the above device further includes:

[0112] a second detection module, configured to detect a field type of the data to be written when writing the read data into the distributed file system or when writing the data requested by the write request into the distributed file system, and determine an encryption level of the data to be written based on the field type;

[0113] The processing module is used to, after writing the data to be written into the above-mentioned distributed file system, upon receiving an access request for the written first data, and if the access level of the second account generating the above-mentioned access request is less than the encryption level of the above-mentioned first data, use special characters to at least partially replace the content of the above-mentioned first data, return the replaced first data to the above-mentioned second account, and if the access level of the above-mentioned second account is greater than or equal to the encryption level of the above-mentioned first data, return the above-mentioned first data to the above-mentioned second account.

[0114] Optionally, in this embodiment, sensitive content can be removed from data to be written to the distributed file system before it is written to the distributed file system. Sensitive content removal can be determined based on the data field type. The encryption level of the data is determined based on the field type. For example, if the field type is 11 digits, the data may be a mobile phone number, and the encryption level can be set to medium. For example, if the field type is 18 digits or 17 digits + 1 digit X, the data may be an ID number, and the encryption level can be high. For example, if the field type is 2-4 Chinese characters, the data may be a name, and the encryption level can be low. Different data types have different encryption levels. When a second account requests data from the distributed file system, the second account's access level is checked. If it is low, the second account can view data with a low encryption level. For example, if the second account wants to view Zhang San's name, ID number, and mobile phone number, based on the second account's access level and the data's encryption level, only Zhang San's name will be displayed, and the ID number and mobile phone number can be replaced with special characters and not displayed; alternatively, partial digits in the ID number and mobile phone number can be replaced with special characters and not displayed in full. For example, the special characters may be asterisks, dashes, exclamation marks, question marks, etc. If the access level of the second account is intermediate, then Zhang San's name and phone number may be displayed, and Zhang San's ID number may be replaced by special characters.

[0115] As an optional example, the above device further includes:

[0116] a determination module, configured to determine a current access volume of the distributed file system upon receiving an access request for accessing data in the distributed file system;

[0117] The second processing module is configured to prohibit a third account generating an access request from accessing data in the distributed file system when the current access volume exceeds a preset access volume.

[0118] Optionally, in this embodiment, access services can be provided externally through a gateway. When users access data in a distributed file system, they can access the gateway. The gateway can provide external functions such as current limiting, black and white list restrictions, unified identity authentication, replay attacks, grayscale publishing, request forwarding, and unified exception handling. For current limiting, when an access request to a distributed file system is obtained, the current access volume of the distributed file system can be queried. If the preset access volume is reached or exceeded, the access request to the distributed file system is denied. For black and white lists, users in the white list do not need to be verified, and users in the black list can be denied access, etc. When the number of times a user accesses the distributed file system reaches a threshold and each access is normal, the user can be added to the white list.

[0119] As an optional example, the above device further includes:

[0120] An acquisition module, configured to acquire a target image, wherein the target image includes text data;

[0121] A recognition module is used to recognize text data in a target image and obtain target data;

[0122] The storage module is used to store the target data in the distributed file system.

[0123] Optionally, this embodiment can also take a photo of a paper document and scan the resulting image to obtain the data information in the image and generate an electronic version of the document. This method can improve the data upload speed when breaking down data barriers between different units and systems.

[0124] For other examples of this embodiment, please refer to the above examples and will not be repeated here.

[0125] Figure 5 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506 and a communication bus 508, wherein the processor 502, the communication interface 504 and the memory 506 communicate with each other through the communication bus 508, wherein,

[0126] Memory 506, for storing computer programs;

[0127] The processor 502 is configured to execute the computer program stored in the memory 506 to implement the following steps:

[0128] Read data from each of the multiple databases and write the read data to the distributed file system;

[0129] When receiving a write request to write data to any one of the multiple databases, writing the data requested by the write request to the distributed file system;

[0130] Register target services for processing data in the distributed file system;

[0131] When a request for processing data in the distributed file system is received, a target service is called to process the data in the distributed file system.

[0132] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The use of only one thick line in the figure does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the above electronic devices and other devices.

[0133] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0134] As an example, the memory 506 may include, but is not limited to, the read module 402, the first write module 404, the second write module 406, the registration module 408, and the call module 410 in the request processing device. Furthermore, the memory 506 may also include, but is not limited to, other module units in the request processing device, which will not be described in detail in this example.

[0135] The above-mentioned processor can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0136] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0137] It can be understood by those skilled in the art that Figure 5The structure shown is for illustration only. The device for implementing the above request processing method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0138] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0139] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned data processing method are executed.

[0140] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0142] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (such as personal computers, servers, or network devices) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0143] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Read data from each of the multiple databases; Write the read data to the distributed file system; When receiving a write request to write data to any one of the multiple databases, writing the data requested to be written by the write request into the distributed file system; Registering a target service for processing data in the distributed file system; In case of receiving a request for processing data in the distributed file system, calling the target service to process the data in the distributed file system; The method further includes: upon receiving an application request for using a sandbox, if no available sandbox currently exists, creating a target sandbox, wherein the target sandbox prohibits downloading data in the target sandbox; in a case where a first account generating the application request has account authority to access the target sandbox, periodically pushing data to be accessed by the first account to the target sandbox; and in a case where the first account logs into the target sandbox, allowing the first account to access the data in the target sandbox.

2. The method according to claim 1, characterized in that When receiving a write request to write data to any one of the multiple databases, writing the data requested to be written by the write request into the distributed file system includes: When receiving a write request to write data into any one of the multiple databases, parsing the data requested to be written by the write request through a parser; The parsed data is transmitted to the stream computing task through the message bus; The stream computing task calculates the parsed data and writes the data into the distributed file system.

3. The method according to claim 1, characterized in that After registering a target service for processing data in the distributed file system, the method further includes: Monitor the service status and life cycle of each service including the target service; When the life cycle of any of the services ends, canceling the service whose life cycle has ended; When the service status of any of the services is abnormal, an alarm message is sent.

4. The method according to claim 1, characterized in that The method further comprises: Periodically detecting the status of the target sandbox; When the target sandbox expires, deleting the data in the target sandbox; The first account is prohibited from accessing the target sandbox.

5. The method according to claim 1, characterized in that The method further comprises: When writing the read data into the distributed file system or when writing the data requested to be written by the write request into the distributed file system, detecting the field type of the data to be written, and determining the encryption level of the data to be written according to the field type; After writing the data to be written into the distributed file system, when an access request for the written first data is received, if the access level of the second account generating the access request is less than the encryption level of the first data, use special characters to at least partially replace the content of the first data, return the replaced first data to the second account, and return the first data to the second account when the access level of the second account is greater than or equal to the encryption level of the first data.

6. The method according to claim 1, characterized in that The method further comprises: In case of receiving an access request for accessing data in the distributed file system, determining a current access volume of the distributed file system; In a case where the current access amount exceeds the preset access amount, the third account that generates the access request is prohibited from accessing the data in the distributed file system.

7. The method according to any one of claims 1 to 3, 5 to 6, characterized in that: The method further comprises: Acquire a target image, wherein the target image includes text data; Recognize the text data in the target image to obtain target data; The target data is stored in the distributed file system.

8. A data processing device, characterized in that: include: A reading module is used to read data from each of the multiple databases; A first writing module, used for writing the read data into the distributed file system; A second writing module is used for, when receiving a writing request to write data into any one of the multiple databases, writing the data requested to be written by the writing request into the distributed file system; A registration module, used for registering a target service for processing data in the distributed file system; A calling module, configured to call the target service to process the data in the distributed file system when receiving a request for processing the data in the distributed file system; The device further includes: upon receiving an application request for using a sandbox, if no available sandbox currently exists, creating a target sandbox, wherein the target sandbox prohibits downloading data in the target sandbox; in a case where a first account generating the application request has account authority to access the target sandbox, periodically pushing data to be accessed by the first account to the target sandbox; and in a case where the first account logs into the target sandbox, allowing the first account to access the data in the target sandbox.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images