Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining user information and organization information corresponding to a target identity source, converting data formats of the user information and the organization information into a target data format, and storing conversion data obtained after format conversion into a temporary data table; reading conversion data in the temporary data table, and obtaining target data including a user organization relationship based on the conversion data; and storing the target data to a target database, and deleting original identity data in the target database, the original identity data including user information and organization information of a historical data import task corresponding to the target identity source. According to the embodiment of the invention, the technical problems that the identity information import task of the office system occupies a relatively large memory and errors such as memory overflow are easily caused in the related technology can be solved.

Description

Data processing methods, apparatus, electronic devices and storage media Technical Field

[0001] This application relates to the field of cloud technology, specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In cloud-based office scenarios, after a cloud office system is launched, it typically requires the enterprise user's administrator to create accounts for employees and assign corresponding permissions before employees can use the system. However, for enterprises with a large number of employees, this task usually requires a significant amount of manpower. To improve the efficiency of user information creation, related technologies often employ the method of batch importing member and organizational information from enterprise users into the cloud office system. However, during the batch import process, the entire set of member and organizational information data usually needs to be loaded into memory before further data import processing can be performed. This consumes a large amount of memory and is prone to errors such as memory overflow, further affecting the efficiency of user information creation. Summary of the Invention

[0003] In view of the above problems, this application provides a data processing method, apparatus, electronic device and storage medium to at least solve the technical problem in the related art that the identity information import task of office system has a large memory occupation and is prone to memory overflow and other errors.

[0004] According to a first aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring user information and organization information corresponding to a target identity source; converting the data format of the user information and the organization information into a target data format; storing the converted data obtained after format conversion in a temporary data table; reading the converted data in the temporary data table; and acquiring target data including user-organization relationships based on the converted data; wherein the user-organization relationships are used to characterize the identity attributes of each user and the organizational attributes of each organization in the target identity source; storing the target data in a target database; and deleting the original identity data in the target database, wherein the original identity data includes user information and organization information of a historical data import task corresponding to the target identity source.

[0005] According to a second aspect of the embodiments of this application, a data processing apparatus is provided, comprising: a first acquisition unit, configured to acquire user information and organization information corresponding to a target identity source, convert the data format of the user information and the organization information into a target data format, and store the converted data obtained after format conversion in a temporary data table; a first processing unit, configured to read the converted data in the temporary data table, and acquire target data including user-organizational relationships based on the converted data; the user-organizational relationships are used to characterize the identity attributes of each user in the target identity source, and the organizational attributes of each organization; and a second processing unit, configured to store the target data in a target database, and delete the original identity data in the target database, the original identity data including user information and organization information of the historical data import task corresponding to the target identity source.

[0006] According to a third aspect of the embodiments of this application, a data processing method is also provided, applied to a cloud office system. The method includes: obtaining user information and organization information corresponding to a target identity source; converting the data format of the user information and the organization information into a target data format; storing the converted data obtained after format conversion in a temporary data table; the target identity source is a data source for importing user information and organization information into the cloud office system; reading the converted data in the temporary data table; obtaining target data including user-organizational relationships based on the converted data; the user-organizational relationships are used to characterize the identity attributes of each user in the target identity source, and the organizational attributes of each organization; storing the target data in a target database of the cloud office system; deleting the original identity data in the target database; the original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0007] According to a fourth aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data processing method of the first aspect through the computer program.

[0008] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the data processing method of the first aspect described above when it is run.

[0009] According to a sixth aspect of the embodiments of this application, a computer program product is also provided, including a computer program, which is executed by a processor using the data processing method of the first aspect described above.

[0010] In this embodiment, the method involves obtaining user and organizational information corresponding to a target identity source, converting the data format of the user and organizational information into a target data format, storing the converted data in a temporary data table, reading the converted data from the temporary data table, obtaining target data including user-organizational relationships based on the converted data, storing the target data in a target database, and deleting the original identity data in the target database. The original identity data includes user and organizational information from the historical data import task corresponding to the target identity source. This application uses a temporary data table to store the full data of member and organizational information, and obtains target data including user-organizational relationships from the data stored in the temporary data table. This eliminates the need to load the full data of member and organizational information into memory for further data import processing, avoids errors such as memory overflow, and greatly improves the efficiency of user information creation. Attached Figure Description

[0011] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0012] Figure 1 is a schematic diagram of an application environment for an optional data processing method according to an embodiment of this application;

[0013] Figure 2 is a flowchart illustrating an optional data processing method according to an embodiment of this application;

[0014] Figure 3 is a schematic diagram of an optional data processing architecture according to related technologies;

[0015] Figure 4 is a schematic diagram of an optional data processing architecture according to an embodiment of this application;

[0016] Figure 5 is a flowchart illustrating another optional data processing method according to an embodiment of this application;

[0017] Figure 6 is a flowchart illustrating another optional data processing method according to an embodiment of this application;

[0018] Figure 7 is a flowchart illustrating another optional data processing method according to an embodiment of this application;

[0019] Figure 8 is a flowchart illustrating another optional data processing method according to an embodiment of this application;

[0020] Figure 9 is a flowchart illustrating another optional data processing method according to an embodiment of this application;

[0021] Figure 10 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0022] Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] As an optional implementation, the data processing method described above in this application can be applied to the application environment shown in Figure 1. As shown in Figure 1, user 102 and electronic device 104 can interact. Electronic device 104 includes a memory 106 and a processor 108; the memory 106 stores user information and organizational information corresponding to each identity source; the processor 108 is used to obtain user information and organizational information corresponding to the target identity source, convert the data format of the user information and organizational information into a target data format, store the converted data obtained after format conversion in a temporary data table; read the converted data in the temporary data table, obtain target data including user organizational relationships based on the converted data; store the target data in a target database, delete the original identity data in the target database, the original identity data including user information and organizational information of the previous data import task corresponding to the target identity source.

[0026] Optionally, the aforementioned electronic device 104 includes, but is not limited to, mobile phones, laptops, tablets, PDAs, mobile internet devices (MIDs), desktop computers, smart TVs, etc. The aforementioned electronic device 104 can also be a server, which can be a single server, a server cluster composed of multiple servers, or a cloud server. The aforementioned cloud server includes, but is not limited to, private cloud servers or public cloud servers. The above is merely an example, and no limitation is made in this embodiment.

[0027] In related technologies, to improve the efficiency of user information creation during the use of online office systems, a common approach is to batch import member and organizational information of enterprise users into the online office system. However, during the batch import process, it is usually necessary to load the full amount of member and organizational information data into memory before further data import processing can be performed. This consumes a large amount of memory, is prone to errors such as memory overflow, and also affects the efficiency of user information creation. In addition, as shown in Figure 2, for different identity data sources (third-party identity sources 1 to 3 in Figure 2), a corresponding data processing component needs to be customized for each identity source during the batch data import process. For example, a customized data processing component Z1 is required for third-party identity source 1, a customized data processing component Z2 is required for third-party identity source 2, and a customized data processing component Z3 is required for third-party identity source 3. This not only increases the workload of program development but also results in poor program code universality and maintainability.

[0028] To address the aforementioned technical problems, as an optional implementation method, as shown in FIG3, this application provides a data processing method, including the following steps:

[0029] S302, obtain the user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data in a temporary data table.

[0030] Specifically, in this embodiment, the target identity source (third-party identity source) can be user information and organizational information stored in the original system currently being used by the user; the target identity source can also be a data document storing user information and organizational information, etc. The current system is the system from which user information and organizational information are to be imported, and the target data format can be the file format and field names used by the current system, etc.

[0031] In one example, assuming the user of the current system is Company A, as shown in Figure 4, before using the current system, Company A needs to create user information and organizational information corresponding to its company within the system. Company A's administrator imports the user information and organizational information (third-party identity source 1) into the current system in batches. After the current system obtains the user information and organizational information corresponding to third-party identity source 1, it uses data conversion component S1 to convert the data format of the user information and organizational information into the target data format. The converted data is then stored in a temporary data table. This temporary data table can be created in the database. By storing the target data, including user and organizational relationships, the data import process can proceed without loading the full amount of user and organizational information data into memory, thus reducing the occurrence of memory overflow and other abnormal situations. It should be noted that the data conversion component S1 is the data conversion component matched to third-party identity source 1. After the current system obtains the identity source identifier of third-party identity source 1, it will automatically match the data conversion component S1 corresponding to that identity source and use data conversion component S1 to convert the data format of the user information and organizational information.

[0032] S304, Read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data; the user organizational relationships are used to characterize the identity attributes of each user in the target identity source, as well as the organizational attributes of each organization.

[0033] Specifically, in this embodiment of the application, as shown in Figure 4, the process includes, but is not limited to, reading transformation data from a temporary data table using a general data processing component. This general data processing component can process transformation data corresponding to different identity sources. This processing operation includes, but is not limited to, obtaining the user organizational relationships corresponding to the identity sources; or performing quota processing on the transformation data, that is, retaining information from third-party identity sources up to the maximum number of users supported by the current system. Taking the aforementioned company A as an example, based on the general data processing component, information such as user attributes, user permissions, and organizational attributes of each organization in the transformation data corresponding to company A is obtained, and target data including the user organizational relationships of company A is determined. The aforementioned user organizational relationships can be, for example, the organization to which a user belongs, i.e., the user's identity attribute; and the sub-member organizations included in each organization, i.e., the organization's organizational attributes, etc.

[0034] S306, store the target data in the target database, and delete the original identity data in the target database. The original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0035] Specifically, in this embodiment, the aforementioned historical data import task includes the previous batch import task of the current batch import task; in one example, after obtaining the target data corresponding to the current batch import task, the target data is stored in the target database, for example, the target data is stored in the current system's formal data table. It should be noted that the batch import task itself is a full data import, that is, the data corresponding to the current batch import task overwrites the data corresponding to the previous batch import task (original identity data).

[0036] This application uses a temporary data table to store the full data of user and organizational information. Target data, including user-organizational relationships, is obtained from the data stored in the temporary data table. This eliminates the need to load the full user and organizational information into memory for further data import processing, avoiding errors such as memory overflow and significantly improving the efficiency of user information creation. Furthermore, this application differs from the data processing flow of the customized data processing component shown in Figure 2. This application divides the identity source data processing flow into a data formatting process and a data import process. Specifically, the data formatting process only requires data format conversion using a customized data conversion component, while the data import process uses a general data processing component to import information. This not only greatly reduces the workload of program development but also improves the versatility and maintainability of the program code.

[0037] In one or more embodiments, prior to obtaining the user information corresponding to the target identity source, the process includes:

[0038] Receive a request to import all information of the target identity source, and in response to the request to import all information, request a resource lock corresponding to the target identity source;

[0039] Based on the resource lock application result, determine whether to execute the data import task corresponding to the full information import request.

[0040] Specifically, in this embodiment, by locking the identity source data that is currently performing a full information import task, it can be ensured that the same identity source corresponds to only one full information import task at any given time. This application receives a full information import request from a target identity source, requests a resource lock corresponding to the target identity source based on the full information import request, and then determines whether to execute the import task corresponding to the full information import request based on the resource lock request result. Through the above technical means, the integrity and consistency of the data of the target identity source during the current system creation process can be ensured.

[0041] In one or more embodiments, determining whether to execute the import task corresponding to the information import request based on the resource lock application result includes:

[0042] If the application result indicates that the resource lock was not obtained, a prompt message is returned, which indicates that the target identity source has an incomplete data import task;

[0043] Based on the application result indicating that the resource lock has been obtained, the data import task corresponding to the full information import request is executed.

[0044] Specifically, in this embodiment, assuming that the current user needs to perform a full information import operation on the target identity source, and other users are also performing data import operations on the target identity source at the same time, the system will generate an application result indicating that the resource lock was not obtained, and then return a prompt message to the current user. The prompt message is used to indicate that there is an incomplete data import task on the target identity source; otherwise, if no other users are performing data import operations on the target identity source, the system will generate an application result indicating that the resource lock was obtained, and then the data import task corresponding to the full information import request can be executed.

[0045] In one or more embodiments, obtaining user information and organization information corresponding to the target identity source, and converting the data format of the user information and the organization information into the target data format, includes:

[0046] Acquire user information and organization information corresponding to the target identity source in batches;

[0047] Based on the data conversion component corresponding to the target identity source, the file format of each batch of user information and organization information is converted into the target file format, and the field names of each batch of user information and organization information are converted into field names in the target format.

[0048] Specifically, in this embodiment, to avoid the impact of excessive data import on system stability, this application employs a batch-based method of retrieving user and organizational information corresponding to the target identity source. Based on the data conversion component corresponding to the target identity source, the file formats of the acquired batches of user and organizational information are converted to the target file format, and the field names in the user and organizational information are converted to the field names in the target format. By processing data in batches, this application avoids excessive consumption of system resources from importing large amounts of data at once, thereby reducing the risk of system downtime or performance degradation. Furthermore, batch processing allows for data verification and error checking within each batch, helping to ensure data integrity and accuracy.

[0049] In one example, as shown in Figure 4, taking a third-party identity source 1 as the target identity source, the aforementioned data conversion component S1 is the data conversion component matched with this third-party identity source 1. After the current system obtains the identity source identifier of the third-party identity source 1, it will automatically match the data conversion component S1 corresponding to that identity source. The data conversion component S1 converts the data formats of the user information and the organization information, converting the file formats in each batch of obtained user information and organization information into the target file format, and converting the field names in the user information and organization information into field names in the target format. The aforementioned target file format is a file format that matches the current system, and the field names in the target format are field names that match the current system.

[0050] In one or more embodiments, reading the transformed data from the temporary data table and obtaining target data including user organizational relationships based on the transformed data includes:

[0051] The transformed data in the temporary data table is obtained in batches;

[0052] The identity attributes of each user and the organizational attributes of each organization are obtained from the converted data;

[0053] Based on the identity attributes of each user and the organizational attributes of each organization, the user-organization relationship corresponding to the target identity source is determined;

[0054] The target data, including the user organizational relationships, is generated based on the transformed data.

[0055] Specifically, in this embodiment, to avoid the large amount of data imported into the system affecting system stability, this application adopts a method of batch-fetching the transformed data from the temporary data table. As shown in Figure 4, taking the aforementioned company A as an example, the identity source corresponding to company A is third-party identity source 1. The user's identity attributes include user attributes and user permissions, etc. Based on the general data processing component, the user attributes, user permissions, organizational attributes, and other information of each user in the transformed data corresponding to company A are obtained to determine the user organizational relationships of company A. Then, target data including the aforementioned user organizational relationships is generated based on the transformed data corresponding to company A. The aforementioned user organizational relationships can be, for example, the organization to which the user belongs, and information such as the sub-member organizations included in each organization.

[0056] As an optional implementation, as shown in FIG5, this application embodiment provides a data processing method, including the following steps:

[0057] S502, obtain the user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data in a temporary data table.

[0058] S504, Read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data.

[0059] S506, The target data is stored in the target database; each user information and organization information in the target data carries a first task identifier of the current data import task.

[0060] S508, based on the existence of a second task identifier in the target database, the user information and organization information corresponding to the second task identifier are determined as the original identity data; the second task identifier is the task identifier corresponding to the historical data import task.

[0061] S510, Delete the original identity data.

[0062] Specifically, in this embodiment, the target database is the database corresponding to the current system. As shown in Figure 4, taking the aforementioned enterprise A as an example, assuming that the identity source corresponding to enterprise A is third-party identity source 1, the first task identifier corresponding to the current full information import task is, for example, task_5, meaning that each user information and organization information corresponding to the current data import task carries task_5. Assuming that the second task identifier corresponding to the historical data import task in the target database is task_4, the user data and organization data corresponding to the second task identifier task_4 are determined as the original identity data, and the original identity data is deleted. Only the user information and organization information corresponding to the current import task are retained.

[0063] The steps S502-S504 above have been explained in the preceding text and will not be repeated here.

[0064] In one or more embodiments, the data processing method further includes:

[0065] Once the current data import task is confirmed to be complete, delete the data in the temporary data table;

[0066] Release the resource lock that has been acquired.

[0067] Specifically, in this embodiment, once it is determined that the current data import task has been completed, the data in the temporary data table can be deleted, and then the resource lock acquired for the target identity source can be released. In this way, subsequent full data import tasks can acquire the resource lock for the target identity source and then import the information.

[0068] As an optional implementation, as shown in FIG6, this application embodiment provides a data processing method, including the following steps:

[0069] S602, obtain the user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data in a temporary data table.

[0070] S604, Read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data.

[0071] S606, store the target data in the target database, and delete the original identity data in the target database. The original identity data includes the user information and organization information of the previous data import task corresponding to the target identity source.

[0072] S608, receive the incremental information synchronization request from the target identity source, and in response to the incremental information synchronization request, obtain change information; the change information includes changed user information and / or changed organization information;

[0073] S610, according to the data conversion component corresponding to the target identity source, the file format of the change information is converted to the target file format, and the field names in the change information are converted to the field names in the target format, so as to obtain the change conversion data corresponding to the change information;

[0074] S612, Based on the change conversion data, generate target change data including the user organizational relationships to be changed;

[0075] S614, Based on the target change data, change the target identity source in the target database.

[0076] Specifically, in this embodiment of the application, after the information of the target identity source is imported into the current system, when the user information and / or organization information in the target identity source changes, the target identity source will send an incremental information synchronization request to the current system, and the current system will obtain the change information, including the changed user information and / or changed organization information, from the target identity source.

[0077] As shown in Figure 4, taking third-party identity source 1 as the target identity source as an example, the data conversion component S1 is a data conversion component matched with third-party identity source 1. Third-party identity source 1 contains changed user information: user P1's latest work department C (not shown in the figure), and changed organizational information: department A1 (not shown in the figure). The data conversion component S1 converts the data formats of the user information and the organizational information, converting the file formats in each batch of acquired user information and organizational information into the target file format, and converting the field names in the user information and organizational information into field names in the target format. The target file format is a file format that matches the current system, and the field names in the target format are field names that match the current system. The system retrieves the identity attributes of each changed user and the organizational attributes of each changed organization from the change and transformation data. Based on the identity attributes of each changed user, the organizational attributes of each changed organization, and the user organizational relationships corresponding to the target identity sources already stored in the target database, the system determines the user organizational relationships to be changed. For example, it compares the latest user information and organizational information in third-party identity source 1 with the information already stored in the target database to check whether the user's identity attributes (such as work department, position, role, etc.) and organizational attributes (such as department name, department structure, etc.) have changed. After detecting a change, it is determined that user P1's identity attributes and / or organizational attributes have changed. If user P1's work department has changed, then user P1's relationship with the new department needs to be changed. If the name of a department has changed, then the relationships of all users belonging to that department also need to be changed.

[0078] Based on the change conversion data, target change data corresponding to the user's organizational relationship to be changed is generated; according to the target change data, the system updates the target identity source in the target database, that is, changes user P1's current work department B to the latest work department C mentioned above, and changes the name of department A to department A1.

[0079] The steps S602-S606 above have been explained in the preceding text and will not be repeated here.

[0080] In one or more embodiments, the data processing method further includes:

[0081] Based on the request times of the full information import request and the incremental information synchronization request, the full information import request and the incremental information synchronization request for the target identity source are added to the data processing queue in sequence.

[0082] The data import tasks corresponding to the incremental information synchronization request and the full information import request in the data processing queue are executed serially.

[0083] Specifically, in this embodiment, as shown in Figure 4, taking a third-party identity source 1 as the target identity source as an example, assuming that there are full information import requests and incremental information synchronization requests for the third-party identity source 1 within a time period, the requests are sequentially added to the data processing queue according to the request time corresponding to the full information import request and the request time corresponding to the incremental information synchronization request. The data import tasks corresponding to the incremental information synchronization request and the full information import request in the data processing queue are executed serially. For example, if a user sends a full information import request to the current system for the third-party identity source 1 at time T, and some data in the third-party identity source 1 changes at time T+T0, the current system will receive an incremental information synchronization request for the corresponding third-party identity source 1. The current system will first execute the data import task corresponding to the full information import request, and then execute the data import task corresponding to the incremental information synchronization request. In one example, the above data processing queue includes, but is not limited to, a RocketMQ message queue, and each data import task in the message queue corresponds to a ShardingKey.

[0084] In one or more embodiments, the above data processing method is applied to a cloud office system, as shown in Figure 7.

[0085] include:

[0086] S702, obtain user information and organization information corresponding to the target identity source, convert the data format of the user information and organization information into the target data format, and store the converted data in a temporary data table; the target identity source is the data source of the user information and organization information to be imported into the cloud office system.

[0087] S704, Read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data; the user organizational relationships are used to characterize the identity attributes of each user in the target identity source, as well as the organizational attributes of each organization;

[0088] S706, store the target data in the target database of the cloud office system, and delete the original identity data in the target database. The original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0089] In one or more embodiments, prior to obtaining the user information corresponding to the target identity source, the process includes:

[0090] The aforementioned cloud office system receives a request to import all information of the target identity source, and in response to the request, requests a resource lock corresponding to the target identity source; the resource lock is a distributed lock.

[0091] Based on the application result indicating that the resource lock has been obtained, the data import task corresponding to the full information import request is executed.

[0092] Based on the above embodiments, as an optional implementation method, this application also provides a data processing method, as shown in FIG8, the data processing method includes:

[0093] S1, the user initiates a data import task to the control component in the system service.

[0094] S2, the control component attempts to acquire the distributed lock of the target identity source corresponding to the data import task.

[0095] S3, if the control component determines that it has not acquired the distributed lock, it returns to the user that there are unfinished tasks at the target identity source.

[0096] S4. If the control component acquires the distributed lock, it generates the task ID corresponding to the data import task.

[0097] S5, the control component sends the above task ID to the user.

[0098] S6, the control component sends an instruction to the data conversion component in the system service to obtain user information and organization information;

[0099] S7, the data transformation component obtains user information and organizational information in batches from third-party data sources.

[0100] S8, third-party data sends user and organizational information to the data transformation component.

[0101] S9, the data conversion component converts the acquired user and organizational information into different formats.

[0102] S10, the data conversion component stores the format-converted data into a temporary data table.

[0103] S11, the data conversion component sends a message to the control component that the task of obtaining the third-party identity source has been completed.

[0104] S12, the control component sends a data import instruction to the general data processing component.

[0105] S13, the general data processing component reads user information and organization information from the temporary data table in batches.

[0106] S14, the general data processing component performs secondary processing on the read data, such as generating user organization relationships corresponding to user information and organization information.

[0107] S15, the general data processing component stores the processed data into the formal data table and associates the processed data with the ID of the current task.

[0108] S16, delete the user information and organization information associated with other task IDs besides the current task ID.

[0109] S17, the control component deletes data from the temporary table.

[0110] S18, the control component releases the distributed lock for the currently imported task request.

[0111] Based on the above embodiments, as an optional implementation method, this application also provides a data processing method, as shown in FIG9, the data processing method includes:

[0112] S1, the user modifies user information and / or organization information from a third-party identity source.

[0113] S2, the third-party identity source sends a notification message about the changed data to the control component of the system service.

[0114] S3, the control component sends a command to the data conversion component in the system service to start the incremental data synchronization task;

[0115] S4, the data transformation component obtains changed user information and / or changed organization information from third-party data sources in batches.

[0116] S5, the data conversion component will convert the obtained changed user information and / or changed organization information into a different format.

[0117] S6, the data conversion component sends the format-converted data to the general data processing component.

[0118] S7, the general data processing component retrieves data associated with the changed data from the formal data table.

[0119] S8, the general data processing component, performs secondary processing on the data after format conversion based on the associated data obtained from the formal data table.

[0120] S9, the general data processing component, stores the processed data into the formal data table.

[0121] S10, delete the original data to be changed in the official table corresponding to the changed data.

[0122] The above-mentioned technical solution of this application has the following beneficial technical effects:

[0123] 1. The data processing workflow is broken down into a data formatting process and a data import process. The data formatting process only handles customized data format conversions, while the data import process handles subsequent generalized steps. If a new identity source is added, developers only need to focus on the data formatting process, greatly reducing the workload of subsequent development.

[0124] 2. Using temporary tables to store data allows for more flexible processing of data that requires full data for further processing, without putting pressure on system memory or causing risks such as memory overflow.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0126] According to another aspect of the embodiments of this application, a data processing apparatus for implementing the above-described data processing method is also provided. As shown in FIG10, the apparatus includes:

[0127] The first acquisition unit 1002 is used to acquire user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data obtained after format conversion into a temporary data table.

[0128] The first processing unit 1004 is used to read the transformation data in the temporary data table and obtain target data including user organizational relationships based on the transformation data; the user organizational relationships are used to characterize the identity attributes of each user in the target identity source and the organizational attributes of each organization;

[0129] The second processing unit 1006 is used to store the target data in the target database and delete the original identity data in the target database. The original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0130] In this embodiment, the method involves obtaining user and organizational information corresponding to a target identity source, converting the data format of the user and organizational information into a target data format, storing the converted data in a temporary data table, reading the converted data from the temporary data table, obtaining target data including user-organizational relationships based on the converted data, storing the target data in a target database, and deleting the original identity data in the target database. The original identity data includes user and organizational information from the historical data import task corresponding to the target identity source. This application uses a temporary data table to store the full data of member and organizational information, and obtains target data including user-organizational relationships from the data stored in the temporary data table. This eliminates the need to load the full data of member and organizational information into memory for further data import processing, avoids errors such as memory overflow, and greatly improves the efficiency of user information creation.

[0131] In one or more embodiments, the data processing apparatus further includes:

[0132] The receiving unit is configured to receive the full information import request of the target identity source and, in response to the full information import request, request the resource lock corresponding to the target identity source.

[0133] The first determining unit is used to determine whether to execute the import task corresponding to the full information import request based on the application result of the resource lock.

[0134] In one or more embodiments, the first determining unit includes:

[0135] The feedback module is used to return a prompt message based on the application result indicating that the resource lock was not obtained. The prompt message is used to indicate that there is an incomplete data import task for the target identity source.

[0136] The execution module is used to execute the data import task corresponding to the full information import request based on the application result indicating that the resource lock has been obtained.

[0137] In one or more embodiments, the first acquisition unit 1002 includes:

[0138] The first acquisition module is used to acquire user information and organization information corresponding to the target identity source in batches;

[0139] The first conversion module is used to convert the file format of each batch of user information and organization information into the target file format according to the data conversion component corresponding to the target identity source, and to convert the field names of each batch of user information and organization information into the field names of the target format.

[0140] In one or more embodiments, the first processing unit 1004 includes:

[0141] The second acquisition module is used to acquire the transformed data in the temporary data table in batches;

[0142] The third acquisition module is used to acquire the identity attributes of each user and the organizational attributes of each organization from the converted data;

[0143] The first determining module is used to determine the user-organization relationship corresponding to the target identity source based on the identity attributes of each user and the organization attributes of each organization.

[0144] A generation module is used to generate the target data, including the user organizational relationships, based on the transformed data.

[0145] In one or more embodiments, each user information in the target data carries a first task identifier for the current data import task, and the second processing unit 1006 includes:

[0146] The second determining unit is used to determine the user data corresponding to the second task identifier as the original identity data based on the existence of the second task identifier in the target database; the second task identifier is the task identifier corresponding to the historical data import task.

[0147] The first deletion unit is used to delete the original identity data.

[0148] In one or more embodiments, the data processing apparatus further includes:

[0149] The second deletion unit is used to delete the data in the temporary data table after determining that the current data import task has been completed.

[0150] The release unit is used to release the resource lock that has been acquired.

[0151] In one or more embodiments, the data processing apparatus further includes:

[0152] The second acquisition unit is configured to receive an incremental information synchronization request from the target identity source, and in response to the incremental information synchronization request, acquire change information; the change information includes changed user information and / or changed organization information;

[0153] The conversion unit is used to convert the file format of the change information into the target file format and the field names in the change information into the field names of the target format according to the data conversion component corresponding to the target identity source, so as to obtain the change conversion data corresponding to the change information;

[0154] The third acquisition unit is used to acquire the identity attributes of each changed user and the organizational attributes of each changed organization from the change conversion data;

[0155] The third determining unit is used to determine the organizational relationship of the user to be changed based on the identity attributes of each changed user and the organizational attributes of each changed organization, and to generate target change data corresponding to the organizational relationship of the user to be changed based on the change conversion data.

[0156] The update unit is used to change the target identity source in the target database according to the target change data.

[0157] In one or more embodiments, the data processing apparatus further includes:

[0158] The adding unit is used to add the full information import request and the incremental information synchronization request for the target identity source to the data processing queue in sequence according to the request time of the full information import request and the incremental information synchronization request.

[0159] The execution unit is used to serially execute the data import tasks corresponding to the incremental information synchronization request and the full information import request in the data processing queue.

[0160] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described data processing method is also provided. This electronic device may be the electronic device shown in FIG1. ​​This embodiment uses a server as an example to illustrate this. As shown in FIG11, the electronic device includes a memory 1102 and a processor 1104. The memory 1102 stores a computer program, and the processor 1104 is configured to execute the steps of any of the above-described method embodiments through the computer program.

[0161] Optionally, in this embodiment, the aforementioned electronic device may be at least one of a plurality of network devices in a computer network.

[0162] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0163] S1, obtain user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data obtained after format conversion into a temporary data table;

[0164] S2, read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data; the user organizational relationships are used to characterize the identity attributes of each user in the target identity source, as well as the organizational attributes of each organization;

[0165] S3, store the target data in the target database, and delete the original identity data in the target database. The original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0166] The memory 1102 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and apparatus in this embodiment. The processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102, thereby implementing the aforementioned data processing method. The memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1102 may further include memory remotely located relative to the processor 1104, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1102 may be used, but is not limited to, to store user information and organizational information.

[0167] As an example, as shown in FIG11, the memory 1102 may include, but is not limited to, the first acquisition unit 1002, the first processing unit 1004, and the second processing unit 1006 in the data processing device. Furthermore, it may include, but is not limited to, other module units in the data processing device, which will not be elaborated upon in this example.

[0168] Optionally, the transmission device 1106 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1106 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1106 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0169] In addition, the aforementioned electronic device also includes a connection bus 1108 for connecting various module components in the aforementioned electronic device.

[0170] In other embodiments, the aforementioned electronic device can be a node in a distributed system, which can be a blockchain system. This blockchain system is formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any type of computing device, such as a server or terminal, can become a node in the blockchain system by joining this peer-to-peer network.

[0171] In one or more embodiments, this application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described above. The computer program is configured to execute the steps of any of the method embodiments described above when running.

[0172] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0173] S1, obtain user information and organization information corresponding to the target identity source, convert the data format of the user information and the organization information into the target data format, and store the converted data obtained after format conversion into a temporary data table;

[0174] S2, read the transformation data in the temporary data table, and obtain target data including user organizational relationships based on the transformation data; the user organizational relationships are used to characterize the identity attributes of each user in the target identity source, as well as the organizational attributes of each organization;

[0175] S3, store the target data in the target database, and delete the original identity data in the target database. The original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

[0176] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0177] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0178] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0179] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed electronic device can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0184] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A data processing method, characterized in that, The method includes: acquiring user information and organization information corresponding to the target identity source; converting the data format of the user information and organization information into a target data format; storing the converted data in a temporary data table; reading the converted data in the temporary data table; and acquiring target data including user-organization relationships based on the converted data; the user-organization relationships are used to characterize the identity attributes of each user and the organization attributes of each organization in the target identity source; storing the target data in a target database; and deleting the original identity data in the target database, wherein the original identity data includes user information and organization information of the historical data import task corresponding to the target identity source.

2. The method according to claim 1, characterized in that, The method further includes: receiving a full information import request from the target identity source; requesting a resource lock corresponding to the target identity source in response to the full information import request; and determining whether to execute the data import task corresponding to the full information import request based on the resource lock request result.

3. The method according to claim 2, characterized in that, The step of determining whether to execute the import task corresponding to the information import request based on the application result of the resource lock includes: returning a prompt message based on the application result indicating that the resource lock was not obtained, wherein the prompt message is used to indicate that there is an incomplete data import task for the target identity source; and executing the data import task corresponding to the full information import request based on the application result indicating that the resource lock has been obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining user information and organization information corresponding to the target identity source and converting the data format of the user information and organization information into the target data format includes: obtaining user information and organization information corresponding to the target identity source in batches; converting the file format of each batch of user information and organization information into the target file format according to the data conversion component corresponding to the target identity source; and converting the field names of each batch of user information and organization information into field names of the target format.

5. The method according to any one of claims 1 to 3, characterized in that, The step of reading the transformation data in the temporary data table and obtaining target data including user organizational relationships based on the transformation data includes: obtaining the transformation data in the temporary data table in batches; obtaining the identity attributes of each user and the organizational attributes of each organization from the transformation data; determining the user organizational relationship corresponding to the target identity source based on the identity attributes of each user and the organizational attributes of each organization; and generating the target data including the user organizational relationships based on the transformation data.

6. The method according to any one of claims 1 to 3, characterized in that, Each user and organization information in the target data carries a first task identifier for the current data import task. Deleting the original identity data in the target database includes: determining the user and organization information corresponding to the second task identifier as the original identity data based on the existence of a second task identifier in the target database; the second task identifier is a task identifier corresponding to a historical data import task; and deleting the original identity data.

7. The method according to claim 3, characterized in that, The method further includes: determining that the current data import task has been completed, deleting the data in the temporary data table; and releasing the resource lock that has been acquired.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: receiving an incremental information synchronization request from the target identity source; responding to the incremental information synchronization request and obtaining change information; the change information includes changed user information and / or changed organization information; according to the data conversion component corresponding to the target identity source, converting the file format of the change information to a target file format and converting the field names in the change information to field names in the target format to obtain change conversion data corresponding to the change information; based on the change conversion data, generating target change data including the user organization relationship to be changed; and changing the target identity source in the target database according to the target change data.

9. The method according to claim 8, characterized in that, The method further includes: adding the full information import request and the incremental information synchronization request for the target identity source to the data processing queue in sequence according to their respective request times; and executing the data import tasks corresponding to the incremental information synchronization request and the full information import request in the data processing queue in sequence.

10. A data processing apparatus, characterized in that, The apparatus includes: a first acquisition unit, configured to acquire user information and organization information corresponding to a target identity source, convert the data format of the user information and the organization information into a target data format, and store the converted data in a temporary data table; a first processing unit, configured to read the converted data in the temporary data table, and acquire target data including user-organization relationships based on the converted data; the user-organization relationships are used to characterize the identity attributes of each user and the organization attributes of each organization in the target identity source; and a second processing unit, configured to store the target data in a target database, and delete the original identity data in the target database, the original identity data including user information and organization information of the historical data import task corresponding to the target identity source.

11. A data processing method, characterized in that, The method, applied to a cloud office system, includes: acquiring user information and organization information corresponding to a target identity source; converting the data format of the user information and organization information into a target data format; storing the converted data in a temporary data table; the target identity source is the data source from which user information and organization information to be imported into the cloud office system; reading the converted data in the temporary data table; and acquiring target data including user-organization relationships based on the converted data; the user-organization relationships are used to characterize the identity attributes of each user and the organizational attributes of each organization in the target identity source; storing the target data in the target database of the cloud office system; and deleting the original identity data in the target database, wherein the original identity data includes user information and organization information from the previous data import task corresponding to the target identity source.

12. The method according to claim 11, characterized in that, The method further includes: receiving a full information import request from the target identity source; responding to the full information import request by applying for a resource lock corresponding to the target identity source; the resource lock is a distributed lock; and based on the application result indicating that the resource lock has been obtained, executing the data import task corresponding to the full information import request.

13. An electronic device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method as described in any one of claims 1 to 9 or 11 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1 to 9 or 11 to 12.

15. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1 to 9 or 11 to 12.

Images