Big data component information processing method, device, equipment, medium and product

Abstract

The application relates to an information processing method and device of a big data component, equipment, medium and product, and relates to the technical field of big data. In the application, a big data component record is read from a database and stored in a dependency chain table; the database stores a plurality of big data component records; a dictionary data structure is filled by traversing the dependency chain table and stores a big data component name and a set of big data component names having a direct correlation relationship with the big data component name; a total mapping structure is filled by traversing the dictionary data structure and stores a big data component name and a set of big data component names having a correlation relationship with the big data component name, so that a dependency relationship processing result is obtained. The application designs a database containing the corresponding relationship between a big data component and a dependent component. For a newly added component, a user only needs to add a component record of the database, so that the upgrading difficulty is reduced. The user can freely select a big data component combination, so that the flexibility of the big data component combination is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to an information processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a big data component. Background Technology

[0002] Currently, most cloud computing vendors implement big data component deployment solutions based on scenarios, such as data lakes, data services, and real-time data streams. In each scenario, the combination of big data components remains fixed. This approach has many drawbacks, specifically: (1) lack of flexibility, as the fixed combination of big data components in each scenario may not meet user needs; (2) cumbersome scenario deployment, as there are many types of big data components and a large number of combinations. If each scenario has a fixed combination of components, it means that different scenarios need to be deployed frequently, which is time-consuming and labor-intensive; (3) difficult to upgrade, if it is necessary to upgrade the version of a certain big data component, all scenarios containing this component need to be upgraded at the same time, resulting in a huge workload for upgrades. Summary of the Invention

[0003] Therefore, it is necessary to provide an information processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for big data components to address the aforementioned technical problems. This would reduce the difficulty of upgrading, enhance the scalability of big data components, and improve the flexibility of combining big data components.

[0004] Firstly, this application provides an information processing method for a big data component, including:

[0005] Read big data component records from the database and store them in the dependency list; the database contains multiple big data component records, each of which includes the big data component name and the dependent component name;

[0006] Traverse the dependency list and populate the data found into the dictionary data structure; the dictionary data structure is used to store the names of big data components and the set of big data component names that have a direct relationship with the big data component names; the direct relationship is either a direct dependency relationship or a direct dependency relationship;

[0007] Traverse the dictionary data structure and fill the traversed data into the overall mapping structure to obtain the dependency processing results; the overall mapping structure is used to store the names of big data components and the set of big data component names that are related to the big data component names; the relationship is a dependency relationship or a dependent relationship; the dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship.

[0008] In one embodiment, the method further includes:

[0009] Receive component configuration instructions; component configuration instructions are used to select or deselect the target big data component.

[0010] Based on the component configuration instructions, query the corresponding overall mapping structure to determine at least one dependent component related to the target big data component;

[0011] Perform appropriate configuration processing on the target big data component and at least one dependent component.

[0012] In one embodiment, traversing the dictionary data structure and filling the traversed data into the overall mapping structure includes:

[0013] Establish a relationship list, an access list, and a processing queue;

[0014] Traverse the dictionary data structure and store the name of the current big data component encountered in the traversal into the access list and processing queue;

[0015] Retrieve the current element from the processing queue;

[0016] Retrieve the set of big data component names corresponding to the current element from the dictionary data structure;

[0017] Iterate through the obtained set of big data component names;

[0018] If the name of the associated component encountered during the iteration is not in the access list, then add the name of the associated component encountered during the iteration to the relationship list, the access list, and the processing queue, respectively.

[0019] If the processing queue is found to be empty, the current big data component name is assigned to the current key of the overall mapping structure, and the relation list is assigned to the current value of the overall mapping structure.

[0020] In one embodiment, after iterating through the acquired set of big data component names, the method further includes:

[0021] If the name of the associated component encountered during the iteration is in the access list, then continue iterating through the obtained set of big data component names.

[0022] In one embodiment, the dependent linked list is traversed, and the traversed data is populated into a dictionary data structure, including:

[0023] Traverse the dependent linked list and retrieve the target name from the linked list objects that are traversed;

[0024] Check if a target key exists in the dictionary data structure that matches the target name;

[0025] If it exists, the attribute value of the linked list object will be stored in the value corresponding to the target key.

[0026] In one embodiment, the method further includes:

[0027] If it does not exist, create a new target object and store the attribute values ​​of the linked list object into the target object;

[0028] Assign the target name of the linked list object to the current key of the dictionary data structure, and assign the target object to the current value of the dictionary data structure.

[0029] Secondly, this application also provides an information processing apparatus for a big data component, comprising:

[0030] The read module is used to read big data component records from the database and store them in the dependency list; the database contains multiple big data component records, each of which includes the big data component name and the dependent component name;

[0031] The first traversal module is used to traverse the dependency list and fill the traversed data into the dictionary data structure. The dictionary data structure is used to store the names of big data components and the set of big data component names that have a direct relationship with the big data component names. The direct relationship is either a direct dependency relationship or a direct dependency relationship.

[0032] The second traversal module is used to traverse the dictionary data structure and fill the traversed data into the main mapping structure to obtain the dependency processing results. The main mapping structure is used to store the names of big data components and the set of big data component names that are related to the big data component names. The relationship is a dependency relationship or a dependent relationship. The dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0034] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0036] The aforementioned information processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for big data components read big data component records from a database and store them in a dependency list. The database contains multiple big data component records, each including a big data component name and dependent component names. The dependency list is traversed, and the traversed data is filled into a dictionary data structure. The dictionary data structure stores the big data component name and a set of big data component names directly related to the big data component name. Direct relationships are either direct dependencies or directly dependent relationships. The dictionary data structure is traversed, and the traversed data is filled into a general mapping structure to obtain the dependency processing result. The general mapping structure stores the big data component name and a set of big data component names related to the big data component name. Relationships are either dependencies or dependent relationships. Dependencies include direct dependencies and indirect dependencies, and dependent relationships include direct dependent relationships and indirect dependent relationships. By employing the above method, a database containing a one-to-one correspondence between big data components and their dependent components is designed. This facilitates the construction of a dictionary data structure through forward or backward traversal, allowing for the reuse of logic for component dependency operations and component-dependent operations, significantly reducing the complexity of the algorithm implementation. For new components, users only need to add component records to the database, without changing existing component records or operational logic, reducing upgrade difficulty and enhancing the scalability of big data components. The database can automatically generate big data component combinations, eliminating the need for fixed scenarios and allowing users to freely choose big data component combinations, thus improving the flexibility of big data component combinations. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating an information processing method for a big data component in one embodiment;

[0039] Figure 2 This is a flowchart illustrating the information processing method of a big data component in another embodiment;

[0040] Figure 3 This is a flowchart illustrating the information processing method of the big data component in yet another embodiment;

[0041] Figure 4 This is a structural block diagram of an information processing device for a big data component in one embodiment.

[0042] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] In one exemplary embodiment, such as Figure 1 As shown, an information processing method for a big data component is provided. This embodiment illustrates the method by applying it to a computer device. It is understood that this method can also be applied to a server, and further to a system including both a computer device and a server, and is implemented through the interaction between the computer device and the server. The method includes:

[0045] Step 102: Read the big data component records from the database and store them in the dependency list; the database contains multiple big data component records, each of which includes the big data component name and the dependent component name.

[0046] In this embodiment, the database stores big data components and their direct dependencies, i.e., it stores a one-to-one correspondence between components and their direct dependencies. This design ensures that the forward and reverse traversal logic is identical. Optionally, the database tables adopt the data table structure shown in Table 1 below. The big data component records stored in the database include the big data component name, dependent component name, creation time, and update time.

[0047] Table 1:

[0048]

[0049] For example, refer to Table 2, which is an example of the dependency data representation of any big data component in the database.

[0050] Table 2:

[0051]

[0052] Specifically, a new dependency list is created to store records of big data components. This involves creating a new database table field mapping object `componentDependency` and a new dependency list `dependencyList`. <componentdependency>Object. Store all records read from the database into a dependencyList data structure.

[0053] Step 104: Traverse the dependency list and fill the traversed data into the dictionary data structure; the dictionary data structure is used to store the names of big data components and the set of big data component names that have a direct relationship with the big data component names; the direct relationship is either a direct dependency relationship or a direct dependency relationship.

[0054] In this context, a direct dependency relationship refers to a situation where the operation of a subject directly relies on the support of an object, with no intermediary between them. A directly dependent relationship refers to a situation where the function of an object is directly depended upon by the subject. Optionally, a first traversal strategy is used to traverse the dependency list to generate a first dictionary data structure, which stores the names of big data components and the set of big data component names that have a direct dependency relationship with the big data component names. Optionally, a second traversal strategy is used to traverse the dependency list to generate a second dictionary data structure, which stores the names of big data components and the set of big data component names that have a direct dependency relationship with the big data component names. The first traversal strategy, for example, is based on the names of big data components in the big data component record; the second traversal strategy, for example, is based on the names of dependent components in the big data component record.

[0055] A dictionary data structure is a map data structure that stores and manipulates data in key-value pairs. Specifically, in the dictionary data structure corresponding to direct dependencies, the key stores the name of the big data component, and the value stores the set of big data component names that have a direct dependency relationship with the big data component name. Similarly, in the dictionary data structure corresponding to directly dependent relationships, the key stores the name of the big data component, and the value stores the set of big data component names that have a direct dependency relationship with the big data component name.

[0056] Step 106: Traverse the dictionary data structure and fill the traversed data into the overall mapping structure to obtain the dependency processing result; the overall mapping structure is used to store the big data component name and the set of big data component names that are related to the big data component name; the relationship is a dependency relationship or a dependent relationship; the dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship.

[0057] The overall mapping structure refers to the object that maps keys to values, i.e., the map data structure. For the dictionary data structure corresponding to direct dependencies, traversing this dictionary data structure allows us to exhaustively identify all big data components that the currently traversed big data component depends on from the first to the nth level, where n is an integer greater than 1. Similarly, for the dictionary data structure corresponding to directly dependent relationships, traversing this dictionary data structure allows us to determine all big data components that directly or indirectly depend on the currently traversed big data component, i.e., exhaustively identify all big data components from the first to the nth level that depend on the current big data component.

[0058] Understandably, based on the overall mapping structure corresponding to dependencies, it is possible to manage all sets of big data components that a big data component depends on, making it convenient for users to query and use combinations of big data components with dependencies. Similarly, based on the overall mapping structure corresponding to dependencies, it is possible to manage all sets of big data components that depend on a big data component, making it convenient for users to query and use combinations of big data components with dependencies.

[0059] In the above-mentioned information processing method for big data components, big data component records are read from the database and stored in a dependency list. The database contains multiple big data component records, each including the big data component name and the names of dependent components. The dependency list is traversed, and the traversed data is filled into a dictionary data structure. The dictionary data structure stores the big data component name and a set of big data component names directly related to the big data component name. Direct relationships are either direct dependencies or direct dependent relationships. The dictionary data structure is traversed, and the traversed data is filled into a general mapping structure to obtain the dependency processing result. The general mapping structure stores the big data component name and a set of big data component names related to the big data component name. Relationships are either dependencies or dependent relationships. Dependencies include direct dependencies and indirect dependencies, and dependent relationships include direct dependent relationships and indirect dependent relationships. Through this method, a database containing a one-to-one correspondence between big data components and their dependent components is designed, facilitating forward or backward traversal to construct the dictionary data structure. This allows for the reuse of logic for component dependency and dependent relationship operations, significantly reducing the algorithm's implementation complexity. For new components, users only need to add component records to the database, without changing existing component records and computational logic, reducing upgrade difficulty and enhancing the scalability of big data components. It can automatically generate big data component combinations based on the database, without being fixed according to specific scenarios, allowing users to freely choose big data component combinations and improving the flexibility of big data component combinations.

[0060] In one exemplary embodiment, such as Figure 2 As shown, the method further includes:

[0061] Step 202: Receive component configuration instructions; component configuration instructions are used to select or deselect the target big data component.

[0062] The computer device provides an input unit through which users input component configuration commands. Users can select or deselect any big data component using these commands; the big data component targeted by the configuration command is the target big data component.

[0063] Step 204: Based on the component configuration instructions, query the corresponding overall mapping structure to determine at least one dependent component related to the target big data component.

[0064] Specifically, if the component configuration directive is used to select a target big data component, the overall mapping structure corresponding to the dependency relationship is queried to determine at least one dependent component that has a direct or indirect dependency relationship with the target big data component. If the component configuration directive is used to cancel a target big data component, the overall mapping structure corresponding to the dependency relationship is queried to determine at least one dependent component that has a direct or indirect dependency relationship with the target big data component.

[0065] Step 206: Perform corresponding configuration processing on the target big data component and at least one dependent component.

[0066] Specifically, when a user selects any big data component, the computer device automatically completes the list of other components that the big data component depends on based on the dependency processing results. Alternatively, when a user cancels any big data component, the computer device automatically removes other components that depend on that big data component based on the dependency processing results. This allows users to freely choose combinations of big data components according to their needs, ensuring successful deployment of these combinations and enhancing system stability and reliability.

[0067] In an exemplary embodiment, step 106 includes: establishing a relationship list, an access list, and a processing queue; traversing the dictionary data structure and storing the current big data component name encountered during traversal into the access list and the processing queue; retrieving the current element from the processing queue; retrieving the set of big data component names corresponding to the current element from the dictionary data structure; traversing the retrieved set of big data component names; if the traversed associated component name is not in the access list, adding the traversed associated component name to the relationship list, access list, and processing queue respectively; if the processing queue is detected to be empty, assigning the current big data component name to the current key of the overall mapping structure and assigning the relationship list to the current value of the overall mapping structure.

[0068] Taking the dictionary data structure corresponding to direct dependencies as an example, the dictionary data structure obtained in step 104 includes a set of direct dependency component names corresponding to each big data component. The big data components in these direct dependency name sets directly depend on other big data components. Step 106 is to exhaustively traverse all big data components that the big data component depends on from the first layer to the nth layer.

[0069] Create a new Map <String, Set <string>The totalMapping structure stores the names of big data components and all other big data components that the corresponding component depends on from the first to the nth layer. The dependencyMap dictionary structure is traversed using the following logic:

[0070] 1. Create a new relationship list (allDependencies), a visit list (visited), and a processing queue (queue); create a new Set for each element of the dependencyMap. <string>The data structures allDependencies object, visited object, and Queue <string>A queue object in the data structure. The relationship list allDependencies stores all the big data components that the key of the currently traversed dependencyMap depends on from the first to the nth level; the visited list records visited nodes to avoid repeated traversal and circular dependencies; the queue stores unvisited nodes.

[0071] 2. Initialize the access list visited and the processing queue: Store the key values ​​(big data component names) of the traversed dependencyMap elements into queue and visited, completing the initialization of queue and visited.

[0072] 3. If the queue is not empty, execute the tracing loop. The logic of each loop is as follows: (1) Pop an element from the head of the queue, which is the name of the big data component; (2) According to the retrieved big data component name, retrieve the set of direct dependent component names corresponding to the big data component name from the dependencyMap; (3) If the retrieved set of direct dependent component names is not empty, iterate through the set of direct dependent component names. The iteration logic is: check whether the element (the associated component name that has been iterated through) is in the visited list. If it is not in the visited list, add the element to the processing queue, add the element to the visited list, and add the element to the relationship list allDependencies.

[0073] 4. After the loop finishes execution, assign the names of the large data components (key values ​​of the dependencyMap elements) to the key of totalMapping, and assign the list of relationships allDependencies to the value of totalMapping.

[0074] 5. After traversing the dependencyMap, the final totalMapping is generated, which is used to store the names of big data components and all big data components that depend on it from the first layer to the nth layer.

[0075] By executing the above traversal logic on the dependentMap dictionary data structure corresponding to the direct dependencies, all big data components that directly or indirectly depend on the major data components can be obtained.

[0076] In this embodiment, when traversing the dictionary data structure, a design is adopted to temporarily store the traversed components, which can avoid repeated traversal of the traversed components and improve the execution efficiency of the algorithm.

[0077] In an exemplary embodiment, after traversing the obtained set of big data component names, the method further includes: if the traversed associated component name is in the access list, then continue to traverse the obtained set of big data component names.

[0078] If the name of the associated component being iterated over is in the access list, then the process continues to iterate over the next element in the big data component name set.

[0079] In an exemplary embodiment, step 104 includes: traversing the dependent linked list and obtaining the target name from the traversed linked list objects; detecting whether there is a target key in the dictionary data structure that matches the target name; if there is, storing the attribute value of the linked list object into the value corresponding to the target key.

[0080] This example uses a dictionary data structure corresponding to direct dependencies. A new dictionary data structure, dependencyMap, is created: Map <String, Set <string>An object of type `<dependencyMap>`. The key of `dependencyMap` is the name of the big data component, and the value is the collection of names of the directly dependent components corresponding to that big data component. It iterates through each element `componentDependency` (i.e., the linked list object) in the dependency list `dependencyList`, with the following logic: First, check if the `name` attribute value of the current `componentDependency` exists in the key of `dependencyMap`. If it exists, store the `dependency` attribute value of `componentDependency` into the target object `collectionDependency` corresponding to the value of the dictionary data structure `dependencyMap`. Finally, assign the target object `collectionDependency` to the value of `dependencyMap`.

[0081] Similarly, create a new dictionary data structure, dependedMap, corresponding to the directly dependent relationships: Map <String,Set <string>An object of type `<dependency>`. The key of `dependedMap` is the name of the big data component that is depended upon, and the value is the collection of names that depend on that big data component. It iterates through each element `componentDependency` (i.e., the linked list object) in the dependency list `dependencyList`, with the following logic: First, check if the `dependency` attribute value of the current `componentDependency` exists in the key of `dependedMap`. If it exists, store the `name` attribute value of `componentDependency` into the target object `collectionDepended` corresponding to the value of `dependedMap`, and finally assign the target object `collectionDepended` to the value of `dependedMap`.

[0082] In an exemplary embodiment, the method further includes: if the target object does not exist, creating a new target object and storing the attribute values ​​of the linked list object into the target object; assigning the target name of the linked list object to the current key of the dictionary data structure, and assigning the target object to the current value of the dictionary data structure.

[0083] Taking the dictionary data structure corresponding to direct dependencies as an example, if the name attribute value of the current componentDependency does not exist in the key of the dependencyMap, a new Set is created. <string>The target object `collectionDependency` of type `componentDependency` is stored in the `collectionDependency` collection. Finally, the `name` attribute value of `componentDependency` is assigned to the key of the `dependencyMap`, and the `collectionDependency` is assigned to the value of the `dependencyMap`. The target object `collectionDependency` can be of type `HashSet` to store unique data.

[0084] Finally, after traversing all elements of the dependency list, a dictionary data structure is obtained. The dictionary data structure stores the names of all big data components, as well as the set of big data component names that have direct dependencies on the names of the big data components.

[0085] Similarly, taking the dictionary data structure corresponding to the direct dependency relationship as an example, if the dependency attribute value of the current componentDependency does not exist in the key of the dependedMap, a new Set is created. <string>The name attribute value of the componentDependency is stored in the collectionDepended. Finally, the dependency attribute value of the componentDependency is assigned to the key of the dependedMap, and the collectionDepended is assigned to the value of the dependedMap. The collectionDepended can be of the hashset type, and is used to store non-repeated data.

[0086] Finally, all elements of the dependencyList are traversed to obtain a dictionary data structure, which stores all big data component names and sets of big data component names having a direct dependency relationship with each big data component name.

[0087] In an optional implementation, taking processing of the dependency relationship of a big data component as an example, the information processing method of the big data component includes the following steps.

[0088] 1. All big data component dependency relationship data in a database is read and stored in the dependencyList.

[0089] 2. A dictionary data structure dependencyMap is created.

[0090] 3. The dependencyList is traversed. If the name attribute value of the current list object does not exist in the key of the dictionary data structure, a target object is created, the dependency attribute value of the current list object is stored in the target object, the name attribute value of the current list is assigned to the key of the dictionary data structure, and the target object is assigned to the value of the dictionary data structure. If the name attribute value of the current list object exists in the key of the dictionary data structure, the dependency attribute value of the current list object is stored in the target object corresponding to the value of the dictionary data structure.

[0091] 4. A total mapping structure totalMapping is newly created, and the dictionary data structure dependencyMap is traversed.

[0092] 5. A relationship list, an access list, and a processing queue are newly created.

[0093] 6. The access list and the processing queue are initialized.

[0094] 7. It is determined whether the processing queue is empty. If the processing queue is empty, the key of the traversed dictionary data structure and the relationship list are stored in the total mapping structure.

[0095] 8. If the processing queue is not empty, the head element of the processing queue is dequeued, and a set of large data component names is fetched from the dictionary data structure according to the head element.

[0096] 9. It is judged whether the fetched set of large data component names is non-empty. If the fetched set of large data component names is non-empty, the fetched set of large data component names is traversed.

[0097] 10. It is checked whether the traversed element exists in the access list. If not, the element is added to the relation list, the element is added to the access list, and the element is added to the processing queue. If yes, the traversal of the fetched set of large data component names is continued.

[0098] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiments of the present application also provide an information processing device of a large data component for implementing the information processing method of the large data component as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more information processing device embodiments of the large data component provided below can refer to the limitations of the information processing method of the large data component described above, which will not be described here again.

[0100] In one exemplary embodiment, as shown in Figure 4 there is provided an information processing device of a large data component, comprising:

[0101] The reading module 402 is configured to read large data component records from a database and store the large data component records in a dependency chain list. The database stores a plurality of large data component records, and each large data component record includes a large data component name and a dependent component name.

[0102] The first traversal module 404 is configured to traverse the dependency chain table and fill the traversed data into a dictionary data structure; the dictionary data structure is used to store a big data component name and a set of big data component names having a direct correlation relationship with the big data component name; the direct correlation relationship is a direct dependency relationship or a direct dependency relationship.

[0103] The second traversal module 406 is configured to traverse the dictionary data structure and fill the traversed data into a total mapping structure to obtain a dependency relationship processing result; the total mapping structure is used to store a big data component name and a set of big data component names having a correlation relationship with the big data component name; the correlation relationship is a dependency relationship or a dependency relationship; the dependency relationship includes a direct dependency relationship and an indirect dependency relationship, and the dependency relationship includes a direct dependency relationship and an indirect dependency relationship.

[0104] In the information processing apparatus of the big data component described above, the big data component record is read from the database and stored in the dependency chain table; the database stores a plurality of big data component records, and each big data component record includes a big data component name and a dependency component name; the dependency chain table is traversed, and the traversed data is filled into a dictionary data structure; the dictionary data structure is used to store a big data component name and a set of big data component names having a direct correlation relationship with the big data component name; the direct correlation relationship is a direct dependency relationship or a direct dependency relationship; the dictionary data structure is traversed, and the traversed data is filled into a total mapping structure to obtain a dependency relationship processing result; the total mapping structure is used to store a big data component name and a set of big data component names having a correlation relationship with the big data component name; the correlation relationship is a dependency relationship or a dependency relationship; the dependency relationship includes a direct dependency relationship and an indirect dependency relationship, and the dependency relationship includes a direct dependency relationship and an indirect dependency relationship. Through the above manner, the database containing the one-to-one correspondence relationship of the big data component and the dependency component is designed, which facilitates the forward traversal or reverse traversal to construct the dictionary data structure, so that the logic of the component dependency relationship operation and the component dependency relationship operation can be reused, and the implementation complexity of the algorithm is greatly reduced. For a newly added component, the user only needs to add the component record of the database, without changing the existing component record and operation logic, thereby reducing the upgrade difficulty and enhancing the expansibility of the big data component. The big data component combination can be automatically generated based on the database without being fixed according to the scene, so that the user can freely select the big data component combination, and the flexibility of the big data component combination is improved.

[0105] In an example embodiment, the information processing apparatus of the big data component further comprises an instruction processing module, configured to receive a component configuration instruction; the component configuration instruction is used to select or cancel a target big data component; according to the component configuration instruction, query the corresponding total mapping structure, determine at least one dependent component related to the target big data component; and perform corresponding configuration processing on the target big data component and the at least one dependent component.

[0106] In an example embodiment, the second traversal module 406 is further configured to establish a relationship list, an access list and a processing queue; traverse the dictionary data structure, and store the current big data component name obtained in the access list and the processing queue; obtain a current element from the processing queue; obtain a set of big data component names corresponding to the current element from the dictionary data structure; traverse the obtained set of big data component names; if the traversed associated component name is not in the access list, add the traversed associated component name to the relationship list, the access list and the processing queue respectively; and in the case of detecting that the processing queue is empty, assign the current big data component name to the current key of the total mapping structure, and assign the relationship list to the current value of the total mapping structure.

[0107] In an example embodiment, the second traversal module 406 is further configured to, if the traversed associated component name is in the access list, continue to traverse the obtained set of big data component names.

[0108] In an example embodiment, the first traversal module 404 is further configured to traverse the dependent chain table, and obtain a target name from the traversed chain table object; detect whether a target key matching the target name exists in the dictionary data structure; if it exists, store the attribute value of the chain table object in the value corresponding to the target key.

[0109] In an example embodiment, the first traversal module 404 is further configured to, if it does not exist, create a target object, store the attribute value of the chain table object in the target object; assign the target name of the chain table object to the current key of the dictionary data structure, and assign the target object to the current value of the dictionary data structure.

[0110] The above-mentioned various modules in the information processing apparatus of the big data component can be realized by software, hardware and their combinations in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0111] In an example embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 5 As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the big data component record. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a big data component information processing method.

[0112] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0113] In one exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps: reading a big data component record from a database and storing it in a dependency chain table; the database stores a plurality of big data component records, each big data component record comprising a big data component name and a dependent component name; traversing the dependency chain table and filling the traversed data into a dictionary data structure; the dictionary data structure is used to store a big data component name and a set of big data component names having a direct association relationship with the big data component name; the direct association relationship is a direct dependency relationship or a direct dependency relationship; traversing the dictionary data structure and filling the traversed data into a total mapping structure to obtain a dependency relationship processing result; the total mapping structure is used to store a big data component name and a set of big data component names having an association relationship with the big data component name; the association relationship is a dependency relationship or a dependent relationship; the dependency relationship includes a direct dependency relationship and an indirect dependency relationship; the dependent relationship includes a direct dependent relationship and an indirect dependent relationship.

[0114] In one embodiment, the processor, when executing the computer program, further implements the following steps: receiving a component configuration instruction; the component configuration instruction is used to select or cancel a target big data component; according to the component configuration instruction, querying a corresponding total mapping structure to determine at least one dependent component related to the target big data component; and performing corresponding configuration processing on the target big data component and the at least one dependent component.

[0115] In one embodiment, the processor, when executing the computer program, further implements the following steps: establishing a relationship list, an access list, and a processing queue; traversing a dictionary data structure, and storing a current big data component name obtained by the traversal into the access list and the processing queue; obtaining a current element from the processing queue; obtaining a set of big data component names corresponding to the current element from the dictionary data structure; traversing the obtained set of big data component names; if a traversed associated component name is not in the access list, then adding the traversed associated component name into the relationship list, the access list, and the processing queue, respectively; and in a case where it is detected that the processing queue is empty, assigning the current big data component name to a current key of the total mapping structure, and assigning the relationship list to a current value of the total mapping structure.

[0116] In one embodiment, the processor, when executing the computer program, further implements the following steps: if the traversed associated component name is in the access list, then continuing to traverse the obtained set of big data component names.

[0117] In one embodiment, the processor, when executing the computer program, further implements the following steps: traversing a dependency chain table, and obtaining a target name from a traversed table object; detecting whether a target key matching the target name exists in the dictionary data structure; if the target key exists, then storing an attribute value of the table object into a value corresponding to the target key.

[0118] In one embodiment, the processor, when executing the computer program, further implements the following steps: if the target key does not exist, then creating a target object, and storing the attribute value of the table object into the target object; assigning the target name of the table object to a current key of the dictionary data structure, and assigning the target object to a current value of the dictionary data structure.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: reading big data component records from a database and storing them in a dependency list; the database stores multiple big data component records, each big data component record including a big data component name and a dependent component name; traversing the dependency list and filling the traversed data into a dictionary data structure; the dictionary data structure is used to store the big data component name and a set of big data component names that have a direct relationship with the big data component name; the direct relationship is a direct dependency relationship or a direct dependent relationship; traversing the dictionary data structure and filling the traversed data into a general mapping structure to obtain the dependency processing result; the general mapping structure is used to store the big data component name and a set of big data component names that have a relationship with the big data component name; the relationship is a dependency relationship or a dependent relationship; the dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship.

[0120] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving a component configuration instruction; the component configuration instruction is used to select or deselect a target big data component; according to the component configuration instruction, querying the corresponding overall mapping structure to determine at least one dependent component related to the target big data component; and performing corresponding configuration processing on the target big data component and at least one dependent component.

[0121] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: establishing a relation list, an access list, and a processing queue; traversing the dictionary data structure and storing the current big data component name encountered in the traversal into the access list and the processing queue; retrieving the current element from the processing queue; retrieving the set of big data component names corresponding to the current element from the dictionary data structure; traversing the retrieved set of big data component names; if the traversed associated component name is not in the access list, adding the traversed associated component name to the relation list, access list, and processing queue respectively; if the processing queue is detected to be empty, assigning the current big data component name to the current key of the overall mapping structure and assigning the relation list to the current value of the overall mapping structure.

[0122] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: if the name of the associated component being traversed is in the access list, then continue traversing the obtained set of big data component names.

[0123] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: traversing the dependent linked list and obtaining the target name from the traversed linked list objects; detecting whether there is a target key in the dictionary data structure that matches the target name; if so, storing the attribute value of the linked list object into the value corresponding to the target key.

[0124] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the target object does not exist, a new target object is created, and the attribute values ​​of the linked list object are stored in the target object; the target name of the linked list object is assigned to the current key of the dictionary data structure, and the target object is assigned to the current value of the dictionary data structure.

[0125] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: reading big data component records from a database and storing them in a dependency list; the database stores multiple big data component records, each including a big data component name and a dependent component name; traversing the dependency list and filling the traversed data into a dictionary data structure; the dictionary data structure stores the big data component name and a set of big data component names that have a direct relationship with the big data component name; the direct relationship is either a direct dependency or a direct dependent relationship; traversing the dictionary data structure and filling the traversed data into a general mapping structure to obtain the dependency processing result; the general mapping structure stores the big data component name and a set of big data component names that have a relationship with the big data component name; the relationship is either a dependency or a dependent relationship; the dependency includes direct dependencies and indirect dependencies, and the dependent relationship includes direct dependent relationships and indirect dependent relationships.

[0126] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving a component configuration instruction; the component configuration instruction is used to select or deselect a target big data component; according to the component configuration instruction, querying the corresponding overall mapping structure to determine at least one dependent component related to the target big data component; and performing corresponding configuration processing on the target big data component and at least one dependent component.

[0127] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: establishing a relation list, an access list, and a processing queue; traversing the dictionary data structure and storing the current big data component name encountered in the traversal into the access list and the processing queue; retrieving the current element from the processing queue; retrieving the set of big data component names corresponding to the current element from the dictionary data structure; traversing the retrieved set of big data component names; if the traversed associated component name is not in the access list, adding the traversed associated component name to the relation list, access list, and processing queue respectively; if the processing queue is detected to be empty, assigning the current big data component name to the current key of the overall mapping structure and assigning the relation list to the current value of the overall mapping structure.

[0128] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: if the name of the associated component being traversed is in the access list, then continue traversing the obtained set of big data component names.

[0129] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: traversing the dependent linked list and obtaining the target name from the traversed linked list objects; detecting whether there is a target key in the dictionary data structure that matches the target name; if so, storing the attribute value of the linked list object into the value corresponding to the target key.

[0130] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the target object does not exist, a new target object is created, and the attribute values ​​of the linked list object are stored in the target object; the target name of the linked list object is assigned to the current key of the dictionary data structure, and the target object is assigned to the current value of the dictionary data structure.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0132] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The memory, database, or other media mentioned in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / string> < / string> < / string> < / string> < / string> < / string> < / string> < / componentdependency>

Claims

1. An information processing method for a big data component, characterized in that, The method includes: Read big data component records from the database and store them in the dependency list; the database contains multiple big data component records, each of which includes the big data component name and the dependent component name; Traverse the dependency list and populate the traversed data into a dictionary data structure; the dictionary data structure is used to store the names of big data components and a set of big data component names that have a direct relationship with the names of the big data components; the direct relationship is a direct dependency relationship or a direct dependency relationship; The dictionary data structure is traversed, and the traversed data is filled into the overall mapping structure to obtain the dependency processing result; the overall mapping structure is used to store the big data component name and the set of big data component names that are associated with the big data component name; the association relationship is a dependency relationship or a dependent relationship; the dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship; The step of traversing the dictionary data structure and filling the traversed data into the overall mapping structure includes: Establish a relationship list, an access list, and a processing queue; Traverse the dictionary data structure and store the name of the current big data component encountered in the traversal into the access list and the processing queue; Retrieve the current element from the processing queue; From the dictionary data structure, obtain the set of big data component names corresponding to the current element; Iterate through the obtained set of big data component names; If the name of the associated component encountered during the traversal is not in the access list, then the name of the associated component encountered during the traversal is added to the relationship list, the access list, and the processing queue, respectively. If the processing queue is detected to be empty, the current big data component name is assigned to the current key of the overall mapping structure, and the relationship list is assigned to the current value of the overall mapping structure.

2. The method according to claim 1, characterized in that, The method further includes: Receive component configuration instructions; the component configuration instructions are used to select or deselect the target big data component; Based on the component configuration instructions, query the corresponding overall mapping structure to determine at least one dependent component related to the target big data component; Perform corresponding configuration processing on the target big data component and the at least one dependent component.

3. The method according to claim 1, characterized in that, After iterating through the obtained set of big data component names, the method further includes: If the name of the associated component encountered during the traversal is in the access list, then continue traversing the obtained set of big data component names.

4. The method according to any one of claims 1 to 3, characterized in that, The step of traversing the dependency list and filling the traversed data into the dictionary data structure includes: Traverse the dependent linked list and obtain the target name from the linked list objects that are traversed; Detect whether a target key matching the target name exists in the dictionary data structure; If it exists, the attribute value of the linked list object is stored in the value corresponding to the target key.

5. The method according to claim 4, characterized in that, The method further includes: If it does not exist, create a new target object and store the attribute values ​​of the linked list object into the target object; Assign the target name of the linked list object to the current key of the dictionary data structure, and assign the target object to the current value of the dictionary data structure.

6. An information processing device for a big data component, characterized in that, The device includes: The reading module is used to read big data component records from the database and store them in the dependency list; the database contains multiple big data component records, each of which includes the big data component name and the dependent component name; The first traversal module is used to traverse the dependency list and fill the traversed data into a dictionary data structure; the dictionary data structure is used to store the names of big data components and a set of big data component names that have a direct relationship with the names of big data components; the direct relationship is a direct dependency relationship or a direct dependency relationship; The second traversal module is used to traverse the dictionary data structure, fill the traversed data into the overall mapping structure, and obtain the dependency processing result; the overall mapping structure is used to store the big data component names and the set of big data component names that are related to the big data component names; the relationship is a dependency relationship or a dependent relationship; the dependency relationship includes direct dependency relationship and indirect dependency relationship, and the dependent relationship includes direct dependent relationship and indirect dependent relationship; The second traversal module is further configured to: establish a relationship list, an access list, and a processing queue; traverse the dictionary data structure, storing the current big data component name found during traversal into the access list and the processing queue; retrieve the current element from the processing queue; retrieve the set of big data component names corresponding to the current element from the dictionary data structure; traverse the retrieved set of big data component names; if the traversed associated component name is not in the access list, add the traversed associated component name to the relationship list, the access list, and the processing queue respectively; if the processing queue is detected to be empty, assign the current big data component name to the current key of the overall mapping structure, and assign the relationship list to the current value of the overall mapping structure.

7. The apparatus according to claim 6, characterized in that, The information processing device for the big data component further includes an instruction processing module, which is used to receive component configuration instructions; the component configuration instructions are used to select or cancel a target big data component; according to the component configuration instructions, the corresponding overall mapping structure is queried to determine at least one dependent component related to the target big data component; and corresponding configuration processing is performed on the target big data component and the at least one dependent component.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images