Data aggregation method and apparatus, electronic device, and storage medium

Abstract

The application provides a data aggregation method and device, electronic equipment and storage medium. The data aggregation method comprises the following steps: receiving an aggregation logic code, the aggregation logic code is written based on a data aggregation requirement, and the code language used by the aggregation logic code is different from that used by a data aggregation tool; translating the aggregation logic code into a translated aggregation code consistent with the code language used by the data aggregation tool through an interpreter; and injecting the translated aggregation code into a data aggregation framework in the data aggregation tool, so that the data aggregation tool after injecting the translated aggregation code can perform data aggregation according to the data aggregation requirement. In this way, the code language used by the data aggregation tool is not limited, more personnel can participate in the development of the data aggregation requirement in the data aggregation platform, a large number of data aggregation requirements can be developed in a short time, and the efficiency of the data aggregation platform for a large number of data aggregation requirements is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data aggregation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advent of the big data era, much data needs to be aggregated before being displayed. For example, data generated by business modules needs to be aggregated in a data aggregation platform before being displayed on a large screen.

[0003] Currently, data aggregation platforms primarily utilize Business Intelligence (BI) tools to achieve multi-data aggregation. When a data aggregation requirement arises, developers need to develop code within the BI tool to meet that requirement. Since BI tools aggregate data using JSON functions in the PG database, developers must write Java code into the BI tool to ensure that, after execution, it can aggregate relevant data generated by business modules according to the aggregation requirements.

[0004] However, writing Java code into BI tools requires a high level of expertise; only those with specialized software development skills can perform this task. Meanwhile, the increasing demand for data aggregation means that a limited number of developers cannot handle the large volume of data aggregation requests within a short period, leading to inefficiency for data aggregation platforms when dealing with large datasets. Summary of the Invention

[0005] The purpose of this application is to provide a data aggregation method, apparatus, electronic device, and storage medium to improve data aggregation efficiency.

[0006] To address the aforementioned technical problems, this application provides the following technical solutions:

[0007] The first aspect of this application provides a data aggregation method, the method comprising: receiving aggregation logic code, the aggregation logic code being written based on data aggregation requirements and differing from the code language used by a data aggregation tool; translating the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool using an interpreter; and injecting the translated aggregation code into a data aggregation framework within the data aggregation tool, so that the data aggregation tool, after being injected with the translated aggregation code, can perform data aggregation according to the data aggregation requirements.

[0008] A second aspect of this application provides a data aggregation apparatus, comprising: a receiving module for receiving aggregation logic code, the aggregation logic code being written based on data aggregation requirements and differing from the code language used by the data aggregation tool; a translation module for translating the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool via an interpreter; and an injection aggregation module for injecting the translated aggregation code into the data aggregation framework of the data aggregation tool, so that the data aggregation tool injected with the translated aggregation code can perform data aggregation according to the data aggregation requirements after running.

[0009] A third aspect of this application provides an electronic device, the electronic device comprising: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the method of the first aspect.

[0010] A fourth aspect of this application provides a computer-readable storage medium comprising: a stored program; wherein, when the program is executed, the device on which the storage medium is located executes the method of the first aspect.

[0011] Compared to existing technologies, the data aggregation method provided in the first aspect of this application involves a data aggregation platform receiving aggregation logic code written based on data aggregation requirements in a different code language than that used by the data aggregation tool. The platform then uses an interpreter to translate the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool. This translated aggregation code is then injected into the data aggregation framework within the data aggregation tool, enabling the tool to perform data aggregation according to the data aggregation requirements after execution. This allows for development within the data aggregation tool based on data aggregation requirements, freeing users from the constraints of using the tool's own code language. Simpler, easier-to-learn code languages ​​can also be used to implement data aggregation requirements within the tool. This allows more personnel to participate in the development of data aggregation requirements within the data aggregation platform, enabling the development of numerous data aggregation requirements to be completed quickly through the sharing of workload among more people, thus improving the efficiency of the data aggregation platform in handling large volumes of data aggregation needs.

[0012] The data aggregation apparatus provided in the second aspect of this application, the electronic device provided in the third aspect, and the computer-readable storage medium provided in the fourth aspect have the same or similar beneficial effects as the data aggregation method provided in the first aspect. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0014] Figure 1 This is a schematic diagram of the scenario architecture for the data aggregation method in the embodiments of this application. Figure 1 ;

[0015] Figure 2 This is a flowchart illustrating the data aggregation method in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram illustrating the process by which the data aggregation tool in this application processes the aggregation logic code. Figure 1 ;

[0017] Figure 4 This is a schematic diagram of the data aggregation framework architecture in the embodiments of this application. Figure 1 ;

[0018] Figure 5 This is a schematic diagram of the scenario architecture for the data aggregation method in the embodiments of this application. Figure 2 ;

[0019] Figure 6 This is a schematic diagram illustrating the process by which the data aggregation tool in this application processes the aggregation logic code. Figure 2 ;

[0020] Figure 7 This is a schematic diagram of the data aggregation framework architecture in the embodiments of this application. Figure 2 ;

[0021] Figure 8 This is a schematic diagram of the data aggregation device in the embodiments of this application;

[0022] Figure 9 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0023] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0024] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0025] Currently, data aggregation platforms fulfill certain data aggregation requirements through BI tools. This requires developers to use Java code within the BI tool, demanding a high level of expertise; only those with specialized software development skills can accomplish this. However, the number of data aggregation requirements is constantly increasing, and the limited number of developers cannot meet the large volume of demands in a short time. This leads to low execution efficiency for data aggregation platforms when dealing with a large number of data aggregation requests.

[0026] The inventors discovered that the main reason for the low efficiency of data aggregation platforms in handling large volumes of data aggregation requests is that requirement development within BI tools requires the use of Java code, which is typically used only by software development professionals. If requirement development within BI tools were no longer limited to Java code, but instead utilized other codebases such as Python scripts or C#, then non-software development personnel, such as operations and management staff, could also develop based on data aggregation requirements within the BI tool. This would allow a wider range of personnel to share the workload of large-scale data aggregation requests, enabling faster completion and improving the platform's efficiency in handling such requests.

[0027] In view of this, embodiments of this application provide a data aggregation method, apparatus, electronic device, and storage medium. Any person can edit aggregation logic code using a different coding language than the data aggregation tools in the data aggregation platform, based on data acquisition needs. The data aggregation platform then uses an interpreter to translate this aggregation logic code into translated aggregation code using the same coding language as the data aggregation tools. This translated aggregation code is then injected into the data aggregation framework of the data aggregation tools. After running, the data aggregation tools can aggregate relevant data according to the data aggregation requirements. Thus, when developing data aggregation tools based on data aggregation needs, it is not limited to using high-level code like Java; simple and easy-to-learn code can also be used to implement data aggregation requirements within the data aggregation tools. More people can participate in the development of data aggregation requirements within the data aggregation platform, enabling the development of a large number of data aggregation requirements to be completed in a short time through the sharing of tasks by more people, thereby improving the efficiency of the data aggregation platform in aggregating large amounts of data.

[0028] To more clearly explain the data aggregation method provided in the embodiments of this application, the application scenario of the data aggregation method provided in the embodiments of this application will first be explained.

[0029] Figure 1 This is a schematic diagram of the scenario architecture for the data aggregation method in the embodiments of this application. Figure 1See Figure 1 As shown, the architecture can include: business modules, a data aggregation platform, and a large screen.

[0030] The business module here refers to the module that generates the data to be aggregated. In practical applications, this can refer to one or more platforms, web pages, or applications (APPs). The specific type of business module is not limited here. After generating the data to be aggregated, the business module sends it to the data aggregation platform.

[0031] The data aggregation platform includes data aggregation tools. The data aggregation method in this embodiment is implemented within these tools. The data aggregation platform receives data to be aggregated. The data aggregation tool within the platform has already been developed according to the data aggregation method in this embodiment, fulfilling the data aggregation requirements. After running, it can directly aggregate the data to be aggregated, obtaining aggregated data, which is then sent to the large screen.

[0032] Once the large screen receives the aggregated data, it can display it to allow users to intuitively view the data produced by the business modules.

[0033] Next, the data aggregation method provided in the embodiments of this application will be described in detail.

[0034] Figure 2 This is a flowchart illustrating the data aggregation method in an embodiment of this application. See also... Figure 2 As shown, the method may include:

[0035] S201: Receive aggregation logic code.

[0036] The aggregation logic code is written based on data aggregation requirements and uses a different code language than the data aggregation tools.

[0037] The data aggregation requirement here can refer to which data needs to be aggregated according to what logic, for example, it is necessary to count the daily page views of clothing, food and electronic products displayed in shopping software A.

[0038] When a data aggregation requirement arises, relevant personnel can write aggregation logic code based on that requirement using any coding language (if the same coding language as the data aggregation tool is used, it is existing technology; if a different coding language is used, it is the data aggregation method described in the embodiments of this application), and send the aggregation logic code to the data aggregation platform. The data aggregation platform can then receive the aggregation logic code written based on the data aggregation requirement, thereby enabling the aggregation logic code to implement the data aggregation requirement in the data aggregation tool.

[0039] S202: Translate the aggregation logic code into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter.

[0040] Because the aggregation logic code received by the data aggregation tool is in a different language than the code used by the tool itself, directly injecting the aggregation logic code into the tool will prevent it from recognizing it and thus from performing data aggregation according to the requirements. Therefore, the aggregation logic code needs to be translated into the code language used by the data aggregation tool, i.e., translated aggregation code. The tool used for code translation is an interpreter.

[0041] For example, the aggregation logic code might be a Python script, while the data aggregation platform uses Java. Therefore, the Python script needs to be input into an interpreter, and the interpreter will output Java code. The aggregation logic in the Python script and the Java code is the same; only the programming language is different.

[0042] S203: Inject the translated aggregation code into the data aggregation framework of the data aggregation tool so that the data aggregation tool with the injected translated aggregation code can perform data aggregation according to the data aggregation requirements after it runs.

[0043] The aggregation tool includes a pre-built data aggregation framework by developers. This framework contains pre-written general code capable of various data aggregations; only the specific aggregation logic is left blank for the translated logic code, edited by relevant personnel, to be filled in. In this way, the data aggregation platform can fulfill a variety of data aggregation needs through the data aggregation framework.

[0044] For example, a data aggregation framework X is pre-built in the data aggregation tool of the data aggregation platform. When storing data aggregation requirement A, person 1 edits aggregation logic code a based on data aggregation requirement A. The interpreter translates aggregation logic code a into translated aggregation code a1, and the data aggregation tool injects the translated aggregation code a1 into the data aggregation framework X. When storing data aggregation requirement B, person 2 edits aggregation logic code b based on data aggregation requirement B. The interpreter translates aggregation logic code b into translated aggregation code b1, and the data aggregation tool injects the translated aggregation code b1 into the data aggregation framework X. In this way, the data aggregation tool can realize data aggregation requirement 1 and data aggregation requirement 2 through the data aggregation framework X.

[0045] Once the translated aggregation code is injected into the data aggregation framework, the data aggregation platform runs based on the injected translated aggregation code, enabling it to aggregate relevant data according to data aggregation requirements.

[0046] As described above, the data aggregation method provided in this application involves a data aggregation platform receiving aggregation logic code written based on data aggregation requirements in a different code language than that used by the data aggregation tool. The platform then uses an interpreter to translate the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool. This translated aggregation code is then injected into the data aggregation framework within the data aggregation tool, enabling the tool to perform data aggregation according to the data aggregation requirements after the code is injected. This allows for development within the data aggregation tool based on data aggregation requirements, freeing users from the constraints of using the tool's own code language. Simpler, easier-to-learn code languages ​​can also be used to implement data aggregation requirements within the tool. This allows more personnel to participate in the development of data aggregation requirements within the data aggregation platform, enabling the development of numerous data aggregation requirements to be completed quickly through the sharing of workload among more personnel, thus improving the efficiency of the data aggregation platform in handling large volumes of data aggregation needs.

[0047] Furthermore, as an extension of the above steps S202 and S203, since the aggregation logic code is not developed directly in the data aggregation tool, there may be security risks. Therefore, when the aggregation logic code is actually running in the data aggregation tool, the execution permissions of the environment in which the aggregation logic code runs need to be restricted.

[0048] Specifically, prior to step S202 above, the method may include:

[0049] Step A1: Store the aggregation logic code in the first container.

[0050] After receiving the aggregation logic code, the data aggregation tool may not need to execute the aggregation logic code immediately. It can first store the aggregation logic code in the first container.

[0051] The first container here can be any container in the data aggregation tool, such as a Java container. The specific type of the first container is not limited here.

[0052] Step A2: In response to the code execution instruction, move the aggregate logic code from the first container to the second container.

[0053] The second container has lower execution permissions than the first container.

[0054] When the aggregate logic code needs to be executed, it is moved from the first container to the second container. The second container is pre-configured with security policies. When the aggregate logic code executes in the second container, its permissions are controlled. For example, it may be prohibited from modifying operating system files or accessing network hardware. The specific details of the execution permissions configured in the second container are not specified here.

[0055] Figure 3 This is a schematic diagram illustrating the process by which the data aggregation tool in this application processes the aggregation logic code. Figure 1 See Figure 3 As shown, the data aggregation tool first receives the aggregation logic code input by relevant personnel, and then stores the aggregation logic code in a first container. When the aggregation logic code needs to be executed, the data aggregation tool moves the aggregation logic code from the first container to a second container with execution permission restrictions.

[0056] After the aggregate logic code is moved from the first container without execution permission restrictions to the second container with operation permission restrictions, the aggregate logic code can be executed in the second container, which can restrict the aggregate logic code from performing any operations that may endanger the platform security.

[0057] Accordingly, step S202 above can be: in the second container, the aggregation logic code is translated into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter.

[0058] Accordingly, step S203 above can be: in the second container, the translated aggregation code is injected into the data aggregation framework of the data aggregation tool, so that the data aggregation tool with the injected translated aggregation code can perform data aggregation according to the data aggregation requirements after it runs.

[0059] As can be seen from the above, placing the aggregation logic code in a second container with execution permission restrictions can prevent the aggregation logic code from performing unsafe operations that could harm the data aggregation tool. This allows the data aggregation tool to be used in various coding languages ​​while ensuring its secure operation.

[0060] Furthermore, as an extension of step A1 above, before storing the aggregate logic code in the first container, the aggregate logic code can be security verified to initially ensure that the aggregate logic code stored in the first container is relatively safe.

[0061] Specifically, prior to step A1 above, the method may further include:

[0062] Step B1: Determine if there are any security risks in the aggregation logic code. If yes, proceed to step B2; otherwise, proceed to step A1.

[0063] Although the aggregation logic code is not executed in the first container, it is used to store the aggregation logic code. In order to prevent malicious writing of code that could cause the aggregation logic code to be executed in the first container and thus endanger the security of the data aggregation tool, it is necessary to determine whether the aggregation logic code has any security risks before storing it in the first container.

[0064] In determining whether aggregate logic code poses a security risk, it can be compared with known insecure code or functions. If the comparison is consistent, the aggregate logic code is considered to have a security risk; otherwise, it is considered not to have a security risk. Alternatively, the aggregate logic code can be submitted to a third party for analysis. If the third party reports a security risk, the aggregate logic code is considered to have a security risk; if they report no security risk, it is considered not to have a security risk. The specific method for determining whether aggregate logic code poses a security risk is not limited here.

[0065] Step B2: Discard the aggregation logic code.

[0066] If a security risk is identified in the aggregation logic code, indicating that its purpose may not be to achieve data aggregation in a specific way, but rather to attack the data aggregation tool, then the aggregation logic code will no longer be stored in the first container and will be discarded. Simultaneously, a notification message can be output to inform the person who entered the aggregation logic code or the administrator of the data aggregation tool that the entered code poses a security risk, prompting the person who entered the code to modify it and the administrator to strengthen security measures regarding entered code.

[0067] Step A1 is: Store the aggregation logic code in the first container.

[0068] If it is determined that the aggregation logic code does not pose a security risk, it means that the purpose of the aggregation logic code is likely to achieve data aggregation in a certain way, rather than to attack the data aggregation tool. At this point, the aggregation logic code can be stored in the first container for later execution.

[0069] As can be seen from the above, before storing the aggregation logic code in the first container, the security risks of the aggregation logic code are verified. Only the aggregation logic code that passes the security risk verification can be stored in the first container. This ensures that the aggregation logic code stored in the first container is relatively safe, and further ensures the safe operation of the data aggregation tool.

[0070] Furthermore, as a refinement of step B1 above, in most cases, malicious attacks on data aggregation tools are carried out through functions in the aggregation logic code. Therefore, the existence of security risks in the aggregation logic code can be judged by using function blacklists or function whitelists.

[0071] Specifically, step B1 above may include:

[0072] Step B11: Determine whether the function used in the aggregate logic code exists in the function blacklist.

[0073] Some functions are created specifically for malicious attacks on networks. These previously discovered functions can be grouped into a list, forming a function blacklist.

[0074] In determining whether a function exists in the aggregate logic code, one approach is to first extract all functions from the code. Then, each extracted function is matched against a function blacklist. If a function matches a function in the blacklist, the function in the aggregate logic code is confirmed to exist in the blacklist. If none of the extracted functions match any function in the blacklist, the function is confirmed not to exist in the blacklist. Alternatively, each function in the blacklist can be matched against the aggregate logic code. If a function in the blacklist matches a function in the aggregate logic code, the function is confirmed to exist in the blacklist. If none of the functions in the blacklist match a function in the aggregate logic code, the function is confirmed not to exist in the blacklist. The specific method for determining whether a function in the aggregate logic code exists in the blacklist is not specified here.

[0075] Step B12: Determine whether the function used in the aggregation logic code is outside the function whitelist.

[0076] Some functions are created specifically to perform certain normal functions on the network. These previously confirmed functions can be grouped into a list, which forms a function whitelist.

[0077] In determining whether a function exists in the aggregate logic code, one approach is to first extract all functions from the code. Then, each extracted function is matched against a function whitelist. If none of the extracted functions match any function in the whitelist, the function in the aggregate logic code is considered to be outside the whitelist. Conversely, if any extracted function matches a function in the whitelist, the function in the aggregate logic code is considered to be outside the whitelist. Alternatively, each function in the whitelist can be matched against the aggregate logic code. If none of the functions in the whitelist match, the function in the aggregate logic code is considered to be outside the whitelist. If any function in the whitelist matches, the function in the aggregate logic code is considered to be outside the whitelist. The specific method for determining whether a function in the aggregate logic code exists outside the whitelist is not specified here.

[0078] It should be noted that steps B11 and B12 can be executed simultaneously, or one of them can be executed. When steps B11 and B12 can be executed simultaneously, if either step B11 or B12 yields a yes result, the aggregate logic code is considered to have a security risk.

[0079] As can be seen from the above, by using function blacklists or function whitelists, functions with security risks in aggregate logic code can be discovered, thereby determining whether the aggregate logic code has security risks. There is no need to judge the content outside the functions in the aggregate logic code, thus improving the verification efficiency of security risks in aggregate logic code.

[0080] Furthermore, as an extension of step A1 above, after storing the aggregate logic code in the first container, the summary information of the aggregate logic code can be extracted, and the extracted summary information can be stored together with the aggregate logic code in the first container. This allows it to detect whether the aggregate logic code has been modified in the first container and facilitates the retrieval of the corresponding aggregate logic code from the first container.

[0081] Specifically, after step A1 above, the method may further include:

[0082] Step C1: Extract the summary information of the aggregate logic code.

[0083] In practical applications, the Message Digest Algorithm (MD5) value can be extracted from the aggregate logic code. Of course, other types of digest information can also be extracted from the aggregate logic code, such as key fields extracted from it. The specific type of digest information extracted is not limited here.

[0084] Step C2: Store the summary information in the first container, corresponding to the aggregation logic code.

[0085] After extracting the summary information from the aggregate logic code, the summary information and the aggregate logic code can be stored together in the first container.

[0086] When aggregate logic code needs to be run, the corresponding aggregate logic code can be found in the first container based on the summary information, or the summary information can be extracted again from the aggregate logic code in the first container and compared with the previously stored summary information to determine whether the aggregate logic code has been modified in the first container.

[0087] As can be seen from the above, while storing the aggregation logic code in the first container, the summary information of the aggregation logic code is also extracted, and the summary information of the aggregation logic code is stored in the first container in correspondence with the aggregation logic code. This allows the stored summary information to determine whether the aggregation logic code has been modified in the first container, thereby improving the security of the data aggregation tool. Furthermore, the summary information enables the tool to quickly and accurately locate the required aggregation logic code from the first container and execute it, thus improving the efficiency and accuracy of the data aggregation tool in performing data aggregation.

[0088] Furthermore, as an extension of step C2 above, in the process of moving the aggregate logic code from the first container to the second container, instead of directly moving the aggregate logic code from the first container to the second container, it is possible to first determine whether the aggregate logic code in the first container has changed. If it is determined that no change has occurred, then the aggregate logic code in the first container is moved to the second container.

[0089] Specifically, step A2 above may include:

[0090] Step C3: Respond to the code execution instruction and extract the aggregate logic code from the first container.

[0091] When the aggregation logic code needs to be executed, relevant personnel will enter the code execution command into the data aggregation tool. The data aggregation tool will respond to the code execution command and extract the aggregation logic code indicated by the code execution command from the first container.

[0092] Step C4: Extract the summary information of the aggregation logic code again.

[0093] After extracting the aggregate logic code from the first container, its summary information is extracted from the extracted aggregate logic code. The specific method for extracting the summary information from the aggregate logic code is the same as step C2 above, and will not be repeated here.

[0094] Step C5: Determine whether the extracted summary information is consistent with the summary information extracted before storage. If yes, proceed to step C6; otherwise, proceed to step C7.

[0095] Step C6: Move the aggregation logic code from the first container to the second container.

[0096] Step C7: Discard the aggregate logic code extracted from the first container.

[0097] When personnel input aggregation logic code into a data aggregation tool, the input code may be perfectly valid, pass security verification, and be stored in the first container. However, once stored in the first container, the aggregation logic code can be modified to attack the data aggregation tool. Therefore, it is necessary to verify whether the aggregation logic code extracted from the first container is the same as the one stored at that time.

[0098] If the content of the aggregation logic code changes, the extracted summary information will also differ. The summary information stored in the first container is based on the aggregation logic code at the time of storage. Then, the summary information is extracted from the aggregation logic code at this point, and the two summaries are compared. If they match, it means the aggregation logic code stored initially is consistent with the extracted aggregation logic code, and the aggregation logic code was not modified during storage. The aggregation logic code extracted from the first container can then be moved to the second container for execution. If they do not match, it means the aggregation logic code stored initially is inconsistent with the extracted aggregation logic code, and the aggregation logic code was modified during storage. In this case, the aggregation logic code extracted from the first container needs to be discarded to prevent it from falling into the second container and posing a security threat to the data aggregation tool. Simultaneously, a prompt message can be output to alert the administrators of the data aggregation tool that the aggregation logic code stored in the first container has been modified, and appropriate protective measures need to be taken for the data aggregation tool.

[0099] As can be seen from the above, after extracting the aggregation logic code from the first container, a summary information is extracted from the extracted aggregation logic code, and the extracted summary information is compared with the summary information stored in the first container to determine whether the aggregation logic code has been modified during its storage in the first container. Only when it is determined that no modification has occurred can the aggregation logic code be transferred to the second container for execution, thus ensuring the security of the data aggregation tool.

[0100] Furthermore, as a response to Figure 2 The method can also be extended to expand the content of the data aggregation framework, so that the data aggregation tool can be applied to more scenarios.

[0101] Figure 4 This is a schematic diagram of the data aggregation framework architecture in the embodiments of this application. Figure 1 See Figure 4 As shown, a data aggregation framework includes at least three modules: a pre-processing module, an aggregation module, and a post-processing module. The pre-processing module precedes the aggregation module and is used to preprocess the data before aggregation. The post-processing module follows the aggregation module and is used to process the aggregated data. The aggregation module is responsible for performing the data aggregation process.

[0102] Specifically, prior to step S201 above, the method may further include:

[0103] Step D1: Receive feature extension information.

[0104] When relevant personnel need to expand the functionality of a data aggregation tool, that is, to add some data processing methods to the data aggregation tool so as to process the data before or after aggregation at the appropriate time, the relevant personnel can edit the data processing methods and the timing of data processing in the function expansion information and input it into the data aggregation tool.

[0105] Step D2: Extract the data processing method and data processing timing from the extended function information.

[0106] Once the data aggregation tool receives the extended functionality information, it can parse the extended functionality information and then extract the data processing methods and timing from it.

[0107] Step D3: Determine the target module from the pre-processing module, aggregation module, and post-processing module based on the data processing timing.

[0108] Different data processing methods need to be implemented in different modules. For example, data authentication needs to be implemented in the front-end module, data aggregation and mock data judgment need to be implemented in the aggregation module, and data sorting, deduplication, push, and log collection and processing need to be implemented in the back-end module. The data processing timing specifies which module should be placed for the corresponding data processing method. Based on the data processing timing, the target module for executing the appropriate data processing method can be selected from the front-end module, aggregation module, and back-end module.

[0109] Step D4: Add the data processing method to the target module.

[0110] Once the target module for executing the data processing method is identified, the data aggregation tool can add the data processing method as code to the corresponding module of the data aggregation framework.

[0111] As can be seen from the above, based on the functional extension information input by relevant personnel, the data processing methods that relevant personnel need to extend can be added to the front-end module, aggregation module, or back-end module of the data aggregation framework, enabling the data aggregation tool to achieve more extended functions and increasing the use cases of the data aggregation tool.

[0112] Furthermore, as an extension of step D4 above, after expanding the functionality of the data aggregation framework, these extended functions can be used simultaneously during the aggregation of subsequent data to be aggregated. One such extended function is to authenticate data sent from different sources within the front-end module of the data aggregation framework, enabling the data aggregation tool to aggregate multi-source data.

[0113] Specifically, the data processing methods are authentication and aggregation, and the target modules are the front-end module and the aggregation module.

[0114] Following step D4 above, the method may further include:

[0115] Step D51: Receive multiple data entries to be aggregated.

[0116] Among them, many of the data to be aggregated come from different data sources.

[0117] Sometimes, the data to be aggregated doesn't originate from a single source; it may be necessary to aggregate data from multiple sources. Multiple data entries to be aggregated refer to data from different data sources that need to be aggregated. For example, three data entries to be aggregated: data entry 1 is aggregated from data source A, data entry 2 is aggregated from data source B, and data entry 3 is aggregated from data source A. In other words, some data from multiple data sources may originate from the same data source, while others may originate from different data sources.

[0118] Step D52: Authenticate multiple data entries to be aggregated in the front-end module.

[0119] Since data aggregation tools access multiple data sources, sometimes only data from a few of these data sources is needed for aggregation. In order to filter out data from unnecessary data sources, it is necessary to authenticate the multiple data entries to be aggregated before aggregation.

[0120] In practical applications, authentication methods such as account passwords and hooks can be used. The specific authentication method is not limited here.

[0121] For example, suppose a data aggregation tool connects to data sources A, B, and C. Data source A generates data 1 and data 2, data source B generates data 3, and data source C generates data 4. Currently, only the data generated by data sources A and B needs to be aggregated. Upon receiving data 1, data 2, data 3, and data 4, it's necessary to check the usernames and passwords used when connecting to the data sources for each of these data sources. That is, to check the usernames and passwords used by data sources A, B, and C. After checking, it's discovered that data source C is not the data source required for this aggregation. Therefore, data 4 generated by data source C is not authenticated, while data 1 and data 2 generated by data source A are authenticated, and data 3 generated by data source B is authenticated.

[0122] Step D53: When authentication is successful, aggregate the authenticated data in the aggregation module.

[0123] If the data to be aggregated passes authentication, it can enter the aggregation module to be aggregated with other data. If the data to be aggregated fails authentication, it cannot enter the aggregation module to participate in data aggregation and can be discarded.

[0124] As can be seen from the above, by authenticating the data to be aggregated, data from different data sources can be aggregated, thus solving the problem that data aggregation tools cannot achieve multi-source data aggregation.

[0125] Furthermore, as an extension of step D4 above, after expanding the functionality of the data aggregation framework, these extended functions can be used simultaneously during the aggregation of subsequent data to be aggregated. One such extended function is to aggregate mock data in the aggregation module of the data aggregation framework according to the actual situation and output it to the large screen for display.

[0126] Specifically, the data processing method is mock data judgment and aggregation, and the target module is the aggregation module.

[0127] Following step D4 above, the method may further include:

[0128] Step D61: Receive the preprocessing data output by the preprocessor module.

[0129] After the data to be aggregated is preprocessed in the front-end module, the preprocessed data is obtained and then sent to the aggregation module for aggregation.

[0130] Step D62: Based on the requirement to display mock data, determine in the aggregation module whether the pre-processed data is mock data. If yes, proceed to step D63; otherwise, proceed to step D64.

[0131] Step D63: Aggregate the pre-processed data in the aggregation module.

[0132] Step D64: Discard the preprocessed data and continue to receive the preprocessed data output by the preprocessing module.

[0133] Before aggregating the pre-processed data output by the front-end module, the aggregation module needs to check if there is a requirement for mock data display. If so, it needs to determine whether the pre-processed data is mock data. If yes, it means the pre-processed data meets the requirements, i.e., the current data to be aggregated meets the mock data display requirements, and the aggregation module then aggregates the pre-processed data. If no, it means the pre-processed data does not meet the requirements, i.e., the current data to be aggregated does not meet the mock data display requirements, and the pre-processed data can be discarded, and the module can continue to receive pre-processed data output by the front-end module.

[0134] As can be seen from the above, by setting a judgment mechanism for Mock data in the aggregation module, when there is a need to display Mock data, the Mock data can be aggregated and then output to the large screen for display, thus enriching the functionality of the data aggregation tool.

[0135] Furthermore, as an extension of step D4 above, after expanding the functionality of the data aggregation framework, subsequent data to be aggregated can also utilize these extended functions during the aggregation process. One such extended function is to perform further processing on the aggregated data in the post-modules of the data aggregation framework.

[0136] Specifically, the data processing methods include sorting, deduplication, pushing, and collecting and processing logs, with the target module being a post-processing module.

[0137] Following step D4 above, the method may further include:

[0138] Step D71: Receive the aggregated data output by the aggregation module.

[0139] After the aggregation module completes the aggregation of the data to be aggregated, it outputs the aggregated data to the subsequent module, which then receives the aggregated data.

[0140] Step D72: In the post-processing module, sort, deduplicate, push, and collect one or more of the processing logs from the aggregation module for the aggregated data.

[0141] Post-processing modules can further process aggregated data according to actual needs. In practical applications, post-processing typically includes one or more of the following: sorting, deduplication, pushing, and collecting processing logs from the aggregation module. The aggregated data can be processed accordingly based on actual needs or the current configuration.

[0142] Since the data is aggregated rather than calculated, the aggregated data still consists of multiple records. To facilitate viewing by relevant personnel, these records can be sorted according to certain rules, and duplicate data can be removed. To meet the needs of different stakeholders, these records can also be pushed out to the relevant parties. If anomalies are detected in the aggregated data, the processing logs of the aggregation module can be collected for review to help relevant personnel find the cause of the anomaly.

[0143] As can be seen from the above, by performing operations such as sorting, deduplication, pushing, and collecting aggregation logs on the aggregated data in the back-end module, the aggregated data can be made easier for relevant personnel to view and use, thus enriching the functionality of the data aggregation tool.

[0144] Finally, a complete embodiment will be used to illustrate the data aggregation method provided in the embodiments of this application.

[0145] Figure 5 This is a schematic diagram of the scenario architecture for the data aggregation method in the embodiments of this application. Figure 2 See Figure 5 As shown, the architecture may include: third-party business modules, data aggregation services, large screens, and BI requesters. The data aggregation service implements the data aggregation method described in this application embodiment.

[0146] First, third-party business modules generate third-party business data, which can originate from Hypertext Transfer Protocol (HTTP) or Java Database Connectivity (JBDC). Then, the data aggregation service collects and buffers this data as HTTP or SQL datasets. When aggregation is needed, a script edited by relevant personnel based on the data aggregation requirements is executed, updating the data and obtaining the aggregation result, which is then displayed via HTTP. Finally, the aggregated data is output to a dashboard via an HTTP display interface, and the displayed HTTP dataset is also output to the BI client. For the BI client, after the third-party business modules generate the third-party business data and the data aggregation service obtains the aggregation result, they can directly obtain both the third-party business data and the aggregation result.

[0147] Figure 6 This is a schematic diagram illustrating the process by which the data aggregation tool in this application processes the aggregation logic code. Figure 2 See Figure 6 As shown, the data aggregation platform includes aggregation services and script execution services.

[0148] When data aggregation is needed, relevant personnel write scripts based on these needs and input them into the aggregation service. Before storing the script, it is submitted to a script repository for blacklist verification. Once verified, it is stored in the repository. When the script needs to be executed, it is loaded from the database into volumes and its MD5 hash is verified in the aggregation service (container1). The aggregation service has already extracted the MD5 hash of the script upon receiving it; it then extracts the MD5 hash of the script from the volumes and verifies it using both hashes. After successful verification, when the script is actually executed, it is placed in the script execution service (container2), effectively moving the script from the aggregation service to the script execution service. The script execution service first uses Jython (JVM) as the Python script interpreter, translating the Python script into code usable by the aggregation platform. Then, it enables SecurityManager to restrict permissions such as I / O and socket access. The aggregation service and the script execution service reside in the same pod but in different containers. Both are Java containers; the script execution container uses SecurityManager and configures security policies to restrict script permissions. Two containers within the same pod, i.e., two local processes, can communicate using `localhsot:podhttp`.

[0149] Figure 7 This is a schematic diagram of the data aggregation framework architecture in the embodiments of this application. Figure 2 See Figure 7 As shown, the core of the data aggregation framework PDM is a data production factory, DataFactory. By defining the production steps of the factory pipeline, and the pre / post-processors (DataFactory postProcessor) before and after each step, and listening to the DataFactorylistenner during the process, the aggregation framework can be flexibly extended.

[0150] In the DataFactory data production pipeline, the following steps are performed sequentially: Pre-production check of DataDefinition; DataDefinition is populated with SetDefinition; production distributed lock is acquired; pre-production processing invokeProductBeforeProcessors, including integrity verification, feature filling, and building an aggregated local structured table (updating the table); then, productData is retrieved; for productData, it is determined whether it is mocked; if so, productData is retrieved again; otherwise, the data production sub-process doProduct is initiated; after production, post-production processing invokeProductPostProcessors, including updating local results; then, completion listening and publishing pushDataFactoryListener, including notification of the interface management module event dataFaceJobPublisher and pushMQ; finally, the production distributed lock is released, and DataDefinition is reset.

[0151] In the data production sub-process doProduct, the following steps are performed sequentially: Pre-processing via invokeCollectBeforeProcessors, which includes pre-processing for HTTP datasets, common parameter handling, Dayu authentication, TSOC authentication, business parameter transmission, time parameters, pagination parameters, and other set parameters; then collection via collectData, which includes the collection factory and collection chain, HTTP collection, and SQL collection; then post-processing via invokeCollectPostProcessors, which includes common processing to be expanded, such as deduplication and sorting; then script execution pre-processing via execScriptPostProcessBefore, which includes processing to be expanded, such as validation; and finally script execution via execScript, which includes the script execution factory and execution chain, Python execution, and JS execution.

[0152] In this way, the data aggregation framework can achieve: HTTP authentication, which solves the authentication problem when accessing different HTTP data sources by extending the postProcessor; meeting the needs of different data usage methods, which solves the different needs of pull and push of aggregation results by extending the postProcessor; supporting secondary development, which makes it easy to extend the features of the aggregation platform through the DataFactory postProcessor, such as adding support for sorting and deduplication of aggregation results; and collecting logs of the aggregation process through the listener.

[0153] Based on the same inventive concept, as an implementation of the above method, this application also provides a data aggregation device. Figure 8 This is a schematic diagram of the data aggregation device in an embodiment of this application. See also... Figure 8 As shown, the device may include:

[0154] The receiving module 801 is used to receive aggregation logic code, which is written based on data aggregation requirements and is different from the code language used by the data aggregation tool.

[0155] Translation module 802 is used to translate the aggregation logic code into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter;

[0156] The injection aggregation module 803 is used to inject the translated aggregation code into the data aggregation framework of the data aggregation tool, so that the data aggregation tool with the injected translated aggregation code can perform data aggregation according to the data aggregation requirements after it runs.

[0157] Furthermore, the device further includes: a transfer module for storing the aggregate logic code in a first container; and, in response to a code execution instruction, moving the aggregate logic code from the first container to a second container, wherein the execution permissions of the second container are lower than those of the first container;

[0158] The translation module is specifically used in the second container to translate the aggregation logic code into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter.

[0159] Furthermore, the device also includes: a verification module, used to determine whether the aggregation logic code has a security risk; if so, the aggregation logic code is discarded;

[0160] The transfer module is specifically used to store the aggregation logic code in the first container if no.

[0161] Furthermore, the verification module is specifically used to determine whether the function used in the aggregation logic code exists in the function blacklist; or, to determine whether the function used in the aggregation logic code exists outside the function whitelist.

[0162] Furthermore, the device further includes: an authentication module, used to extract digest information of the aggregation logic code; and to store the digest information in the first container corresponding to the aggregation logic code.

[0163] Furthermore, the transfer module is specifically configured to respond to a code execution instruction by extracting the aggregated logic code from the first container; extracting the summary information of the aggregated logic code again; determining whether the extracted summary information is consistent with the summary information extracted before storage; if so, moving the aggregated logic code from the first container to the second container; if not, discarding the aggregated logic code extracted from the first container.

[0164] Furthermore, the data aggregation framework includes an aggregation module, and a pre-module located before the aggregation module or a post-module located after the aggregation module. The device further includes an extension module, configured to receive functional extension information; extract data processing methods and data processing timings from the functional extension information; determine a target module from the pre-module, the aggregation module, and the post-module according to the data processing timings; and add the data processing methods to the target module.

[0165] Furthermore, the data processing method is authentication and aggregation, the target module is a front-end module and an aggregation module, and the device further includes: an execution module, used to receive multiple pieces of data to be aggregated, the multiple pieces of data to be aggregated coming from different data sources; to authenticate the multiple pieces of data to be aggregated in the front-end module; and when the authentication is successful, to aggregate the authenticated data to be aggregated in the aggregation module.

[0166] Furthermore, the data processing method is Mock data judgment and aggregation, the target module is an aggregation module, and the device further includes: an execution module, used to receive pre-processed data output by the front-end module; based on the Mock data display requirements, determine in the aggregation module whether the pre-processed data is Mock data; if so, aggregate the pre-processed data in the aggregation module; if not, discard the pre-processed data and continue to receive pre-processed data output by the front-end module.

[0167] Furthermore, the data processing method includes one or more of sorting, deduplication, pushing, and collecting processing logs. The target module is a post-module. The device further includes an execution module, which is used to receive the aggregated data output by the aggregation module; and in the post-module, sorting, deduplicating, pushing, and collecting one or more of the processing logs of the aggregation module from the aggregated data.

[0168] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0169] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 9 This is a schematic diagram of the electronic device in an embodiment of this application. See also... Figure 9 As shown, the electronic device may include: a processor 901, a memory 902, and a bus 903; wherein the processor 901 and the memory 902 communicate with each other through the bus 903; the processor 901 is used to call program instructions in the memory 902 to execute the methods in one or more of the above embodiments.

[0170] It should be noted that the descriptions of the above electronic device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the electronic device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0171] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium, which may include: a stored program; wherein, when the program is running, it controls the device where the storage medium is located to execute the methods in one or more of the above embodiments.

[0172] It should be noted that the descriptions of the storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0173] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data aggregation method, characterized in that, The method includes: Receive aggregation logic code, which is written based on data aggregation requirements and is different from the code language used by the data aggregation tool; The aggregation logic code is translated into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter. The translated aggregation code is injected into the data aggregation framework of the data aggregation tool, so that the data aggregation tool with the translated aggregation code injected can perform data aggregation according to the data aggregation requirements after it runs. The method further includes, before translating the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool via an interpreter: The aggregation logic code is stored in the first container; In response to a code execution instruction, the aggregated logic code is moved from the first container to the second container, where the execution permissions of the second container are lower than those of the first container. The step of translating the aggregation logic code into translated aggregation code consistent with the code language used by the data aggregation tool via an interpreter includes: In the second container, the aggregation logic code is translated into translated aggregation code that is consistent with the code language used by the data aggregation tool by an interpreter.

2. The method according to claim 1, characterized in that, Before storing the aggregation logic code in the first container, the method further includes: Determine whether the aggregation logic code poses any security risks; If so, then discard the aggregation logic code; The step of storing the aggregation logic code in the first container includes: If not, the aggregation logic code is stored in the first container.

3. The method according to claim 2, characterized in that, The determination of whether the aggregation logic code has security risks includes: Determine whether the function used in the aggregation logic code exists in the function blacklist; or, Determine whether the function used in the aggregation logic code exists outside the function whitelist.

4. The method according to claim 1, characterized in that, After storing the aggregation logic code in the first container, the method further includes: Extract the summary information of the aggregation logic code; The summary information is stored in the first container in correspondence with the aggregation logic code.

5. The method according to claim 4, characterized in that, The response code execution instruction moves the aggregation logic code from the first container to the second container, including: In response to a code execution instruction, the aggregation logic code is extracted from the first container; Extract the summary information of the aggregation logic code again; Determine whether the extracted summary information is consistent with the summary information extracted before storage; If so, the aggregation logic code is moved from the first container to the second container; If not, the aggregation logic code extracted from the first container is discarded.

6. The method according to any one of claims 1 to 5, characterized in that, The data aggregation framework includes an aggregation module, and a pre-module located before the aggregation module or a post-module located after the aggregation module. Before receiving the aggregation logic code, the method further includes: Receive extended function information; Extract the data processing method and data processing timing from the aforementioned extended function information; The target module is determined from the front-end module, the aggregation module, and the back-end module based on the data processing timing. Add the data processing method to the target module.

7. The method according to claim 6, characterized in that, The data processing method is authentication and aggregation, the target module is a front-end module and an aggregation module, and after adding the data processing method to the target module, the method further includes: Receive multiple data points to be aggregated, which come from different data sources; The multiple data entries to be aggregated are authenticated in the front-end module; When authentication is successful, the authenticated data to be aggregated is aggregated in the aggregation module.

8. The method according to claim 6, characterized in that, The data processing method is mock data judgment and aggregation, the target module is an aggregation module, and after adding the data processing method to the target module, the method further includes: Receive the preprocessing data output by the pre-processing module; Based on the requirement for displaying Mock data, the aggregation module determines whether the pre-processed data is Mock data. If so, the preprocessed data is aggregated in the aggregation module; If not, discard the preprocessed data and continue to receive the preprocessed data output by the preprocessing module.

9. The method according to claim 6, characterized in that, The data processing method is one or more of sorting, deduplication, pushing, and log collection and processing. The target module is a post-module. After the data processing method is added to the target module, the method further includes: Receive the aggregated data output by the aggregation module; The back-end module sorts, deduplicates, pushes, and collects one or more of the processing logs from the aggregation module for the aggregated data.

10. A data aggregation device, characterized in that, The device includes: The receiving module is used to receive aggregation logic code, which is written based on data aggregation requirements and is different from the code language used by the data aggregation tool. The translation module is used to translate the aggregation logic code into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter; An injection aggregation module is used to inject the translated aggregation code into the data aggregation framework of the data aggregation tool, so that the data aggregation tool with the injected translated aggregation code can perform data aggregation according to the data aggregation requirements after it runs. The device further includes: a transfer module for storing the aggregation logic code in a first container; and, in response to a code execution instruction, moving the aggregation logic code from the first container to a second container, wherein the execution permissions of the second container are lower than those of the first container; The translation module is specifically used in the second container to translate the aggregation logic code into translated aggregation code that is consistent with the code language used by the data aggregation tool through an interpreter.

11. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The storage medium includes: a stored program; wherein, when the program is executed, it controls the device where the storage medium is located to perform the method as described in any one of claims 1 to 9.

Images