Stock equity penetration calculation method supporting unbalanced dynamic expansion of resources

Abstract

The invention discloses an equity penetration calculation method supporting unbalanced dynamic expansion of resources. According to the method, additional configuration is not needed, and the computing resources can be easily and horizontally expanded only by increasing consumers of the message queue. For each calculation node, the total calculation amount can be predicted in combination with the historical penetration scale of the current calculation task, the prediction value is continuously corrected in combination with the current actual calculation condition, and through the prediction value, the task throughput can be automatically and almost adjusted in real time without human intervention, so that the calculation efficiency is improved. When the load is relatively high, task calculation is reduced and sent to avoid overload, and when the load is relatively low, tasks are increased and sent to fully utilize idle resources. And the throughput of stock equity penetration computing tasks and the utilization rate of overall computing resources are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a method for realizing infinite penetration of equity relationship calculation of shareholders or investment companies of an enterprise and supporting dynamic expansion of resource imbalance, belonging to the technical fields of knowledge graph, distributed batch processing and cloud computing. Background Art

[0002] With the increasingly prosperous business activities, frequent investment and mergers and acquisitions, and the surge in data in the information age, the equity structure of enterprises has become increasingly complex. Especially for some large companies or investment groups, they may involve multi-level subsidiaries, affiliated companies and various partners. However, the public data only has directly related first-degree data, and relying solely on "first-degree data" may not fully reveal the complex corporate network or potential risks. Because in addition to direct shareholding, there may also be multi-level, cross-cutting or hidden control and influence structures. For example, if enterprise C indirectly controls B after passing through many companies, it is necessary to consider the more distant link relationship between the two, or enterprise A indirectly realizes actual control of a certain company by holding many corporate equity hierarchical structures.

[0003] If it is possible to infinitely penetrate all shareholders or investment subsidiaries of the target company and find out all its related companies, shareholding ratios and relationship paths, and apply them in the field of risk control, it can provide data support for risk managers to better evaluate and identify potential problems, such as discovering debt problems, related transactions, financial fraud and other behaviors in advance, bringing important value to the business of financial clients such as banks, funds, and private equity.

[0004] In order to achieve the goal of calculating the target company's infinitely penetrating equity relationship, it is necessary to deal with the company's complex equity relationship network. A system or method is needed to efficiently integrate data to form a directed graph, and traverse all nodes of the directed graph to calculate the final penetrated shareholding ratio. At the same time, when the equity relationship of the enterprise changes, it can be recalculated to reflect the latest changes.

[0005] In order to serve this goal, it is necessary to first construct the direct relationship data of all companies or natural persons (nodes) and equity relationships (edges between two points) in a graph database, where a graph database is a database that specializes in storing huge graph networks and retrieving information from them. It can efficiently store the data in the graph as points and edges, and can also attach attributes (such as shareholding ratio) to points and edges. At present, the graph database already has data constructed based on 1-degree equity relationships, but querying all equity penetration paths and penetration ratios of the target company cannot be simply implemented using the query statements provided by the graph database. It is necessary to start from Company A, continuously expand outward to explore and traverse the directed graph, find all paths, and finally calculate the final penetration ratio.

[0006] During the calculation process, the equity structures of different companies vary greatly. A complex structure needs to penetrate hundreds of thousands of companies and generate tens of millions of different equity relationship paths, while a simple structure may only require dozens of companies and generate only hundreds or thousands of equity relationships. Obviously, the resources (memory and CPU) required for these two types of calculations are completely different. If the calculation is performed with a fixed task concurrency, it is easy to encounter resource overload and machine crashes at certain times because there are too many companies that need to be penetrated by the companies being calculated at the same time, or at certain times because there are too few companies that need to be penetrated by the companies being calculated at the same time, resulting in a large amount of idle resources. In order to shorten the full calculation time and improve the overall throughput, it is necessary to design a system or method to adapt to this imbalance of computing resources. Under given computing resources, when the calculation is busy, wait for the resources to be released after the calculation ends, and when the computing resources are sufficient, increase task concurrency to make full use of idle resources. Summary of the invention

[0007] The purpose of the present invention is to solve the problem of uneven computing resource requirements of different companies in the process of equity penetration calculation.

[0008] In order to achieve the above object, the technical solution of the present invention is to disclose a method for equity penetration calculation supporting dynamic expansion of resource imbalance, which is characterized by comprising the following steps:

[0009] Split the full computing task into subtasks with the smallest granularity of unlimited penetration of all shareholders or investment subsidiaries of a target company, and achieve horizontal expansion of computing by adding computing nodes;

[0010] By using the message middleware, the split subtasks are encapsulated into messages and sent to a message queue. All consumers of the queue are monitored. According to the consumer messages, the task of calculating all paths and shareholding ratios of the target company's equity penetration corresponding to the message is executed. The number of consumer threads is adaptively adjusted according to the current calculation situation, further including the following steps:

[0011] When a consumer starts consuming messages, it maintains a total weight value W sumP For any of the consumption threads, when starting to consume a message, the weight value W calculated last time for the target company corresponding to the message is determined pred And the total weight value W sumP Is the sum less than the maximum weight value W? max , that is, W pred +W sumP <W max :If it is less than the maximum weight value W max , then update W sumP =W sumP +W pred, and consume the message; otherwise, send the message back to the message queue;.

[0012] When W pred +W sumP <W max When the equity penetration calculation of the target company begins, the actual weight value W is continuously updated during the calculation process according to the number and depth of equity penetration paths currently explored dynamically. cur If W appears during the update process cur >W pred , then update W pred The update formula is W pred =W pred *2 i , where i indicates the occurrence of W cur >W pred The number of times, and update the corresponding W sumP The value of

[0013] When the equity penetration calculation of the target company is completed, the actual weight value W is reset cur is 0, and W is updated at the same time sumP =W sumP -W pred ;

[0014] At the same time, each computing node has a scheduled task to compare W at regular intervals. sumP and W max , when W sumP >W max When W of all consumer threads of the current computing node is summed up cur and W max Compare, if the former is greater than the latter, find the thread that started the calculation most recently and set an exit flag; set several checkpoints in the thread calculation process to check the status. If it is found that the flag is the exit status, send the message back to the message queue and release all memory references generated by the current calculation. At the same time, set W cur is 0, and finally suspends the current computing thread for a period of time. When it wakes up again, it updates W sumP =W sumP -W pred At the same time, the scheduled task sets a sliding time window and counts the number of thread suspensions within the sliding window time range. If the number of suspensions exceeds a certain threshold, it means that the current calculation is busy. The thread that started the calculation most recently is found and a dormant state is set. Similarly, the current calculation thread sends the message back to the message queue and exits to release memory. At the same time, the thread is suspended for a long time, waiting to be awakened.

[0015] Preferably, the summed weight value W sumP Initially 0.

[0016] Preferably, the weight value W pred The technical method is: the number of all paths of the target company recorded in the last calculation * the average depth of the path * the size of a single point instance * the coefficient, where the coefficient is greater than 1.

[0017] Preferably, if the data calculated last time is not recorded, the weight value W pred is a preset initial value.

[0018] Preferably, if the message is sent back to the consumer queue after three retries, it means that the current calculation is busy, and the current calculation thread is suspended for a period of time.

[0019] Preferably, statistics W are calculated within the sliding window time range. sumP Mean and peak values, if W sumP If both the mean and peak values ​​are lower than a certain threshold, the number of dormant consumer threads is increased or awakened.

[0020] Preferably, the threshold is based on W max A certain proportion of the calculation.

[0021] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0022] 1. No additional configuration is required. It is easy to horizontally scale out computing resources by simply adding consumers to the message queue.

[0023] 2. For each computing node, the overall computing volume can be predicted based on the historical penetration scale of the computing task being performed, and the predicted value can be continuously revised based on the current actual computing situation. Through this predicted value, the task throughput can be automatically and nearly adjusted in real time without human intervention. When the load is high, the task concurrency can be reduced to avoid overload, and when the load is low, the task concurrency can be increased to make full use of idle resources. This significantly improves the throughput of equity penetration computing tasks and the utilization rate of overall computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It illustrates the dynamic exploration of equity penetration paths through consumption message execution;

[0025] Figure 2 Flowchart for adaptive resource imbalance calculation. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0027] The embodiment of the present invention discloses a method for calculating equity penetration that supports dynamic expansion of resource imbalance, which splits the full computing task into subtasks with the smallest granularity of unlimited penetration of all shareholders or investment subsidiaries of a target company. Because each subtask has no mutual dependence, it is easy to achieve horizontal expansion of computing by adding computing nodes. By using message middleware, the split subtasks are encapsulated into messages and sent to a message queue. All consumers of the queue are monitored, and the task of calculating all paths and shareholding ratios of the equity penetration of the target company corresponding to the message is executed according to the consumption message. The whole process is as follows: Figure 1 shown.

[0028] In this way, the computational concurrency of the entire batch task can be equivalent to: number of consumers * number of consumer threads per consumer. The question here is how to set the size of the number of consumer threads? If you are too optimistic, a large number of concurrent calculations will easily cause memory overflow. If the size is set too cautiously, it will cause a large number of node resources to be idle, thereby reducing the overall throughput. However, even if a suitable value is found, it may also cause memory overflow in extreme cases because too many target companies are penetrated by the companies calculated simultaneously on a node.

[0029] Therefore, it is necessary to design an adaptive method that can adaptively adjust the number of consumer threads according to the current computing situation.

[0030] In order to achieve this goal, the present invention will count the total number of all penetration paths and the average path depth of the company each time the equity penetration of the target company is calculated, and a weight value is calculated by the formula: the number of all paths of the target company * the average path depth * the size of a single point instance * the coefficient (the coefficient is greater than 1, leaving a certain safety margin), and record it as W pred , if it has not been recorded before, set an initial value.

[0031] For each consumer thread, the actual weight value is calculated based on the number of all penetration paths and path depths generated during the dynamic calculation process, and is recorded as W cur .

[0032] At the same time, for each consumer node, a value is calculated based on the maximum memory set when the node program is started, and recorded as the maximum weight value W max .

[0033] When a consumer starts consuming messages, it maintains a total weight value recorded as W sumP , which is initially 0. For any of the consumption threads, when starting to consume a message, it is determined whether the sum of the last calculated weight value and the total weight value of the target company corresponding to the message is less than the maximum weight value, that is, W pred +W sumP <W max : If it is less than the maximum weight value, update W sumP =W sumP +W pred , and consume the message; otherwise, send the message back to the message queue. If the message is sent back to the consumption queue after three retries, it means that the current calculation is busy and will suspend itself for a while.

[0034] When W pred +W sumP <W max When the equity penetration calculation of the target company begins, W will be continuously updated according to the number and depth of equity penetration paths currently explored dynamically during the calculation process. cur If W appears during the update process cur >W pred , then update W pred The update formula is W pred =W pred *2 i , where i indicates the occurrence of W cur >W pred The number of times, and update the corresponding W sumP The value of .

[0035] When the equity penetration calculation of the target company is completed, reset W cur is 0, and W is updated at the same time sumP =W sumP -W pred .

[0036] At the same time, each computing node will have a scheduled task to compare W at regular intervals. sumP and W max , when W sumP >W max When W of all consumer threads of the current node is summed up cur and W max Compare, if the former is greater than the latter, find the thread that started the calculation most recently, set an exit flag (several checkpoints will be set during the thread calculation process to check the status. If it is found that the flag is the exit status, the message will be sent back to the message queue, and all memory references generated by the current calculation will be released. At the same time, W is set cur0, and finally suspend itself for a period of time. When it wakes up again, it updates W sumP =W sumP -W pred ).

[0037] At the same time, the scheduled task will set a sliding time window, and count the number of thread suspensions within the sliding window time range. If the number of suspensions exceeds a certain threshold, it means that the current calculation is busy. Find the thread that started the calculation recently and set it to a dormant state (the thread sends the message back to the message queue and exits to release memory. At the same time, the thread is suspended for a long time and waits to be awakened); in addition, count W within the sliding window time range. sumP Mean and peak values, if W sumP Both the mean and peak values ​​are below a certain threshold (according to W max A certain proportion of the calculation), then increase or wake up the number of dormant consumer threads.

[0038] In summary, the present invention calculates a summary prediction weight value W based on the historical data of the target company calculated by all threads under the current node. sumP , and correct W based on the actual calculation results sumP With this prediction value and combined with the maximum weight value W max , when W sumP Short term greater than W max , indicating that the current calculation is insufficient, no new calculation tasks are accepted and the thread is suspended; when W sumP Long term greater than W max , indicating that the current node is busy with computing tasks, and the consumption thread is reduced or dormant through scheduled tasks; when W sumP If the value is less than a certain threshold for a long time, it means that the current computing task is relatively idle, so wake up the dormant thread or directly increase the consumption thread. Figure 2 shown.

Claims

1. A method for equity penetration calculation that supports dynamic expansion of resource imbalance, characterized in that: The following steps are involved: Split the full computing task into subtasks with the smallest granularity of unlimited penetration of all shareholders or investment subsidiaries of a target company, and achieve horizontal expansion of computing by adding computing nodes; By using the message middleware, the split subtasks are encapsulated into messages and sent to a message queue. All consumers of the queue are monitored. According to the consumer messages, the task of calculating all paths and shareholding ratios of the target company's equity penetration corresponding to the message is executed. The number of consumer threads is adaptively adjusted according to the current calculation situation, further including the following steps: When a consumer starts consuming messages, it maintains a total weight value W sumP For any of the consumption threads, when starting to consume a message, the weight value W calculated last time for the target company corresponding to the message is determined. pred And the total weight value W sumP Is the sum less than the maximum weight value W? max , that is, W pred +W sumP <W max :If it is less than the maximum weight value W max , then update W sumP =W sumP +W pred , and consume the message; otherwise, send the message back to the message queue;. When W pred +W sumP <W max When the equity penetration calculation of the target company begins, the actual weight value W is continuously updated during the calculation process according to the number and depth of equity penetration paths currently explored dynamically. cur If W appears during the update process cur >W pred , then update W pred The update formula is W pred =W pred *2 i , where i indicates the occurrence of W cur >W pred The number of times, and update the corresponding W sumP The value of When the equity penetration calculation of the target company is completed, the actual weight value W is reset cur is 0, and W is updated at the same time sumP =W sumP -W pred ; At the same time, each computing node has a scheduled task to compare W at regular intervals. sumP and W max , when W sumP >W max When W of all consumer threads of the current computing node is summed up cur and W max Compare, if the former is greater than the latter, find the thread that started the calculation most recently and set an exit flag; set several checkpoints in the thread calculation process to check the status. If it is found that the flag is the exit status, send the message back to the message queue and release all memory references generated by the current calculation. At the same time, set W cur is 0, and finally suspends the current computing thread for a period of time. When it wakes up again, it updates W sumP =W sumP -W pred At the same time, the scheduled task sets a sliding time window and counts the number of thread suspensions within the sliding window time range. If the number of suspensions exceeds a certain threshold, it means that the current calculation is busy. The thread that started the calculation most recently is found and a dormant state is set. Similarly, the current calculation thread sends the message back to the message queue and exits to release memory. At the same time, the thread is suspended for a long time, waiting to be awakened.

2. A method for equity penetration calculation supporting dynamic expansion of resource imbalance as claimed in claim 1, characterized in that: The summed weight value W sumP Initially 0.

3. The equity penetration calculation method supporting dynamic expansion of resource imbalance as claimed in claim 1, characterized in that: The weight value W pred The technical method is: the number of all paths of the target company recorded in the last calculation * the average depth of the path * the size of a single point instance * the coefficient, where the coefficient is greater than 1.

4. The equity penetration calculation method supporting dynamic expansion of resource imbalance as claimed in claim 1, characterized in that: If the data calculated last time is not recorded, the weight value W pred is a preset initial value.

5. The equity penetration calculation method supporting dynamic expansion of resource imbalance as claimed in claim 1, characterized in that: If the message is sent back to the consumer queue after three retries, it means that the current calculation is busy and the current calculation thread will be suspended for a period of time.

6. The equity penetration calculation method supporting dynamic expansion of resource imbalance as claimed in claim 1, characterized in that: Statistics W within the sliding window time range sumP Mean and peak values, if W sumP If both the mean and peak values ​​are lower than a certain threshold, the number of dormant consumer threads is increased or awakened.

7. The equity penetration calculation method supporting dynamic expansion of resource imbalance as claimed in claim 6, characterized in that: The threshold is based on W max A certain proportion of the calculation.

Images