Operation data management method based on artificial intelligence

Abstract

The invention relates to the technical field of data processing, in particular to an artificial intelligence-based operation data management method, which comprises the following steps of: determining a data source network, deploying web crawlers in each server in the data source network, and capturing data contents generated by each server in the data source network in real time based on the deployed web crawlers; the method comprises the steps of capturing data content, traversing all captured data content, identifying relevant parameters of the data content, analyzing the value of the data content based on the relevant parameters of the data content, creating a cloud database, and setting a distinguished storage interval in the cloud database. The data content generated in real time in the operation network is captured, value analysis is further carried out on the data content, so that differentiated storage of the data content is achieved, meanwhile, data iteration logic and recommendation logic are configured, convenience is provided for users calling and reading the operation data, and therefore the convenience of the operation data in the use stage is effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an operation data management method based on artificial intelligence. Background Art

[0002] Operational data management is a key part of enterprise operations. It covers processes such as data collection, integration, analysis and application. Through accurate collection of multi-channel data and in-depth analysis using professional tools, we can gain insight into operational trends, provide a basis for decision-making, help optimize strategies, improve efficiency, enhance competitiveness, and drive the sustainable and stable development of enterprises.

[0003] The invention patent with application number 202311428663.9 discloses a data management method based on artificial intelligence, which includes: analyzing the received data of the data receiving channel, and dividing the storage space of the data storage center based on the analysis results of the received data; establishing an encrypted database according to the data receiving channel, and encrypting and protecting the data receiving channel; identifying the parameter attributes of the received data, and storing the received data in each storage space divided by the data storage center according to the parameter attributes; obtaining the storage data content and storage space division logic of the data storage center; monitoring the storage space capacity of the data storage center in real time, and generating new storage space based on the acquired storage data content and storage space division logic of the data storage center when the space capacity overflows to increase the space capacity.

[0004] This application aims to solve the problem that: "In the related art, the data management method often classifies and stores the data after receiving the data. However, when the amount of received data is huge, data management not only increases the difficulty of data management, but also reduces the efficiency of data management. It is difficult to achieve effective classification and management of data, and there is room for improvement."

[0005] However, since operational data are mostly large and complex, they face many challenges in storage, management, and classification.

[0006] To this end, we propose an AI-based approach to operational data management. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides an operation data management method based on artificial intelligence, which solves the technical problems raised in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: An operation data management method based on artificial intelligence, comprising: Determine the data source network, deploy web crawlers on each server in the data source network, and capture the data content generated by each server in the data source network in real time based on the deployed web crawlers; traverse all captured data content, identify relevant parameters of each data content, analyze the value of the data content based on the relevant parameters of the data content, create a cloud database, set differentiated storage intervals in the cloud database, and store the data content in differentiated storage intervals based on the results of the data content value analysis; identify the data content of each differentiated storage interval in the cloud database, continuously monitor the activity of each data content in each differentiated storage interval, and iterate the data content stored in each differentiated storage interval in the cloud database based on the data content activity monitoring results; create a recommended content data package, and feed back the recommended content data package to the user end in real time; monitor the reading status information of the recommended content data package read by the user end, and move the data content in the recommended content data package based on the reading status information.

[0009] Furthermore, the data source network is a network used for daily operations of the user end. When deploying web crawlers, the server in the data source network decides the number of web crawlers to be deployed based on the amount of data generated by the server operation, and the number of web crawlers deployed in each server is not less than two; Among them, when deciding the number of web crawlers deployed on the server, the following is followed: a data statistical period is set, the number of data items generated by the server within the data statistical period is counted based on the data statistical period, and the statistical results are averaged based on the data statistical period. The larger the corresponding average result of the server, the more web crawlers are deployed in the server, and vice versa, the fewer web crawlers are deployed in the server.

[0010] Furthermore, the web crawler deployed in each of the servers is controlled to run based on a user-defined data capture cycle. After each of the web crawlers deployed in the server runs based on the data capture cycle and reaches the data capture cycle, the data content captured by each web crawler is summarized and a cleaning operation is performed on the data content. ; Where: is the similarity of two data; is the distance between two data after being converted into Hamming sequence; is the total length of the sequence; Among them, based on the above formula, the similarity calculation is performed on all data contents captured by the web crawler in groups of two. When the calculation result is 1, the two data are determined to be the same data, and any one of the two identical data is further selected to be deleted, and the data content cleaning operation is completed based on the above.

[0011] Furthermore, the data content related parameters include: the number of data content reads, the total number of data content read users, the frequency of data content reads, the number of data content downloads, and the number of data content retrievals; The data content value analysis logic is expressed as: ; Where: The value of data content; is the normalization factor; The number of times the data content is read; Read the total number of users for data content; Read frequency for data content; The number of times the data content is downloaded; The number of times the data content is retrieved; Among them, the value of each data content is calculated based on the above formula, and the normalization factor >1.

[0012] Furthermore, the operation of setting up the storage partition in the cloud database is: Obtain the analysis results of each data content based on the value analysis logic, capture the maximum value of the analysis result, set the number of differentiated storage intervals, and further set the interval for storing the target data content corresponding to each differentiated storage interval as follows: ; Where: The maximum value of the data content value analysis result; Set the quantity to distinguish the storage intervals; The maximum value of the data content stored in each partition is limited to not included.

[0013] Furthermore, the activity monitoring logic for distinguishing the data content stored in the storage interval in the cloud database is expressed as: Monitor whether any three of the data content reading times, the total number of data content reading users, the data content reading frequency, the data content download times, and the data content retrieval times among the relevant parameters of the data content are continuously reduced. If the monitoring result is yes, the data content is determined to be inactivated. If the monitoring result is no, the data content is determined to be still active. If the same data content is determined to be inactivated twice in a row, the data content is taken as an iteration target and an iteration operation is performed. Among them, the iterative operation of the data content and the operation of deleting the data content in the partitioned storage interval where the data content is located.

[0014] Furthermore, the recommended content data package is integrated from data contents stored in storage intervals of each partition in at least three cloud databases.

[0015] Furthermore, the creation logic of the recommended content data package is: Obtain historical activity determination results of each piece of data content in each partitioned storage interval in the cloud database, perform cumulative measurement on the activity determination results of each piece of data content, arrange each piece of data content in descending order based on the cumulative measurement amount of the determination results of each piece of data content being active in the measurement results, set the number of data content items included in the recommended content data packet, and pick up a corresponding number of data content items at the front position in the data content descending order queue based on the set number of items to form recommended content data.

[0016] Furthermore, the recommended content data packet is transmitted to the user terminal through the data source network. When the recommended content data packet is transmitted to the user terminal, the mobile computer device held by the user terminal is used as the transmission target to perform the transmission operation of the recommended content data packet.

[0017] Furthermore, the reading status information includes: a single reading time of the data content, a number of reading times of the data content, and the operation of moving the data content in the recommended content data packet is: Obtain the reading status information of each data in the recommended content data package, calculate the cumulative reading time of each data in the recommended content data package, select the data content with the longest cumulative reading time as the moving target, and move the data content to the parent partition storage interval of the partition storage interval where it is located.

[0018] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects: The present invention provides an operation data management method based on artificial intelligence. During the execution of the method, the data content generated in real time in the operation network is captured by setting up multiple network crawlers, and the value of the data content is further analyzed, so as to realize the differentiated storage of the data content. At the same time, data iteration logic and recommendation logic are configured to provide convenience for users who call to read the operation data, thereby effectively ensuring the convenience of the operation data in the use stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 The figure is a flowchart of an operational data management method based on artificial intelligence. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The present invention will be further described below in conjunction with the embodiments.

[0023] Embodiment 1: An operation data management method based on artificial intelligence in this embodiment, such as Figure 1 As shown, including: Determine a data source network, deploy a web crawler on each server in the data source network, and capture the data content generated by each server in the data source network in real time based on the deployed web crawler; Traverse all captured data content, identify relevant parameters of each data content, analyze the value of the data content based on the relevant parameters of the data content, create a cloud database, set up separate storage intervals in the cloud database, and store the data content in separate storage intervals based on the results of the data content value analysis; The web crawler deployed in each server is controlled by a user-defined data capture cycle. After the web crawler deployed in each server runs based on the data capture cycle and reaches the data capture cycle, the data content captured by each web crawler is summarized and a cleaning operation is performed on the data content. ; Where: is the similarity of two data; is the distance between two data after being converted into Hamming sequence; is the total length of the sequence; The above logical formula provides prerequisite support for data content cleaning.

[0024] Among them, based on the above formula, all the data contents captured by the web crawler are calculated for similarity in pairs. When the calculation result is 1, the two data are determined to be the same data, and any one of the two identical data is further selected to be deleted, and the data content cleaning operation is completed based on the above; Data content related parameters include: number of data content reads, total number of data content read users, frequency of data content reads, number of data content downloads, and number of data content retrievals; The logic of data content value analysis is expressed as: ; Where: The value of data content; is the normalization factor; The number of times the data content is read; Read the total number of users for data content; Read frequency for data content; The number of times the data content is downloaded; The number of times the data content is retrieved; The value of each data content is calculated through the above logic formula, thereby providing parameter support for further execution of the method in this embodiment.

[0025] Among them, the value of each data content is calculated based on the above formula, and the normalization factor >1; Identify the data content of each partition storage interval in the cloud database, continuously monitor the activity of each data content in each partition storage interval, and iterate the data content stored in each partition storage interval in the cloud database based on the data content activity monitoring result; Create a recommended content data package and feed back the recommended content data package to the user end in real time; Monitoring the reading status information of the recommended content data packet read by the user end, and moving the data content in the recommended content data packet based on the reading status information; The operations for setting up storage partitions in the cloud database are: Obtain the analysis results of each data content based on the value analysis logic, capture the maximum value of the analysis result, set the number of differentiated storage intervals, and further set the interval for storing the target data content corresponding to each differentiated storage interval as follows: ; Where: The maximum value of the data content value analysis result; Set the quantity to distinguish the storage intervals; The maximum value of the data content stored in each partition is limited to not included; The activity monitoring logic of data stored in different storage intervals in the cloud database is expressed as: Monitor whether any three of the data content reading times, the total number of data content reading users, the data content reading frequency, the data content download times, and the data content retrieval times among the relevant parameters of the data content are continuously reduced. If the monitoring result is yes, the data content is determined to be inactivated. If the monitoring result is no, the data content is determined to be still active. If the same data content is determined to be inactivated twice in a row, the data content is taken as an iteration target and an iteration operation is performed. Among them, the iterative operation of data content and the operation of deleting data content in the partitioned storage interval where it is located; The creation logic of the recommended content data package is: Obtaining historical activity determination results of each piece of data content in each storage interval of each partition in the cloud database, accumulating and measuring the activity determination results of each piece of data content, arranging each piece of data content in descending order based on the total amount of the accumulated measurement of the determination results of each piece of data content being active in the measurement results, setting the number of data content pieces included in the recommended content data packet, and picking up a corresponding number of data content pieces at the front position in the data content descending order queue based on the set number of pieces, so as to form a recommended content data packet; The read status information includes: the single read time of the data content, the number of reads of the data content, and the operation of moving the data content in the recommended content data package is: Obtain the reading status information of each data in the recommended content data package, calculate the cumulative reading time of each data in the recommended content data package, select the data content with the longest cumulative reading time as the moving target, and move the data content to the parent partition storage interval of the partition storage interval where it is located.

[0026] By executing the method in the above embodiment, a new storage and management service is provided for the operation data, providing effective auxiliary support for the subsequent use of the operation data.

[0027] Embodiment 2: The data source network is the network used by the user for daily operations. When deploying web crawlers, the server in the data source network decides the number of web crawlers to be deployed based on the amount of data generated by the server operation. The number of web crawlers deployed in each server shall not be less than two. Among them, when deciding the number of web crawlers deployed on the server, the following is followed: a data statistical period is set, and the number of data generated by the server within the data statistical period is counted based on the data statistical period, and the statistical results are averaged based on the data statistical period. The larger the corresponding average result of the server is, the more web crawlers are deployed in the server, and vice versa, the fewer web crawlers are deployed in the server; The activity monitoring logic for data stored in different storage intervals in the cloud database is expressed as: Monitor whether any three of the data content reading times, the total number of data content reading users, the data content reading frequency, the data content download times, and the data content retrieval times among the relevant parameters of the data content are continuously reduced. If the monitoring result is yes, the data content is determined to be inactivated. If the monitoring result is no, the data content is determined to be still active. If the same data content is determined to be inactivated twice in a row, the data content is taken as an iteration target and an iteration operation is performed. Among them, the iterative operation of data content and the operation of deleting data content in the partitioned storage interval where it is located; The recommended content data packet is transmitted to the user terminal through the data source network. When the recommended content data packet is transmitted to the user terminal, the mobile computer device held by the user terminal is used as the transmission target to perform the transmission operation of the recommended content data packet.

[0028] Through the above settings, further execution data support is provided for the execution of the method in Example 1, thereby ensuring the stable execution of the method in Example 1.

[0029] In summary, during the execution of the method in the above embodiment, multiple web crawlers are set up to capture the data content generated in real time in the operation network, and further value analysis is performed on the data content, thereby realizing differentiated storage of the data content, and at the same time configuring data iteration logic and recommendation logic to provide convenience for users who call to read the operation data, thereby effectively ensuring the convenience of the operation data during the use stage.

[0030] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operation data management method based on artificial intelligence, characterized in that: include: Determine a data source network, deploy a web crawler on each server in the data source network, and capture the data content generated by each server in the data source network in real time based on the deployed web crawler; Traverse all captured data content, identify relevant parameters of each data content, analyze the value of the data content based on the relevant parameters of the data content, create a cloud database, set up separate storage intervals in the cloud database, and store the data content in separate storage intervals based on the results of the data content value analysis; Identify the data content of each partition storage interval in the cloud database, continuously monitor the activity of each data content in each partition storage interval, and iterate the data content stored in each partition storage interval in the cloud database based on the data content activity monitoring result; Create a recommended content data package and feed back the recommended content data package to the user end in real time; The reading status information of the recommended content data packet read by the user end is monitored, and the data content in the recommended content data packet is moved based on the reading status information.

2. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The data source network is a network used by the user end for daily operations. When deploying web crawlers, the server in the data source network decides the number of web crawlers to be deployed based on the amount of data generated by the server operation, and the number of web crawlers deployed in each server is not less than two; Among them, when deciding the number of web crawlers deployed on the server, the following is followed: a data statistical period is set, the number of data items generated by the server within the data statistical period is counted based on the data statistical period, and the statistical results are averaged based on the data statistical period. The larger the corresponding average result of the server, the more web crawlers are deployed in the server, and vice versa, the fewer web crawlers are deployed in the server.

3. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The web crawlers deployed in each of the servers are controlled to run according to a user-defined data capture cycle. After the web crawlers deployed in each of the servers run according to the data capture cycle and reach the data capture cycle, the data contents captured by each web crawler are summarized and a cleaning operation is performed on the data contents. ； Where: is the similarity of two data; is the distance between two data after being converted into Hamming sequence; is the total length of the sequence; Among them, based on the above formula, the similarity calculation is performed on all data contents captured by the web crawler in groups of two. When the calculation result is 1, the two data are determined to be the same data, and any one of the two identical data is further selected to be deleted, and the data content cleaning operation is completed based on the above.

4. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The data content related parameters include: the number of data content reads, the total number of data content read users, the frequency of data content reads, the number of data content downloads, and the number of data content retrievals; The data content value analysis logic is expressed as: ； Where: The value of data content; is the normalization factor; The number of times the data content is read; Read the total number of users for data content; Read frequency for data content; The number of times the data content is downloaded; The number of times the data content is retrieved; Among them, the value of each data content is calculated based on the above formula, and the normalization factor >

1.

5. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The operations for setting up separate storage intervals in the cloud database are: Obtain the analysis results of each data content based on the value analysis logic, capture the maximum value of the analysis result, set the number of differentiated storage intervals, and further set the interval for storing the target data content corresponding to each differentiated storage interval as follows: ; Where: The maximum value of the data content value analysis result; Set the quantity to distinguish the storage intervals; The maximum value of the data content stored in each partition is limited to not included.

6. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The activity monitoring logic for distinguishing the data stored in the storage interval in the cloud database is expressed as: Monitor whether any three of the data content reading times, the total number of data content reading users, the data content reading frequency, the data content download times, and the data content retrieval times among the relevant parameters of the data content are continuously reduced. If the monitoring result is yes, the data content is determined to be inactivated. If the monitoring result is no, the data content is determined to be still active. If the same data content is determined to be inactivated twice in a row, the data content is taken as an iteration target and an iteration operation is performed. Among them, the iterative operation of the data content and the operation of deleting the data content in the partitioned storage interval where the data content is located.

7. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The recommended content data package is integrated from data contents stored in storage intervals of each partition in at least three cloud databases.

8. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The creation logic of the recommended content data package is: Obtain historical activity determination results of each piece of data content in each partitioned storage interval in the cloud database, perform cumulative measurement on the activity determination results of each piece of data content, arrange each piece of data content in descending order based on the cumulative measurement amount of the determination results of each piece of data content being active in the measurement results, set the number of data content pieces included in the recommended content data packet, and pick up a corresponding number of data content pieces at the front position in the data content descending order queue based on the set number of pieces to form a recommended content data packet.

9. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The recommended content data packet is transmitted to the user terminal through the data source network. When the recommended content data packet is transmitted to the user terminal, the mobile computer device held by the user terminal is used as the transmission target to perform the transmission operation of the recommended content data packet.

10. The operation data management method based on artificial intelligence according to claim 1, characterized in that: The reading status information includes: a single reading time of the data content, a number of reading times of the data content, and the operation of moving the data content in the recommended content data packet is: Obtain the reading status information of each data in the recommended content data package, calculate the cumulative reading time of each data in the recommended content data package, select the data content with the longest cumulative reading time as the moving target, and move the data content to the parent partition storage interval of the partition storage interval where it is located.

Images