Automatic data circulation method and system

Abstract

The invention provides an automatic data circulation method. The method comprises the following steps: acquiring initial data; based on a data value evaluation model, performing value evaluation on the initial data to obtain a value score of the initial data; and based on the change of the value score, performing periodic scheduling on the upgrading and downgrading operation and the compression archiving operation of the initial data to complete automatic circulation. The invention further provides an automatic data circulation system. According to the method, the data management efficiency is improved, and the flowability, the measurability and the intelligent analysis capability of the data are realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically to a method and system for automatic data circulation. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, enterprises are facing unprecedented data challenges. Various business systems continue to generate massive amounts of data, and enterprises often have redundancy in data storage.

[0003] Many companies have accumulated a large amount of historical data, but due to the lack of effective data value assessment standards, these data are often idle and cannot be properly classified, which also causes a huge waste of corporate resources. Secondly, in the actual application of data, it is difficult for companies to evaluate the true value of these data. The lack of objective evaluation methods leads to low efficiency in data use, which further affects the scientificity and timeliness of decision-making. In addition, many companies currently rely on manual operations in data governance and lack systematic and automated management tools. This method of relying on manual operations is not only inefficient, but also easily leads to delays in data updates and maintenance.

[0004] Therefore, there is an urgent need for a method to solve problems such as data value assessment, data classification management and automated governance in the data management process. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a method and system for automatic data circulation, which improves the efficiency of data management and realizes the fluidity, measurability and intelligent analysis capabilities of data.

[0006] According to the first aspect of the present disclosure, a method for automated data circulation is provided, the method comprising: obtaining initial data; performing value assessment on the initial data based on a data value assessment model to obtain a value score of the initial data; and periodically scheduling upgrade and downgrade operations and compression and archiving operations of the initial data based on changes in the value score to complete automated circulation.

[0007] According to an embodiment of the present disclosure, the method further includes: monitoring the consumption process of the initial data, and issuing a warning message when abnormal access to the initial data occurs.

[0008] According to an embodiment of the present disclosure, the initial data includes text data, picture data and video data, and the data value assessment model includes a text data value assessment model, a picture data value assessment model and a video data value assessment model.

[0009] According to an embodiment of the present disclosure, based on a text data value assessment model, a value assessment is performed on text data to obtain a value score of the text data, including: converting the text data into a dense vector representation by using word embedding technology; inputting the dense vector representation into a neural network model of an attention mechanism to learn the importance weights of text data keywords; based on the importance weights of text data keywords, according to a called frequency index, a counted frequency index, and an updated frequency index of the text data, the value score of the text data is calculated.

[0010] According to an embodiment of the present disclosure, a value assessment is performed on image data based on an image data value assessment model to obtain a value score of the image data, including: extracting visual features of the image data based on a convolutional neural network; inputting the visual features of the image data into the image data value assessment model, wherein the parameters of the image data value assessment model include image metadata and access behavior data; and calculating the value score of the image data based on the access volume indicator and the download volume indicator of the image data.

[0011] According to an embodiment of the present disclosure, based on a video data value assessment model, a value assessment is performed on video data to obtain a value score of the video data, including: extracting time series features of the video data through a timing model; inputting the time series features of the video data into the video data assessment model, wherein the parameters of the video data assessment model include video metadata and user interaction data; and calculating the value score of the video data based on a playback duration indicator and an interaction volume indicator of the video data.

[0012] According to an embodiment of the present disclosure, based on the change of the value score, the upgrading and compression archiving of the initial data are periodically scheduled to complete the automated circulation, including: when the value score of the initial data is higher than a preset first threshold, the format of the initial data is upgraded to a usage format; when the value score of the initial data is lower than the preset first threshold, the format of the initial data is downgraded to a compressed format.

[0013] According to an embodiment of the present disclosure, based on the change of the value score, the upgrading and downgrading and compressed archiving of the initial data are periodically scheduled to complete the automated circulation, and also includes: when the value score of the initial data is lower than a preset first threshold, the initial data is compressed or archived, and the compressed or archived initial data is transferred to a storage medium.

[0014] The second aspect of the present disclosure provides a data automation circulation system, which includes: a data acquisition module for acquiring initial data; a data evaluation module for performing value evaluation on the initial data based on a data value evaluation model to obtain a value score of the initial data; and a data cycle management module for periodically scheduling the upgrade and downgrade operations and compression and archiving operations of the initial data based on changes in the value score to complete the automated circulation.

[0015] According to an embodiment of the present disclosure, the system further includes a consumption monitoring module for monitoring the consumption process of the initial data and issuing an early warning message when abnormal access to the initial data occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart of a method for automatic data transfer according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 2 The framework diagram of the data automation flow system according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer-readable storage medium storing instructions, on which executable instructions are stored, which, when executed by the processor, causes the processor to perform the method of the present disclosure.

[0024] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0025] An embodiment of the present disclosure provides a method for automated data circulation, including: obtaining initial data; performing value assessment on the initial data based on a data value assessment model to obtain a value score for the initial data; and periodically scheduling upgrade and downgrade operations and compression and archiving operations of the initial data based on changes in the value score to complete automated circulation.

[0026] Through the embodiments of the present disclosure, a flexible and scalable method for automated data circulation is provided, which realizes intelligent management of data by designing a data value assessment model, dynamic lifecycle management, and real-time consumption monitoring. This method can automatically assess the value of data and dynamically adjust the storage and processing of data according to the value, greatly improving the utilization efficiency of idle data assets. At the same time, the method can also monitor the use of data assets in real time and provide data management decision support for enterprises. Compared with traditional data management methods, this embodiment has a higher level of intelligence and automation, and can be widely used in various data asset management scenarios.

[0027] Figure 1 The flowchart of the data automatic circulation method according to the embodiment of the present disclosure is schematically shown.

[0028] like Figure 1 As shown, in the embodiment of the present disclosure, the data automatic flow method, the process may further include S110~S130.

[0029] In operation S110, initial data is acquired.

[0030] Specifically, these initial data can be a large amount of unused data accumulated in enterprise data management. These data may have some problems, such as the inability to price data assets, difficulty in measuring value, and poor data flow.

[0031] For another example, these idle initial data are also often referred to as "cold data" or "static data". These data are collected or generated at a certain point in time, but are not frequently accessed or used in subsequent operations. Cold data may include expired transaction records, historical logs, old user information, etc. Due to their low frequency of use, enterprises and organizations often store them on low-cost storage media to reduce overhead. With the development of big data technology, some enterprises will use data lakes or data warehouses to effectively manage and analyze different types of data so that they can be quickly retrieved and analyzed when needed.

[0032] In the embodiment of the present disclosure, these idle initial data include text data, picture data, video data, etc. After obtaining these initial data, data value evaluation is performed, which is described in detail below.

[0033] In operation S120, the initial data is valued based on the data value evaluation model to obtain a value score of the initial data.

[0034] In an embodiment of the present disclosure, the data value assessment model includes a text data value assessment model, a picture data value assessment model and a video data value assessment model.

[0035] Specifically, based on the text data value assessment model, the text data is evaluated to obtain the value score of the text data, including: converting the text data into a dense vector representation by using word embedding technology; inputting the dense vector representation into the neural network model of the attention mechanism to learn the importance weights of the text data keywords; based on the importance weights of the text data keywords, according to the text data's called frequency index, counted frequency index and updated frequency index, the text data's value score is calculated.

[0036] Exemplarily, the acquired text data is preprocessed, such as removing invalid characters, punctuation marks, and standardizing text formats. The preprocessed text data is converted into a dense vector representation using a word embedding technology (such as Word2Vec, GloVe, or BERT).

[0037] Furthermore, a neural network model with an attention mechanism is designed. The input of the model is the dense vector representation of the text data, and the output is the importance weight of the keyword. The model structure includes an embedding layer, an attention layer, and a fully connected layer, and is trained through a back-propagation algorithm to optimize the keyword weight. Then, the dense vector of the text data is input into the neural network model. After multiple iterations of training, the model can learn the importance weight of each keyword, reflecting its relative importance in the text.

[0038] Furthermore, call frequency indicators (such as the number of API calls), statistical frequency indicators (such as the number of keyword appearances), and update frequency indicators (such as the number of content modifications) of the text data are collected.

[0039] Finally, based on the importance weight of the keyword and the above three indicators, the importance of the keyword is combined with its corresponding frequency indicator through the weighted average method to obtain the final value score of each text data.

[0040] Through the embodiments of the present disclosure, not only can the value of text data be effectively evaluated, but also the importance weight of keywords can be updated according to real-time data, with strong dynamic adaptability. By combining word embedding technology and attention mechanism, accurate value evaluation of text data can be achieved, providing an important reference for enterprises and organizations in information management and decision support.

[0041] In an embodiment of the present disclosure, based on a picture data value assessment model, a value assessment is performed on picture data to obtain a value score of the picture data, including: extracting visual features of the picture data based on a convolutional neural network; inputting the visual features of the picture data into the picture data value assessment model, wherein parameters of the picture data value assessment model include picture metadata and access behavior data; and calculating the value score of the picture data based on the access volume index and the download volume index of the picture data.

[0042] Specifically, the architecture of the image data value assessment model includes a data collection module, a feature extraction module, and a value assessment module. For example, the image data and its metadata are first collected, including shooting time, location, resolution, author information, title and tags, etc., and the user's access behavior data, such as the number of visits, downloads, and likes, are recorded.

[0043] Furthermore, based on a convolutional neural network (CNN), the image data is processed to extract visual features of the image. The specific step may include: inputting the original image data into a pre-trained CNN model. Obtaining an image feature vector from a hidden layer of the CNN, the feature vector is used to represent the high-dimensional features of the image.

[0044] Furthermore, the extracted visual features are combined with metadata and access behavior data and input into the value assessment model. The model uses machine learning algorithms, such as random forest or support vector machine, for training and prediction.

[0045] Based on the above-mentioned image data value assessment model, the extracted visual feature vector is spliced ​​with the metadata of the image and the user's access behavior data to form a comprehensive feature vector, in which the visit volume index (such as the number of visits per month) and the download volume index (such as the number of downloads) are defined and normalized to ensure the comparability of different indicators. Then the image data to be evaluated is input into the trained image data value assessment model, and the model will return the corresponding value score. For example, the value score can range from 0 to 100, and the higher the value, the higher the value of the image data.

[0046] Through the embodiments of the present disclosure, an effective image data value assessment model is established by combining the visual feature extraction technology of convolutional neural networks with multidimensional data analysis to achieve accurate value assessment of image data.

[0047] In an embodiment of the present disclosure, based on a video data value assessment model, a value assessment is performed on video data to obtain a value score of the video data, including: extracting time series features of the video data through a timing model; inputting the time series features of the video data into the video data assessment model, wherein parameters of the video data assessment model include video metadata and user interaction data; and calculating the value score of the video data based on a playback duration indicator and an interaction volume indicator of the video data.

[0048] Specifically, video data is collected, such as video files, video metadata (such as title, tags, upload time, etc.) and user interaction data (such as number of views, likes, comments, shares, etc.). The collected video data is then cleaned and standardized to ensure data consistency and integrity.

[0049] Furthermore, a time series model (such as a long short-term memory network LSTM) is used to process the video data to extract time series features. In the embodiments of the present disclosure, the time series features include but are not limited to: playback time (i.e., the total time the video is viewed), the active time period of the viewing user, the time distribution of user interaction, etc.

[0050] Furthermore, the extracted time series features are combined with the video metadata and user interaction data to form an input matrix. A video data evaluation model is constructed using machine learning algorithms (such as random forests, support vector machines, or deep learning models), which are trained based on historical data so that the data value of new videos can be effectively evaluated.

[0051] Furthermore, the playback time index and interaction volume index of the video data are input into the constructed video data evaluation model, and the corresponding value score is output, which reflects the overall value of the video data in the current environment.

[0052] Through the embodiments of the present disclosure, based on the above-mentioned design of corresponding value assessment models according to different data types, the value score of each initial data is obtained.

[0053] In operation S130, based on the change in the value score, the upgrade and downgrade operations and the compression and archiving operations of the initial data are periodically scheduled to complete the automatic flow.

[0054] Based on the above data value assessment results, data lifecycle scheduling is performed to achieve intelligent management of data assets, which is explained in detail below.

[0055] In the embodiment of the present disclosure, based on the change of the value score, the upgrading and downgrading and compression and archiving of the initial data are scheduled periodically to complete the automatic circulation, including: when the value score of the initial data is higher than the preset first threshold, the format of the initial data is upgraded to the use format; when the value score of the initial data is lower than the preset first threshold, the format of the initial data is downgraded to the compression format. When the value score of the initial data is lower than the preset first threshold, the initial data is compressed or archived, and the compressed or archived initial data is transferred to the storage medium.

[0056] Exemplarily, based on the value score of the initial data, it is determined whether the initial data needs to be upgraded or downgraded. In this embodiment, when the value score of the initial data is higher than the preset first threshold, that is, the initial data with high value, high frequency and depth, it is upgraded to a normal usage format to increase its access speed and usage experience; when the value score of the initial data is lower than the preset first threshold, that is, the initial data with lower value, it can be downgraded to a compressed format to save storage space. For initial data with lower value, it can be automatically compressed or archived and transferred to a storage medium with larger capacity but slower access speed to save online storage space.

[0057] Through the embodiments of the present disclosure, high-value, high-frequency data are automatically upgraded to core data assets and can be stored on high-performance storage devices; low-value, low-frequency data are automatically downgraded to non-core data assets and can be automatically compressed and archived and stored on low-cost storage devices, further improving the storage efficiency of idle data.

[0058] In the embodiments of the present disclosure, based on the automatic monitoring of data value changes, data upgrade, compression, archiving and other operations can be regularly scheduled according to a preset lifecycle management strategy to dynamically adjust data storage.

[0059] It should be noted that, in this embodiment, the process of automatic data flow transfer also includes monitoring the consumption process of the initial data, and when abnormal access to the initial data occurs, an early warning message is issued.

[0060] For example, real-time monitoring of data usage provides support for data management decisions of enterprises. Specifically, it includes: 1) Data access statistics, recording the number of visits, downloads, interactions and other indicators of various data assets, and forming detailed data consumption reports; 2) Abnormal monitoring, real-time monitoring of abnormal data access behaviors, such as sudden high visits, illegal downloads, etc., and issuing early warning information to administrators. 3) Decision support, based on the value assessment results and consumption statistics of data, provides enterprises with decision-making suggestions for data asset management, such as focusing on protecting high-value data and eliminating low-value data.

[0061] Through the embodiments of the present disclosure, the use of data assets is monitored in real time to provide data management decision support for enterprises. Compared with traditional data management methods, this embodiment has a higher level of intelligence and automation, and can be widely used in various data asset management scenarios. In addition, the method realizes automatic classification, compression, and archiving of data by introducing rule systems and machine learning technologies, and supports real-time data conversion monitoring. At the same time, the method can support two-way automatic flow from the big data platform to the data asset platform, improve the utilization rate of data and the input-output ratio, and realize efficient data management.

[0062] According to the above-mentioned data automatic circulation method, the present disclosure also provides a data automatic circulation system.

[0063] like Figure 2 As shown, Figure 2 The framework diagram of the data automation flow system according to the embodiment of the present disclosure is schematically shown.

[0064] like Figure 2 As shown, the data automatic circulation system 200 of this embodiment includes a data acquisition module 210 , a data evaluation module 220 and a data cycle management module 230 .

[0065] The data acquisition module 210 is used to acquire initial data. In one embodiment, the data acquisition module 210 can be used to perform the operation S110 described above, which will not be described in detail here.

[0066] The data evaluation module 220 is used to perform a value evaluation on the initial data based on the data value evaluation model to obtain a value score of the initial data. In one embodiment, the data evaluation module 220 can be used to perform the operation S120 described above, which will not be described in detail here.

[0067] The data cycle management module 230 is used to periodically schedule the upgrade and downgrade operations and the compression and archiving operations of the initial data based on the changes in the value score to complete the automatic circulation. In one embodiment, the data cycle management module 230 can be used to perform the operation S130 described above, which will not be repeated here.

[0068] In the embodiment of the present disclosure, the data automatic circulation system further includes a consumption monitoring module 240 for monitoring the consumption process of the initial data and issuing an early warning message when abnormal access to the initial data occurs.

[0069] Through the embodiments of the present disclosure, the system realizes intelligent management of data by designing a data value assessment model, dynamic lifecycle management and real-time consumption monitoring. The system can automatically assess the value of data and dynamically adjust the storage of data according to the value, greatly improving the utilization efficiency of idle data assets.

[0070] The flow chart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the system and method of various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0071] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0072] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for automatic data transfer, characterized in that: The method comprises: Get initial data; Based on the data value assessment model, the initial data is value assessed to obtain a value score of the initial data; Based on the change of the value score, the upgrading and downgrading operations and the compression and archiving operations of the initial data are periodically scheduled to complete the automatic flow.

2. The data automatic circulation method according to claim 1, characterized in that: The method further comprises: The consumption process of the initial data is monitored, and when abnormal access to the initial data occurs, an early warning message is issued.

3. The data automatic circulation method according to claim 1, characterized in that: The initial data includes text data, picture data and video data, and the data value assessment model includes a text data value assessment model, a picture data value assessment model and a video data value assessment model.

4. The data automatic circulation method according to claim 1 or 3, characterized in that: Based on the text data value assessment model, the text data is value assessed to obtain a value score of the text data, including: By using word embedding technology, the text data is converted into a dense vector representation; Inputting the dense vector representation into a neural network model of an attention mechanism to learn the importance weights of the keywords in the text data; Based on the importance weights of the keywords in the text data, the value score of the text data is calculated according to the call frequency index, the counted frequency index and the updated frequency index of the text data.

5. The data automatic circulation method according to claim 1 or 3, characterized in that: Based on the image data value assessment model, the image data is value assessed to obtain a value score of the image data, including: Extracting visual features of the image data based on a convolutional neural network; Inputting the visual features of the image data into an image data value assessment model, wherein parameters of the image data value assessment model include image metadata and access behavior data; Based on the visit volume index and the download volume index of the image data, the value score of the image data is calculated.

6. The data automatic circulation method according to claim 1 or 3, characterized in that: Based on the video data value assessment model, the video data is value assessed to obtain a value score of the video data, including: Extracting time series features of the video data through a time series model; Inputting the time series features of the video data into a video data evaluation model, wherein parameters of the video data evaluation model include video metadata and user interaction data; Based on the playback time index and the interaction volume index of the video data, the value score of the video data is calculated.

7. The data automatic circulation method according to claim 1, characterized in that: Based on the change of the value score, the initial data is periodically scheduled for upgrading and downgrading and compressing and archiving to complete the automatic circulation, including: When the value score of the initial data is higher than a preset first threshold, upgrading the format of the initial data to a usage format; When the value score of the initial data is lower than a preset first threshold, the format of the initial data is downgraded to a compressed format.

8. The data automatic circulation method according to claim 1 or 7, characterized in that: The periodic scheduling of upgrading and compressing the initial data based on the change of the value score to complete the automatic circulation also includes: When the value score of the initial data is lower than a preset first threshold, the initial data is compressed or archived, and the compressed or archived initial data is transferred to a storage medium.

9. A data automatic circulation system, characterized in that: The system comprises: A data acquisition module, used to acquire initial data; A data evaluation module, used to perform a value evaluation on the initial data based on a data value evaluation model to obtain a value score of the initial data; The data cycle management module is used to periodically schedule the upgrade and downgrade operations and the compression and archiving operations of the initial data based on the changes in the value score to complete the automatic flow.

10. The data automatic circulation system according to claim 9, characterized in that: The system further comprises a consumption monitoring module for monitoring the consumption process of the initial data and issuing an early warning message when abnormal access to the initial data occurs.

Images