Data asset management method and system

By designing a data asset management system, the complexity and security issues of Internet data are solved, and efficient integration, analysis and security management of different types of data are achieved, providing support for the business decisions of newspapers.

CN119088900AInactive Publication Date: 2024-12-06GUANGDONG NANFANG NEWSPAPER MEDIA GRP NEW MEDIA CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410943458.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The complexity and diversity of Internet data have led to the challenges of data asset management in acquisition, integration, storage and analysis. At the same time, how to ensure data security avoids data breaches and abuse is also an urgent problem.

Method used

A data asset management system was designed, including data acquisition unit, data preprocessing unit, data analysis unit, data application unit, data display unit and server unit, and obtain different types of target data (structured, unstructured, video preprocessing, analysis and display, and judge and warning in combination with preset safety threshold logic.

Benefits of technology

It has achieved efficient integration and analysis of massive data, supported the data security management and early warning mechanism, and provided strong support for the newspaper's business decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088900A_ABST
    Figure CN119088900A_ABST
Patent Text Reader

Abstract

The invention discloses a data asset management method and system, and the method comprises the steps: obtaining different types of target data, carrying out the preprocessing of the target data, enabling the target data to at least comprise structured data, unstructured data, video data and specific data, and enabling the specific data to comprise data obtained through different specific channels; performing correlation calculation on the pre-processed target data, establishing a data analysis system according to a calculation result, and judging target real-time data in the pre-processed target data in combination with preset safety threshold logic; and performing corresponding keyword analysis and early warning according to the keywords input by the user side in combination with the data analysis system and the judgment result. Not only can mass data be efficiently integrated and analyzed, but also a safety management and early warning mechanism of the data is supported, and powerful support is provided for business decision-making of newspaper offices.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of newspaper data asset management, and in particular to a data asset management method and system. Background Art

[0002] The complexity and diversity of Internet data have brought great challenges to data asset management. On the one hand, how to effectively acquire, integrate, store and analyze this data to support newspapers' business decisions and operational optimization; on the other hand, how to ensure data security and avoid data leakage and abuse have become urgent issues to be solved in newspaper data asset management. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a data asset management method and system, which can solve the problems mentioned in the background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a data asset management system, comprising:

[0008] A data acquisition unit, a data preprocessing unit, a data analysis unit, a data application unit, a data display unit and a server unit; the data acquisition unit is used to acquire different types of target data and transmit the target data to the data preprocessing unit, the target data includes target historical data and target real-time data, and the different types include at least structured data, unstructured data and video data;

[0009] The data preprocessing unit is used to preprocess different types of target data transmitted by the data acquisition unit, and transmit the preprocessing results to the data analysis unit and the data application unit, wherein the preprocessing at least includes data deduplication, data error correction and processing of video data;

[0010] The data analysis unit is used to calculate and establish a data analysis system according to the preprocessed target historical data transmitted by the data preprocessing unit, and verify the data analysis system through the preprocessed target real-time data, and save the verified data analysis system to the server unit;

[0011] The data application unit is used to save the preset security threshold logic, and determine whether the target real-time data has hot spots or public opinion data. If so, corresponding processing is performed through topic tracking and prediction control, and the judgment results, hot topics and public opinion monitoring results are saved to the server unit;

[0012] The data display unit is used to retrieve the data analysis system, judgment results, hot topics and public opinion monitoring result data stored in the server unit, and perform data analysis and early warning according to the keywords input by the user;

[0013] The server unit is used to store data and provide callable data.

[0014] As a preferred solution of the data asset management system described in the present invention, wherein: the data acquisition unit includes a structured data acquisition module, an unstructured data acquisition module, a video data acquisition module and a specific data acquisition module;

[0015] The structured data acquisition module is used to acquire structured data from the Internet or a fixed data resource platform, wherein the structured data at least includes fixed website data, Weibo data, WeChat data, and digital newspaper data. After acquiring the structured data, the structured data is transmitted to the data preprocessing unit through a first preset communication interface;

[0016] The unstructured data acquisition module is used to acquire unstructured data from the Internet or a fixed data resource platform, wherein the unstructured data includes semi-structured data and unstructured data, and the unstructured data also includes at least forum data and blog data. After the unstructured data is acquired, the unstructured data is transmitted to the data preprocessing unit through a second preset communication interface;

[0017] The video data acquisition module is used to acquire video data from the Internet or a fixed data resource platform, wherein the video data includes video data within a preset time period and data within a preset time range, and after acquiring the video data, the video data is transmitted to the data preprocessing unit through a third preset communication interface;

[0018] The specific data acquisition module is used to acquire data from a media-specific offline data source, wherein the media-specific offline data source includes at least newspaper topic selection, reader reports, journalist surveys, and online questionnaires. After acquiring the specific data, the specific data is transmitted to the data preprocessing unit through a fourth preset communication interface.

[0019] The fixed data resource platform at least includes an official statistical data platform, an academic research database, and a historical archive database;

[0020] The structured data acquisition module, the unstructured data acquisition module, the video data acquisition module and the specific data acquisition module are respectively communicatively connected with the data preprocessing unit.

[0021] As a preferred solution of the data asset management system described in the present invention, wherein: the data pre-processing unit includes a data cleaning module, a data deduplication module, a data error correction module and a video data processing module;

[0022] The data cleaning module is used to obtain data transmitted from the structured data acquisition module, the unstructured data acquisition module and the specific data acquisition module, and perform data cleaning on the transmitted data, and transmit the data after the data cleaning operation to the data deduplication module;

[0023] The data deduplication module is used to perform a deduplication operation on the data after the data cleaning operation by the data cleaning module, and transmit the data after the deduplication operation to the data error correction module;

[0024] The data error correction module is used to perform an error correction operation on the data after the deduplication operation of the data error correction module, and transmit the data after the error correction operation to the data analysis unit;

[0025] The video data processing module is used to obtain the video data transmitted by the video data acquisition module, and perform transcoding, key frame extraction, voice-to-text conversion and image-to-text conversion on the video data;

[0026] The data cleaning module, the data deduplication module, and the data error correction module are communicatively connected in sequence, the data cleaning module is communicatively connected with the structured data acquisition module, the unstructured data acquisition module, and the specific data acquisition module, and the video data processing module is communicatively connected with the video data acquisition module;

[0027] The video data processing module is in communication connection with the data cleaning module, and the communication connection interface is in a disconnected state. When the video data processing module generates processed video data, the communication connection interface enters a closed connection state;

[0028] The data cleaning module, the data deduplication module, the data error correction module and the video data processing module are respectively connected to the data analysis unit, the data application unit and the server unit for communication.

[0029] As a preferred solution of the data asset management system described in the present invention, wherein: the data analysis unit includes a keyword analysis module, a similarity analysis module, a word relationship building module and a label system building module;

[0030] The keyword analysis module is used to extract keywords from the target historical data transmitted by the data preprocessing unit according to a preset keyword extraction logic, and transmit the extracted keywords to the similarity analysis module, the word relationship building module and the label system establishment module;

[0031] The similarity analysis module is used to perform similarity calculation on the keywords transmitted by the keyword analysis module according to a preset similarity analysis algorithm, and transmit the similarity calculation results in a structured form to the word relationship building module;

[0032] The word relationship building module is used to build word relationships according to the keywords transmitted by the keyword analysis module and the similarity calculation results transmitted by the similarity analysis module, and transmit the word relationship building results to the label system establishment module;

[0033] The label system establishment module is used to establish a label system according to the keywords transmitted by the keyword analysis module and the word relationship building results transmitted by the word relationship building module, and verify the established label system according to the target real-time data transmitted by the data preprocessing unit, and transmit the verified label system to the server unit;

[0034] The data analysis system includes keyword analysis, similarity analysis, word relationship building and label system establishment;

[0035] The keyword analysis module is respectively connected to the similarity analysis module, the word relationship building module and the label system building module, and the similarity analysis module, the word relationship building module and the label system building module are connected to each other in sequence;

[0036] The label system establishment module is in communication connection with the data pre-processing unit;

[0037] The label system establishing module is communicatively connected with the server unit.

[0038] As a preferred solution of the data asset management system described in the present invention, wherein: the data application unit includes a topic tracking module, a hot spot monitoring module, a public opinion monitoring module and a prediction control module;

[0039] The subject tracking module is used to track the target real-time data in the data acquisition unit in real time according to the preset subject set, and to pre-track the predicted transmission path transmitted to the prediction control module in advance;

[0040] The hotspot monitoring module is used to determine whether there is a topic that has become a hotspot in the target in the data acquisition unit in real time, and transmit the determination result to the visualization display module and the early warning notification module;

[0041] The public opinion monitoring module is used to monitor the target real-time data in the data acquisition unit according to a preset public opinion monitoring mechanism, and transmit the detection results to the visualization display module and the early warning notification module;

[0042] The prediction control module is used to establish a prediction control model based on the machine learning algorithm according to the historical data stored in the server unit, input the target real-time data into the prediction control model to predict the data transmission path of the target real-time data, and send the prediction result to the early warning notification module to perform prediction control on public opinion;

[0043] The topic tracking module, hotspot monitoring module, public opinion monitoring module, visualization display module and early warning notification module are respectively communicated and connected;

[0044] The prediction control module is communicatively connected with the early warning notification module;

[0045] The data application unit is communicatively connected with the server unit.

[0046] As a preferred solution of the data asset management system described in the present invention, wherein: the data display unit includes a visualization display module, a keyword search module and an early warning notification module;

[0047] The visualization display module is used to display the topic tracking results, hot spot monitoring results, public opinion monitoring results and prediction control results transmitted by the data application unit in real time, and to display the corresponding keyword tag system in response to the user's keyword search operation;

[0048] The keyword search module is used for users to search for target keywords, and the keyword search results are displayed through the visual display module;

[0049] The warning notification module is used to issue different warning notifications for different states of the topic tracking module, hot spot monitoring module and public opinion monitoring module, and at the same time display the warning information through the visual display module;

[0050] The visual display module is communicatively connected with the keyword retrieval module;

[0051] The visualization display module and the warning notification module are communicatively connected with the data application unit;

[0052] The data display unit is communicatively connected with the server unit.

[0053] As a preferred solution of the data asset management system described in the present invention, wherein: the server unit includes a data storage module and a data calling module;

[0054] The data storage module is used to store the data acquired by the data acquisition unit and the data generated by other modules;

[0055] The data to be called module is used to convert the data in the data storage module into callable data.

[0056] In a second aspect, the present invention provides a data asset management method, comprising:

[0057] Acquire different types of target data and preprocess the target data, wherein the target data includes target historical data and target real-time data, and the different types include at least structured data, unstructured data, video data and specific data, and the specific data includes data acquired through different specific channels;

[0058] Calculate the similarity of the target data after preprocessing, establish a data analysis system based on the calculation results, and judge the target real-time data in the target data after preprocessing in combination with the preset safety threshold logic;

[0059] Based on the keywords input by the user, combined with the data analysis system and judgment results, corresponding keyword analysis and early warning are carried out.

[0060] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when executing the computer program.

[0061] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described above when executed by a processor.

[0062] Compared with the prior art, the present invention proposes a data asset management method and system, which obtains different types of target data and preprocesses the target data, wherein the target data at least includes structured data, unstructured data, video data and specific data, and the specific data includes data obtained through different specific channels; performs correlation calculation on the target data after preprocessing, establishes a data analysis system based on the calculation results, and judges the target real-time data in the target data after preprocessing in combination with the preset security threshold logic; performs corresponding keyword analysis and early warning based on the keywords input by the user end in combination with the data analysis system and the judgment results. It can not only efficiently integrate and analyze massive data, but also support data security management and early warning mechanisms, providing strong support for the business decision-making of newspapers. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1A schematic diagram of the system structure of a data asset management method and system provided by one embodiment of the present invention;

[0064] Figure 2 A method flow chart of a data asset management method and system provided by an embodiment of the present invention;

[0065] Figure 3 An internal structural diagram of a computer device of a data asset management method and system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0067] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0068] In the first embodiment of the present invention, referring to Figure 1-Figure 3 This embodiment provides a data asset management method and system, including a data asset management method and a data asset management system, wherein a data asset management system includes:

[0069] Before describing the embodiments of the present application in detail, some related concepts are first explained for the sake of clarity.

[0070] Tagging system: In the field of data management and analysis, the tagging system refers to an organizational framework that uses a series of keywords or phrases to mark and classify data. It helps users quickly locate and understand data content and improves data search efficiency and accuracy. For example, in news articles, social media posts, or any type of information management, content analysis can automatically or manually assign corresponding tags, such as "technology", "finance", "sports", etc. These tags form a hierarchical tagging system that facilitates content classification, retrieval, and recommendation.

[0071] Public opinion monitoring: Public opinion monitoring refers to the process of using information technology to monitor, collect, analyze and manage public sentiment, opinions, attitudes and discussion trends on a specific topic or brand on the Internet. It helps companies and government agencies to timely understand the public's feedback on an event, policy or product, evaluate the public opinion environment, and prevent and respond to public relations crises. Public opinion monitoring usually involves natural language processing, sentiment analysis, topic modeling and other technologies to extract valuable information from massive network data in an automated way.

[0072] Hotspot monitoring: Hotspot monitoring refers to the technology and process of identifying and tracking information or topics that are highly popular and spread quickly on the Internet within a certain period of time. These hotspots can be emergencies, hot events, popular topics, etc. Hotspot monitoring is crucial for news media, marketing, advertising and other fields because it can help companies or institutions quickly capture public interest points, adjust strategies to seize opportunities, increase exposure or guide public opinion.

[0073] Topic tracking: Topic tracking is a technology in the field of information monitoring and analysis, which focuses on continuously monitoring and analyzing the evolution and development of a specific theme or topic at different times and from different sources (such as news reports, social media, forums, etc.). It not only includes identifying the current hot topics, but also involves in-depth analysis of the changes in the content of the topic, the relevant people, the development context of the event, and the changes in the public's attitude towards it. Topic tracking helps to gain in-depth insights into trends, predict future development directions, and provide support for decision-making.

[0074] There are many problems in related technologies, such as scattered data, diverse formats, low processing efficiency, and lack of effective data integration and early warning mechanisms.

[0075] This application provides a method that can effectively solve the above-mentioned problems. Next, we will combine multiple embodiments to explain in detail how to implement the data asset management system design;

[0076] Figure 1 The system structure diagram of a data asset management method and system is shown, including: a data acquisition unit 100, a data preprocessing unit 200, a data analysis unit 300, a data application unit 400, a data display unit 500 and a server unit 600;

[0077] The data acquisition unit 100 is used to acquire different types of target data and transmit the target data to the data preprocessing unit 200. The target data includes target historical data and target real-time data.

[0078] Specifically, the data preprocessing unit 200 is used to preprocess different types of target data transmitted by the data acquisition unit 100, and transmit the preprocessing results to the data analysis unit 300 and the data application unit 400. The preprocessing includes at least data cleaning, data deduplication, data error correction and processing of video data;

[0079] Furthermore, the data analysis unit 300 is used to calculate and establish a data analysis system according to the preprocessed target historical data transmitted by the data preprocessing unit 200, and verify the data analysis system through the target real-time data, and save the verified data analysis system to the server unit 600;

[0080] Furthermore, the data application unit 400 is used to save the preset security threshold logic, and determine whether the target real-time data determines whether there is hot spot or public opinion data. If so, corresponding processing is performed through topic tracking and prediction control, and the data generated by the entire judgment process is saved to the server unit 600;

[0081] Furthermore, the data display unit 500 is used to retrieve the data analysis system stored in the server unit 600 and the data generated by different judgment processes, and perform relevant analysis and early warning according to the keywords input by the user;

[0082] Furthermore, the server unit 600 is used to store data generated by each module and provide callable data for each module.

[0083] In the embodiment of the present invention, the data acquisition unit 100 includes a structured data acquisition module 101, an unstructured data acquisition module 102, a video data acquisition module 103 and a specific data acquisition module 104;

[0084] It should be noted that the structured data acquisition module 101 is used to acquire structured data from the Internet or a fixed data resource platform, and the structured data at least includes fixed website data, Weibo data, WeChat data, digital newspaper data, and other APP data. After the structured data is acquired, the structured data is transmitted to the data preprocessing unit 200 through the first preset communication interface;

[0085] It should be noted that the unstructured data acquisition module 102 is used to acquire unstructured data from the Internet or a fixed data resource platform. The unstructured data includes semi-structured data and unstructured data. The unstructured data also includes at least forum data and blog data. After the unstructured data is acquired, the unstructured data is transmitted to the data preprocessing unit 200 through the second preset communication interface.

[0086] It should be noted that the video data acquisition module 103 is used to acquire video data from the Internet or a fixed data resource platform, the video data including video data within a fixed time period and data within a fixed time range, and after acquiring the video data, the video data is transmitted to the data preprocessing unit 200 through the third preset communication interface;

[0087] It should be noted that the specific data acquisition module 104 is used to acquire data from media-specific offline data sources, which at least include newspaper topic selection, reader reports, journalist surveys, and online questionnaires. After acquiring the specific data, the specific data is transmitted to the data preprocessing unit 200 through the fourth preset communication interface.

[0088] It should be noted that the structured data acquisition module 101 , the unstructured data acquisition module 102 , the video data acquisition module 103 and the specific data acquisition module 104 are respectively communicatively connected to the data preprocessing unit 200 .

[0089] In a specific embodiment of the present invention, the fixed data resource platform can be an official statistical data platform, that is, official statistical data released by the government, international organizations or industry associations, such as census data, economic indicators, industry reports, etc. It can also be an academic research database, that is, a database of academic research results such as scientific journals, conference papers, and dissertations, such as PubMed, IEEE Xplore, JSTOR, etc. It can also be a historical archive database, that is, a platform for storing historical documents, old newspapers, ancient books, historical images, and audio and video materials, such as the National Library Digital Collection, Europeana (European Digital Cultural Heritage Platform), etc.

[0090] In the embodiment of the present invention, the first preset communication interface is used for data transmission between the structured data acquisition module (101) and the data preprocessing unit (200). Since structured data (such as database records) are usually in a fixed format and easy to process, this interface may adopt a standardized data exchange format such as JSON, XML or CSV, and transmit through the HTTP / HTTPS protocol to ensure the structural integrity and transmission security of the data.

[0091] The second preset communication interface serves the data exchange between the unstructured data acquisition module (102) and the data preprocessing unit (200). There are many types of unstructured data (such as text files, pictures, audio, etc.). This interface may adopt a more flexible transmission protocol, such as FTP for large file transmission, or WebSocket for two-way communication of real-time data streams. In addition, the data may need to be standardized in format before preprocessing through a specific API to meet the needs of the preprocessing module.

[0092] The third preset communication interface is specially designed between the video data acquisition module (103) and the video data processing module (204). Considering the large amount of video data and the complex processing, this interface may adopt streaming media protocols such as RTMP (Real-Time Messaging Protocol) or HLS (HTTP Live Streaming) to achieve real-time or near real-time video data transmission. It may also include specific video encoding format requirements, such as H.264 / H.265, to reduce the transmission bandwidth demand and speed up the processing process.

[0093] The fourth preset communication interface is the communication between the specific data acquisition module (104) and the data preprocessing unit (200). Since such data may come from a specific channel or have customized requirements, the interface design may be more flexible and changeable. For example, a RESTful API is used for data exchange, supporting customized data formats and security authentication mechanisms, ensuring that data obtained from specific sources (such as internal databases, private APIs, or customized sensor data) can be safely and efficiently integrated into the system processing flow.

[0094] In summary, establishing a data acquisition unit can ensure that the system can comprehensively and efficiently collect various types of target data, providing a solid foundation for subsequent data analysis and application.

[0095] In the embodiment of the present invention, the data pre-processing unit 200 includes a data cleaning module 201, a data deduplication module 202, a data error correction module 203 and a video data processing module 204;

[0096] It should be noted that the data cleaning module 201 is used to obtain data transmitted from the structured data acquisition module 101, the unstructured data acquisition module 102 and the specific data acquisition module 104, and clean the transmitted data, and transmit the data after the data cleaning operation to the data deduplication module 202;

[0097] It should be noted that the data deduplication module 202 is used to perform a deduplication operation on the data after the data cleaning operation by the data cleaning module 201, and transmit the deduplication-operated data to the data error correction module 203;

[0098] It should be noted that the data error correction module 203 is used to perform error correction operation on the data after the deduplication operation of the data error correction module 203, and transmit the data after the error correction operation to the data analysis unit 300;

[0099] It should be noted that the video data processing module 204 is used to obtain the video data transmitted by the video data acquisition module 103, and perform transcoding, key frame extraction, voice-to-text conversion, and image-to-text conversion on the video data;

[0100] It should be noted that the data cleaning module 201, the data deduplication module 202, and the data error correction module 203 are communicatively connected in sequence, the data cleaning module 201 is communicatively connected with the structured data acquisition module 101, the unstructured data acquisition module 102, and the specific data acquisition module 104, and the video data processing module 204 is communicatively connected with the video data acquisition module 103;

[0101] It should be noted that the video data processing module 204 is in communication connection with the data cleaning module 201, and the communication connection interface is in a disconnected state. When the video data processing module 204 generates processed video data, the communication connection interface enters a closed connection state.

[0102] It should be noted that the data cleaning module 201 , the data deduplication module 202 , the data error correction module 203 and the video data processing module 204 are respectively connected to the data analysis unit 300 , the data application unit 400 and the server unit 600 in communication.

[0103] In a specific embodiment of the present invention, the specific operation steps of the data cleaning module may be:

[0104] First, data is received from the structured data acquisition module, the unstructured data acquisition module and the specific data acquisition module, and different data formats (such as CSV, JSON, XML or custom formats) are parsed and recognized.

[0105] Secondly, check whether there are missing values ​​(empty values ​​or NULL) in the data, and handle these empty values ​​according to the preset strategy, such as filling with default values, deleting records containing empty values, or using interpolation methods.

[0106] Secondly, convert the data format to ensure the consistency and standardization of date, time, numbers and other data, such as unifying the date format to YYYY-MM-DD.

[0107] Third, identify and handle outliers, such as values ​​in numerical data that are far outside the normal range, possibly by marking or replacing them through statistical methods (such as Z-score or IQR).

[0108] Thirdly, remove irrelevant characters in the data, such as spaces and special symbols, and perform text standardization, such as case conversion and removal of stop words.

[0109] Finally, convert the data to the correct data type as required, such as converting string numbers to integers or floating-point numbers.

[0110] In a specific embodiment of the present invention, the specific operation steps of the data deduplication module may be:

[0111] First, identify duplicate data based on the key fields of the data (such as ID, date, title, etc.) using techniques such as hash tables, unique indexes, or primary key constraints.

[0112] Secondly, decide on a deduplication strategy based on business needs, such as retaining the earliest record, the latest record, or selecting the best record based on quality criteria.

[0113] Again, duplicate data is removed according to the determined strategy to ensure the uniqueness of the data set.

[0114] Finally, record the record information removed during the deduplication process to facilitate auditing and problem tracing.

[0115] In a specific embodiment of the present invention, the specific operation steps of the data error correction module may be:

[0116] First, apply preset error rule sets, such as common spelling errors, format errors, etc., for automatic error correction.

[0117] Secondly, compare the relationships between data, such as logical consistency verification, to ensure that the dependencies between data are correct.

[0118] Again, compare data with authoritative databases or third-party APIs to correct inconsistent or erroneous data items.

[0119] Again, suspected errors that cannot be corrected automatically are marked and submitted for manual review, and corrections are made based on the review results.

[0120] Finally, record each error correction operation, including error details, corrective measures, and operation time, to facilitate quality control and tracking.

[0121] In a specific implementation of the present invention, the specific operation steps of the video data processing module may be:

[0122] First, convert the video to a unified, easy-to-process format, such as MP4, to ensure compatibility and processing efficiency.

[0123] Second, key frames in the video are automatically detected and extracted for quick browsing or further analysis.

[0124] Again, use speech recognition technology to convert the voice content in the video into text to facilitate text analysis and search.

[0125] Next, perform OCR (Optical Character Recognition) on the static images in the video to extract the text information in the images.

[0126] Again, use computer vision technology to analyze video content, identify objects, faces, scenes, etc., and enhance the usability of video data.

[0127] Again, while ensuring sufficient clarity, the video should be appropriately compressed to reduce storage space.

[0128] Finally, the metadata of the video (such as shooting time, location, author, etc.) is extracted and labeled according to the content to facilitate retrieval and classification.

[0129] It should be noted that the establishment of a data preprocessing unit ensures the quality and consistency of the data, providing high-quality raw materials for subsequent data analysis and application. Through a series of operations such as data cleaning, deduplication, error correction and video data processing, the data preprocessing unit can eliminate noise, redundancy and errors in the data, thereby improving the accuracy and efficiency of data analysis.

[0130] In the embodiment of the present invention, the data analysis unit 300 includes a keyword analysis module 301, a similarity analysis module 302, a word relationship building module 303, and a tag system building module 304;

[0131] It should be noted that the keyword analysis module 301 is used to extract keywords from the historical data transmitted by the data preprocessing unit 200 according to the preset keyword extraction logic, and transmit the extracted keywords to the similarity analysis module 302, the word relationship building module 303 and the label system establishment module 304;

[0132] It should be noted that the similarity analysis module 302 is used to perform similarity calculation on the keywords transmitted by the keyword analysis module 301 according to a preset similarity analysis algorithm, and transmit the similarity calculation results in a structured form to the word relationship building module 303;

[0133] It should be noted that the word relationship building module 303 is used to build word relationships according to the keywords transmitted by the keyword analysis module 301 and the similarity calculation results transmitted by the similarity analysis module 302, and transmit the word relationship building results to the label system establishment module 304;

[0134] It should be noted that the label system establishment module 304 is used to establish a label system according to the keywords transmitted by the keyword analysis module 301 and the word relationship building results transmitted by the word relationship building module 303, and to verify the established label system according to the real-time data transmitted by the data preprocessing unit 200, and to transmit the verified label system to the server unit 600;

[0135] It should be noted that the keyword analysis module 301 is respectively connected to the similarity analysis module 302, the word relationship building module 303 and the label system building module 304, and the similarity analysis module 302, the word relationship building module 303 and the label system building module 304 are connected to each other in sequence;

[0136] It should be noted that the label system establishment module 304 is in communication connection with the data pre-processing unit 200;

[0137] It should be noted that the label system establishing module 304 is communicatively connected to the server unit 600 .

[0138] In a specific embodiment of the present invention, the preset keyword extraction logic can be performed through the following steps:

[0139] Step 1: Data preprocessing:

[0140] Text acquisition: Collecting raw text data from data sources such as news articles, social media posts, web content, etc. Removing irrelevant characters: including punctuation, numbers, special characters, etc.

[0141] Convert to lowercase: Unify the text case and reduce the complexity of case sensitivity.

[0142] Remove stop words: Remove common words such as "and", "or", and "but" that do not carry key information.

[0143] Stemming or lemmatization: restoring words to their base form, such as “running” to “run”.

[0144] Step two word segmentation:

[0145] Split continuous text into words or phrases. This process may require the use of specific word segmentation tools or algorithms depending on the language.

[0146] Step 3: Word frequency statistics:

[0147] Calculate the frequency of each word in the document or corpus to provide basic data for subsequent steps.

[0148] Step 4: Calculate keyword weight:

[0149] Calculate the term frequency (TF) of each word, which is the number of times a word appears in the document divided by the total number of words in the document.

[0150] The inverse document frequency (IDF) is calculated, which is the logarithm of the number of documents divided by the number of documents containing the word, to measure the general importance of the word.

[0151] Combining TF and IDF, we get the TF-IDF (Term Frequency-Inverse Document Frequency) value of each word. Words with high TF-IDF values ​​are usually more discriminative and have more keyword characteristics.

[0152] Step 5: Keyword filtering and sorting:

[0153] Sort all words according to TF-IDF value or other keyword weight indicators.

[0154] Set a threshold, filter out words below the threshold, and keep only the top N words as candidate keywords.

[0155] It may also be further filtered based on business needs, such as excluding common industry terms, brand names and other non-critical information.

[0156] Step 6: Keyword optimization and manual review:

[0157] Manual review of the machine-extracted keyword list to ensure the accuracy and relevance of the keywords.

[0158] Adjust algorithm parameters or add custom rules based on feedback to optimize keyword extraction results.

[0159] Step 7 Result Application:

[0160] Apply the finalized keywords to indexing, tag generation, content summarization, search engine optimization and other fields.

[0161] In a specific embodiment of the present invention, the preset similarity analysis algorithm can be performed by the following steps:

[0162] Step 1: Text preprocessing

[0163] Split the text into words or phrases, denoted by T1 = [w 11 ,w 12 ,...,w 1n ] and T2 = [w 21 ,w 22 ,...,w 2m ], where n and m are the number of words in the two texts respectively.

[0164] Step 2: Construct word frequency vector

[0165] Count the frequency of each word in different texts to form a word frequency vector, and convert it into a numerical vector using the bag-of-words model or TF-IDF.

[0166] Assume that after processing, we get the word frequency vector V1 = [f 11 ,f 12 ,...,f 1k ] and V2=[f 21 ,f 22 ,...,f 2k ], where k is the size of the vocabulary, and f ij Represents the frequency of the jth word in the i-th text.

[0167] Step 3: Vector Normalization

[0168] Computes the magnitude (length) of two vectors.

[0169] Vector V i The length L(V i ) is given by:

[0170] Step 4: Calculate cosine similarity

[0171] Cosine similarity calculation: The cosine similarity cos(θ) of two vectors is calculated using the inner product and length of the vector. The formula is as follows:

[0172] Step 5: Determine the text similarity based on the cosine similarity value:

[0173] When cos(θ) = 1, it means that the two vectors are exactly the same, that is, the text is exactly the same;

[0174] When cos(θ) = 0, it means that the two vectors are orthogonal, that is, there is no shared vocabulary between the texts or the directions are completely different;

[0175] When cos(θ)<0, it means that the two vectors point in roughly opposite directions, and the text may contain opposing or very little overlapping vocabulary;

[0176] In the embodiment of the present invention, it is generally believed that cos(θ)>0.7 indicates high similarity, and the specific threshold can be adjusted according to the application scenario.

[0177] In a specific embodiment of the present invention, word relationship building can be performed through the following steps:

[0178] Step 1: Construct a co-occurrence matrix

[0179] First, extract all unique words from the preprocessed text and construct a vocabulary V = [v1, v2, ..., v n ], where n is the size of the vocabulary. Traverse the text and record the co-occurrence of each pair of words. Construct an n×n co-occurrence matrix C, where C ij Represents vocabulary v i and v j The number of times they co-occur in the text.

[0180] Step 2: Matrix smoothing and normalization

[0181] In order to avoid the zero-frequency problem, the co-occurrence matrix is ​​smoothed. A common approach is to add a small constant α to each element (such as C ij =C ij +α), in order to eliminate the word frequency effect, the co-occurrence matrix is ​​row normalized or row + column normalized. The elements of the row-normalized matrix C′ Represents vocabulary v i When it appears, with v j The probability of co-occurrence.

[0182] Step 3: Build a word relationship graph

[0183] Based on the normalized co-occurrence matrix C′, an undirected weighted graph G = (V, E) is constructed, where the node V is the word in the vocabulary, the edge E represents the co-occurrence relationship between words, and the weight w of the edge ij =C′ ij Represents vocabulary v i and v j The strength of the association between them.

[0184] Step 4: Graph Dimensionality Reduction and Pruning

[0185] To simplify the graph structure and improve the interpretability of the relationship, graph embedding methods (such as TF-IDF, PCA, t-SNE, Word2Vec, etc.) can be used to map high-dimensional co-occurrence relationships into low-dimensional space. Set a threshold θ and remove edges with weights less than θ to reduce noise and retain only strongly associated word pairs.

[0186] In a specific embodiment of the present invention, the labeling system can be established by the following steps:

[0187] Step 1: Data preparation and preprocessing

[0188] Collect text data from the keywords transmitted by the keyword analysis module and the word relationship building results transmitted by the word relationship building module;

[0189] Step 2: Feature extraction

[0190] Convert text into vector form. Common methods include bag-of-words model and TF-IDF to quantify the importance of words in each document.

[0191] Step 3: Apply the topic model

[0192] Perform LDA (Latent Dirichlet Allocation) model training, initialize parameters, including the number of topics K, document-topic distribution θ, topic-word distribution β, and iteratively update document-topic distribution and topic-word distribution until convergence. Each topic can be regarded as a label, described by its related high-frequency words.

[0193] Exemplary document-topic distribution: θ d =(θ d1 ,θ d2 ,...,θ dK ), where θ dkRepresents the probability that document d belongs to topic k. Topic-word distribution: β k =(β k1 ,β k2 ,...,β kv ), where β kj Represents the probability of word j under topic k. The objective function of the LDA model: maximize the probability of document generation, which can be expressed as Where D is the total number of documents, W d is the number of words in document d, z d is the topic assignment for document d, P(w|z,β) is the probability of word w appearing given topic z, and P(z|θ) is the probability of document d choosing topic z.

[0194] Step 4: Tag generation and optimization

[0195] Based on the high-frequency words of each topic in the LDA model, labels are manually or automatically summarized and generated. The generated labels are adjusted, merged or removed according to business needs and the readability of the labels.

[0196] Step 5: Verification and Evaluation

[0197] Invite domain experts to review the generated labeling system, evaluate its accuracy and practicality, or collect user feedback on the use of labels in actual applications, such as click-through rate, search results, etc., for continuous optimization. If the labeling system is applied to document classification tasks, the classification accuracy can be calculated through methods such as cross-validation.

[0198] It should be noted that the establishment of a data analysis unit can provide comprehensive and in-depth text analysis capabilities, helping users to extract valuable information from massive amounts of text data, thereby supporting key business activities such as decision making and trend forecasting.

[0199] First, through the calculation of chord similarity, the data analysis unit can accurately measure the similarity between texts, which is crucial for tasks such as discovering duplicate content and identifying related documents. Especially in the fields of information retrieval and recommendation systems, chord similarity calculation makes it possible to efficiently and accurately screen and recommend content.

[0200] Secondly, by building word relationships, the data analysis unit can reveal the intrinsic connections between words and build a rich vocabulary network. This network not only helps to understand the deep meaning of the text content, but also provides strong support for advanced analysis tasks such as text clustering and topic extraction. In addition, the dimensionality reduction and pruning of the word relationship graph further simplifies the network structure and improves the analysis efficiency and accuracy.

[0201] Finally, by establishing a labeling system, the data analysis unit can convert complex text content into concise and clear labels, allowing users to browse, search and manage text data more conveniently. The accuracy and practicality of the labeling system are of great significance to improving user experience and enhancing system value. At the same time, through the verification and evaluation links, the labeling system can be continuously optimized to ensure that it is always highly consistent with user needs.

[0202] In summary, the data analysis unit has broad application prospects and important practical value in the field of text analysis. It can not only improve the efficiency and accuracy of text processing, but also provide users with richer and more in-depth analysis results to support them in making more informed decisions.

[0203] In the embodiment of the present invention, the data application unit 400 includes a topic tracking module 401, a hot spot monitoring module 402, a public opinion monitoring module 403 and a prediction control module 404;

[0204] It should be noted that the topic tracking module 401 is used to track the relevant data in the data acquisition unit 100 in real time according to the preset topic set, and to pre-track the predicted transmission path to which the prediction control module 404 is transmitted in advance;

[0205] It should be noted that the hotspot monitoring module 402 is used to determine whether there is a topic that has become a hotspot in the relevant data in the data acquisition unit 100, and transmit the determination result to the visualization display module 501 and the early warning notification module 503;

[0206] It should be noted that the public opinion monitoring module 403 is used to perform public opinion monitoring on the relevant data in the data acquisition unit 100 according to the preset public opinion monitoring mechanism, and transmit the detection results to the visualization display module 501 and the early warning notification module 503, and the public opinion monitoring mechanism is obtained according to the data in the data to be called module 602 in the server unit 600;

[0207] It should be noted that the prediction control module 404 is used to establish a prediction control model based on the label system and other data stored in the calling server unit 600, predict the data transmission path based on the prediction control model, and send the prediction result to the early warning notification module 503 to perform prediction control on public opinion;

[0208] It should be noted that the topic tracking module 401, the hot spot monitoring module 402, the public opinion monitoring module 403 are respectively connected in communication with the visualization display module 501 and the early warning notification module 503;

[0209] It should be noted that the prediction control module 404 is in communication connection with the warning notification module 503;

[0210] It should be noted that the data application unit 400 is communicatively connected to the server unit 600 .

[0211] In a specific embodiment of the present invention, the preset topic set may be {international news, domestic politics, economy and business, science and technology and innovation, society and people's livelihood, culture and art, sports events, health and life, education and academics, entertainment and celebrities, environment and sustainable development, etc.}.

[0212] In a specific embodiment of the present invention, the specific judgment logic for judging whether there is a topic that has become a hot topic in the relevant data in the data acquisition unit is as follows:

[0213] First, regularly obtain the latest hot topics and keywords from social media, news websites, trend analysis platforms, etc.

[0214] Secondly, establish a hot topic database to record the keywords, occurrence frequency, relevance score and heat decay model (such as time-based exponential decay) of each hot topic.

[0215] Thirdly, the text data in the data acquisition unit is preprocessed, including noise removal, word segmentation, stop word removal, stem extraction, etc., to facilitate subsequent analysis.

[0216] Again, to extract keywords from the preprocessed text, algorithms such as TF-IDF and TextRank can be used to identify important words in the text.

[0217] Thirdly, the extracted keywords are matched with the keywords in the hot topic library, and the similarity between the keywords is calculated, such as using cosine similarity or Jaccard similarity.

[0218] Again, a threshold is set. If the similarity exceeds the threshold, it is considered that the text contains hot keywords.

[0219] Again, based on the number and relevance of the matched hot keywords, combined with the publication date of the text and the hot spot decay model, a hot spot relevance score is calculated for each piece of data.

[0220] Again, if the score exceeds a preset threshold, it is determined that the data contains at least one current hot topic.

[0221] Finally, adjust the hotspot threshold and attenuation model according to actual conditions to improve the accuracy and timeliness of judgment.

[0222] For example, suppose that today's hot topic library contains the following keywords: "football match", "new energy vehicle subsidy policy", "Mars probe new discovery". The data sample is a text data summary from sports news: "Xiao Ming scored twice in last night's football qualifiers, helping the team beat the opponent."

[0223] The exemplary judgment process is as follows:

[0224] After text preprocessing, a list of keywords is obtained, such as "football qualifiers" and "defeat".

[0225] From the preprocessing results, "football qualifiers" is related to "football match" in the hot topic library.

[0226] Calculate the similarity between "football qualifiers" and "football match", assuming that the cosine similarity is used to get a high score (close to 1). Considering that the news release time is relatively new and the current popularity of "football match", the hot spot relevance score of the data calculated according to the decay model exceeds the preset threshold.

[0227] In a specific embodiment of the present invention, the preset public opinion monitoring mechanism can be performed by the following steps:

[0228] First, various data sources are collected from the data acquisition unit 100, including but not limited to social media (Weibo, WeChat, Douyin, etc.), news websites, forums, blogs, comment areas, etc.

[0229] Secondly, remove irrelevant information (such as advertisements and spam), perform text cleaning (remove non-Chinese characters, punctuation marks, numbers, etc.), and perform word segmentation processing to prepare for subsequent analysis.

[0230] Again, unify the text format, such as converting to lowercase, removing stop words, etc., to ensure consistency in subsequent processing.

[0231] Again, use TF-IDF, TextRank and other methods to extract keywords from the text and build a keyword library.

[0232] Again, by monitoring social media trends, news reporting frequency, etc., identify current hot topics and potential public opinion hotspots.

[0233] Again, sentiment analysis is performed on each piece of data to determine its positive, negative or neutral sentiment, which is done using a lexicon method, a machine learning model or a deep learning model.

[0234] Secondly, analyze the sentiment distribution of keywords and topics to identify the overall emotional trend of public opinion.

[0235] Again, use classification algorithms (such as SVM, naive Bayes, deep neural networks) to classify the text, such as positive reports, negative reports, neutral information, rumors, etc.

[0236] Again, based on the classification results and keywords, label each piece of data accordingly to facilitate subsequent management and analysis.

[0237] Again, through time series analysis, monitor the changes in the frequency of mentions of specific topics or keywords and identify the outbreak points of public opinion.

[0238] Again, set a threshold. When negative public opinion reaches a certain level or grows too fast, the early warning mechanism will be automatically triggered to notify relevant personnel.

[0239] Secondly, analyze the geographical sources of public opinion data and identify public opinion hotspots.

[0240] Again, through user behavior analysis, we can understand the main characteristics of the people participating in the discussion, such as age, gender, interests, etc.

[0241] Secondly, the results of the public opinion analysis are presented in the form of charts, dashboards, etc., including sentiment distribution maps, hot trend maps, regional distribution maps, etc.

[0242] Secondly, write a public opinion report based on the analysis results, summarizing the current status, trends, influencing factors and recommended countermeasures of public opinion.

[0243] Secondly, continue to monitor public opinion trends, collect feedback information, and evaluate the effectiveness of public opinion response measures.

[0244] Ultimately, based on actual results and feedback, we continuously optimize the keyword library, algorithm model, and warning threshold to improve detection accuracy and efficiency.

[0245] In a specific embodiment of the present invention, the prediction control model can be established by the following steps:

[0246] First, extract historical data from the server unit 600, including but not limited to data transmission time, path selection, data volume, network status indicators, etc. Apply data cleaning to process missing values ​​and abnormal values. Combined with the label system, classify and encode the data or construct features, such as constructing lagged variables based on time series characteristics.

[0247] Second, choose or build a model structure, such as using linear regression, time series models (ARIMA, state-space models), machine learning models (random forest, neural network), or deep learning models (RNN, LSTM).

[0248] Again, the preprocessed data set is divided into a training set and a validation set. The selected model is fitted using the training set data, the model parameters are optimized, and the optimal parameters are determined by minimizing the loss function (such as mean square error MSE).

[0249] Again, evaluate the model performance on the validation set and check the prediction error (such as MAE, RMSE). If necessary, adjust the model parameters or structure and perform cross-validation to avoid overfitting.

[0250] Finally, based on the prediction model, predict the future data transmission path. Design control strategies, such as adjusting the data transmission priority, path, or time window based on the prediction results. Implement online or offline model update mechanisms to adapt to environmental changes.

[0251] For example, suppose we try to predict the data transmission delay y as the data transmission amount x1, x2, ..., x at the past n time points n The function of , the model can be expressed as:

[0252] y=β0+β1x1+β2x2+...+β n x n +∈

[0253] Among them, β0,β1,...,β n is the model parameter, ∈ is the error term. During the training process, the parameter β is estimated by minimizing the loss function (such as mean square error) i :

[0254]

[0255] Here, m is the number of training samples, y i ,x i,1 ,...,x i,n are the true delay and eigenvalues ​​of the i-th sample respectively.

[0256] It should be noted that the data application unit 400 is capable of integrating information from multiple data sources and providing organizations with in-depth and accurate public opinion insights through advanced text processing and sentiment analysis technologies. These insights not only help organizations to quickly respond to and deal with negative public opinion, but also to seize opportunities in positive public opinion and further enhance their brand image and market position.

[0257] In the embodiment of the present invention, the data display unit 500 includes a visualization display module 501, a keyword search module 502, and an early warning notification module 503;

[0258] It should be noted that the visualization display module 501 is used to display the topic tracking results, hot spot monitoring results, public opinion monitoring results and prediction control results transmitted by the data application unit 400 in real time, and to display the corresponding keyword tag system in response to the user's keyword search operation;

[0259] It should be noted that the keyword search module 502 is used for users to search for target keywords, and the keyword search results are displayed through the visual display module 501;

[0260] It should be noted that the warning notification module 503 is used to issue different warning notifications for different states of the topic tracking module 401, the hot spot monitoring module 402, and the public opinion monitoring module 403, and simultaneously display the warning information through the visual display module 501;

[0261] It should be noted that the visual display module 501 is in communication connection with the keyword search module 502;

[0262] It should be noted that the visual display module 501 and the warning notification module 503 are in communication connection with the data application unit 400;

[0263] It should be noted that the data display unit 500 is communicatively connected to the server unit 600 .

[0264] It should be noted that the data display unit 500 provides an intuitive and interactive interface, allowing users to easily understand and explore large amounts of data. Through the visualization display module 501, users can observe the results of topic tracking, hot spot monitoring, public opinion monitoring, and predictive control in real time, so as to better understand the current data trends and dynamics.

[0265] The keyword search module 502 enables users to customize queries according to specific needs. Whether the user wants to learn more about a specific topic or wants to track the frequency of mentions of a keyword, it can be easily achieved through the keyword search module 502. This flexibility enables the data display unit 500 to meet the diverse needs of different users.

[0266] The early warning notification module 503 is an important function in the data display unit 500. It can automatically monitor and report any potential risks or changes based on preset thresholds. For example, when the discussion of a certain topic suddenly soars, or the mention frequency of a certain keyword increases abnormally, the early warning notification module 503 can immediately issue an alarm to ensure that the user can respond quickly. This timeliness is crucial for many application scenarios, such as crisis management, market monitoring, etc.

[0267] Through the collaborative work of the visualization module 501, the keyword search module 502, and the early warning notification module 503, the data display unit 500 not only improves the interactivity between the user and the data, but also greatly enhances the efficiency and accuracy of data analysis. This makes the data display unit 500 a powerful tool that can help users better understand and use data.

[0268] Finally, through the communication connection with the server unit 600, the data display unit 500 can obtain the latest data and analysis results in real time, ensuring that users can always access the latest and most accurate information. This real-time performance is crucial for many application scenarios that require rapid response, such as social media monitoring, financial market analysis, etc.

[0269] In the embodiment of the present invention, the server unit 600 includes a data storage module 601 and a data calling module 602;

[0270] It should be noted that the data storage module 601 is used to store the data acquired by the data acquisition unit 100 and the data generated by other modules;

[0271] It should be noted that the data to be called module 602 is used to convert the data in the data storage module 601 into callable data.

[0272] In a specific embodiment of the present invention, when the user enters the keyword "A" in the keyword search module 502 of the data display unit 500, the processing flow of the entire system may be as follows:

[0273] Keyword search trigger: The user inputs "A" in the keyword search module 502, and the system receives this query request.

[0274] Data display unit processing: The keyword search module 502 first responds to the user request, performs a matching search in the established tag system and historical data based on the keyword "A", and prepares to filter out data related to "A".

[0275] Server unit call: In order to obtain more comprehensive data, the keyword search module 502 sends a request to the data call module 602 of the server unit 600, requesting to call data related to the keyword "A". This includes but is not limited to historical data analysis results, topic tracking records, hot spot monitoring data, public opinion monitoring reports, etc.

[0276] The server unit responds: After receiving the request, the data calling module 602 retrieves all pre-processed data and analysis results related to "A" from the data storage module 601. These data may have been pre-processed such as data cleaning, deduplication, and error correction, and may have been labeled with labels related to "A".

[0277] Data is transmitted to the data display unit: The retrieved relevant data is transmitted back to the keyword retrieval module 502 through the data calling module 602, including but not limited to news articles, social media posts, video summaries, etc. related to "A".

[0278] Result display and analysis: The visualization display module 501 displays hot information, public opinion analysis, historical trends, etc. related to the keyword "A" in real time based on the returned data, which may be displayed in the form of charts, heat maps, time series analysis, etc. If the data related to "A" triggers the preset warning threshold, the warning notification module 503 will display the warning information through the visualization display module 501 or directly notify the user.

[0279] Fusion of real-time data and historical data: If the system is configured with a real-time data monitoring mechanism, the hotspot monitoring module 402 and the public opinion monitoring module 403 of the data application unit 400 may perform real-time hotspot tracking and public opinion analysis based on the keyword "A" queried by the user in combination with the real-time data stream to ensure that the displayed information contains both historical trends and reflects the latest developments.

[0280] User interaction and feedback: Users view the analysis results related to "A" through the visualization display module 501, and can further adjust the query conditions or take actions based on the displayed analysis results. The system continues to provide corresponding services based on new instructions or user behaviors, forming a dynamic interactive loop.

[0281] In a specific embodiment of the present invention, when a topic "B" in the topic set is detected as a public opinion or hot spot, the entire data asset management system will respond quickly and coordinate the linkage work of each module. The specific process can be as follows:

[0282] Public opinion or hot spot monitoring trigger: In the data application unit 400, the hot spot monitoring module 402 and the public opinion monitoring module 403 continuously monitor the data collected from the data acquisition unit 100, and identify abnormal changes in the topic "B" by analyzing indicators such as the frequency of mention, emotional tendency, and propagation speed of the data, and determine that it has become a hot spot or public opinion.

[0283] Warning and notification: The warning notification module 503 immediately identifies that "B" has become a hot topic or public opinion based on the preset threshold logic, and sends a warning signal to the pre-set recipients through email, text message, internal system notification, etc. At the same time, the emergency status of the topic is highlighted on the visual display module 501 to ensure that relevant personnel can quickly learn about it and intervene to deal with it.

[0284] Topic tracking and in-depth analysis: The topic tracking module 401 immediately starts or strengthens the tracking of topic "B" and closely monitors its latest developments in various data sources, including but not limited to social media, news reports, forum discussions, etc. The data analysis unit 300 performs in-depth analysis of topic "B" based on real-time data, including keyword analysis, similarity analysis, word relationship construction, etc., to understand the context of public opinion development and changes in public sentiment in more detail.

[0285] Prediction control and strategy adjustment: Based on historical data and current trends, the prediction control module 404 predicts the future development direction of topic "B", possible diffusion paths, and possible secondary public opinions or hot spots. Based on the prediction results, strategic recommendations are provided to relevant departments or teams to prevent or mitigate potential negative impacts.

[0286] Data display and decision support: The visualization display module 501 is dynamically updated to display the public opinion evolution map, sentiment analysis results, and communication paths of topic "B", providing decision makers with intuitive and visual decision support information. The keyword search module 502 may also automatically adjust the recommended keywords based on the current hot spots to guide users to pay attention to the latest information related to topic "B".

[0287] Data archiving and feedback loop: Data from the entire processing process, including warning information, analysis results, treatment measures, etc., are all saved to the data storage module 601 of the server unit 600 for subsequent audit, review or further analysis. The data call module 602 ensures that these data can be quickly called for continuous learning and optimization of the system's future response mechanism, forming a feedback loop of continuous improvement.

[0288] In a second embodiment of the present invention, the present invention provides a data asset management method, including:

[0289] Acquire different types of target data and pre-process the target data, where the target data at least includes structured data, unstructured data, video data, and specific data, where the specific data includes data acquired through different specific channels;

[0290] Perform correlation calculation on the target data after preprocessing, establish a data analysis system based on the calculation results, and judge the target real-time data in the target data after preprocessing in combination with the preset safety threshold logic;

[0291] Based on the keywords input by the user, combined with the data analysis system and judgment results, corresponding keyword analysis and early warning are carried out.

[0292] In summary, the present invention proposes a data asset management method and system, which obtains different types of target data and preprocesses the target data, wherein the target data at least includes structured data, unstructured data, video data and specific data, and the specific data includes data obtained through different specific channels; performs correlation calculation on the target data after preprocessing, establishes a data analysis system based on the calculation results, and judges the target real-time data in the target data after preprocessing in combination with the preset security threshold logic; performs corresponding keyword analysis and early warning based on the keywords input by the user end in combination with the data analysis system and the judgment results. It can not only efficiently integrate and analyze massive data, but also support data security management and early warning mechanisms, providing strong support for the business decision-making of newspapers.

[0293] The above-mentioned unit modules may be embedded in or independent of the processor in the computer device in the form of hardware, or may be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0294] In the third embodiment of the present invention, the present invention further provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 3 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data asset management method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0295] In a fourth embodiment of the present invention, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0296] Acquire different types of target data and pre-process the target data, where the target data at least includes structured data, unstructured data, video data, and specific data, where the specific data includes data acquired through different specific channels;

[0297] Perform correlation calculation on the target data after preprocessing, establish a data analysis system based on the calculation results, and judge the target real-time data in the target data after preprocessing in combination with the preset safety threshold logic;

[0298] Based on the keywords input by the user, combined with the data analysis system and judgment results, corresponding keyword analysis and early warning are carried out.

[0299] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0300] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0301] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0302] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0303] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0304] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0305] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data asset management system, characterized in that: include: A data acquisition unit (100), a data preprocessing unit (200), a data analysis unit (300), a data application unit (400), a data display unit (500) and a server unit (600); The data acquisition unit (100) is used to acquire different types of target data and transmit the target data to the data preprocessing unit (200), wherein the target data includes target historical data and target real-time data, and the different types include at least structured data, unstructured data and video data; The data preprocessing unit (200) is used to preprocess different types of target data transmitted by the data acquisition unit (100), and transmit the preprocessing results to the data analysis unit (300) and the data application unit (400), wherein the preprocessing at least includes data deduplication, data error correction and processing of video data; The data analysis unit (300) is used to calculate and establish a data analysis system based on the preprocessed target historical data transmitted by the data preprocessing unit (200), and to verify the data analysis system through the preprocessed target real-time data, and to save the verified data analysis system to the server unit (600); The data application unit (400) is used to store the preset security threshold logic and determine whether the target real-time data contains hot spots or public opinion data. If so, corresponding processing is performed through topic tracking and prediction control, and the determination result, hot topics and public opinion monitoring results are saved to the server unit (600). The hot spots data include data that meet the preset hot topics, and the public opinion data include data that meet the preset public opinion monitoring mechanism. The data display unit (500) is used to retrieve the data analysis system and judgment results, hot topics and public opinion monitoring results stored in the server unit (600), and perform data analysis and early warning according to the keywords input by the user; The server unit (600) is used to store data and provide callable data.

2. The data asset management system according to claim 1, characterized in that: The data acquisition unit (100) comprises a structured data acquisition module (101), an unstructured data acquisition module (102), a video data acquisition module (103) and a specific data acquisition module (104); The structured data acquisition module (101) is used to acquire structured data from the Internet or a fixed data resource platform, wherein the structured data at least includes fixed website data, Weibo data, WeChat data, and digital newspaper data. After the structured data is acquired, the structured data is transmitted to the data preprocessing unit (200) via a first preset communication interface. The unstructured data acquisition module (102) is used to acquire unstructured data from the Internet or a fixed data resource platform, wherein the unstructured data includes semi-structured data and unstructured data, and the unstructured data also includes at least forum data and blog data. After the unstructured data is acquired, the unstructured data is transmitted to the data preprocessing unit (200) via a second preset communication interface; The video data acquisition module (103) is used to acquire video data from the Internet or a fixed data resource platform, wherein the video data includes video data within a preset time period and data within a preset time range, and after the video data is acquired, the video data is transmitted to the data preprocessing unit (200) via a third preset communication interface; The specific data acquisition module (104) is used to acquire data from a media-specific offline data source, wherein the media-specific offline data source at least includes newspaper topic selection, reader reports, journalist surveys, and online questionnaires, and after acquiring the specific data, transmit the specific data to the data preprocessing unit (200) via a fourth preset communication interface; The fixed data resource platform at least includes an official statistical data platform, an academic research database, and a historical archive database; The structured data acquisition module (101), the unstructured data acquisition module (102), the video data acquisition module (103) and the specific data acquisition module (104) are respectively communicatively connected to the data preprocessing unit (200).

3. The data asset management system according to claim 1 or 2, characterized in that: The data pre-processing unit (200) comprises a data cleaning module (201), a data deduplication module (202), a data error correction module (203) and a video data processing module (204); The data cleaning module (201) is used to obtain data transmitted from the structured data acquisition module (101), the unstructured data acquisition module (102) and the specific data acquisition module (104), and to clean the transmitted data, and to transmit the cleaned data to the data deduplication module (202); The data deduplication module (202) is used to perform a deduplication operation on the data after the data cleaning operation of the data cleaning module (201), and transmit the data after the deduplication operation to the data error correction module (203); The data error correction module (203) is used to perform an error correction operation on the data after the deduplication operation of the data error correction module (203), and transmit the data after the error correction operation to the data analysis unit (300); The video data processing module (204) is used to obtain the video data transmitted by the video data acquisition module (103), and perform transcoding, key frame extraction, voice-to-text conversion and image-to-text conversion on the video data; The data cleaning module (201), the data deduplication module (202), and the data error correction module (203) are communicatively connected in sequence; the data cleaning module (201) is communicatively connected to the structured data acquisition module (101), the unstructured data acquisition module (102), and the specific data acquisition module (104); and the video data processing module (204) is communicatively connected to the video data acquisition module (103); The video data processing module (204) is in communication connection with the data cleaning module (201), and the communication connection interface is in a disconnected state; when the video data processing module (204) generates processed video data, the communication connection interface enters a closed connection state; The data cleaning module (201), the data deduplication module (202), the data error correction module (203) and the video data processing module (204) are respectively connected to the data analysis unit (300), the data application unit (400) and the server unit (600) for communication.

4. The data asset management system according to claim 1, characterized in that: The data analysis unit (300) includes a keyword analysis module (301), a similarity analysis module (302), a word relationship building module (303) and a tag system building module (304); The keyword analysis module (301) is used to extract keywords from the target historical data transmitted by the data preprocessing unit (200) according to a preset keyword extraction logic, and transmit the extracted keywords to the similarity analysis module (302), the word relationship building module (303) and the label system establishment module (304); The similarity analysis module (302) is used to perform similarity calculation on the keywords transmitted by the keyword analysis module (301) according to a preset similarity analysis algorithm, and transmit the similarity calculation results in a structured form to the word relationship building module (303); The word relationship building module (303) is used to build word relationships according to the keywords transmitted by the keyword analysis module (301) and the similarity calculation results transmitted by the similarity analysis module (302), and transmit the word relationship building results to the label system establishment module (304); The label system establishment module (304) is used to establish a label system according to the keywords transmitted by the keyword analysis module (301) and the word relationship building results transmitted by the word relationship building module (303), and to verify the established label system according to the target real-time data transmitted by the data preprocessing unit (200), and to transmit the verified label system to the server unit (600); The data analysis system includes keyword analysis, similarity analysis, word relationship building and label system establishment; The keyword analysis module (301) is respectively connected to the similarity analysis module (302), the word relationship building module (303) and the label system building module (304), and the similarity analysis module (302), the word relationship building module (303) and the label system building module (304) are connected to each other in sequence; The label system establishment module (304) is in communication connection with the data pre-processing unit (200); The label system establishment module (304) is communicatively connected to the server unit (600).

5. The data asset management system according to claim 1, characterized in that: The data application unit (400) includes a topic tracking module (401), a hot spot monitoring module (402), a public opinion monitoring module (403) and a prediction control module (404); The subject tracking module (401) is used to track the target real-time data in the data acquisition unit (100) in real time according to a preset subject set, and to pre-track the predicted transmission path transmitted by the prediction control module (404); The hotspot monitoring module (402) is used to determine whether there is a topic that has become a hotspot in the target real-time data in the data acquisition unit (100), and transmit the determination result to the visualization display module (501) and the early warning notification module (503); The public opinion monitoring module (403) is used to perform public opinion monitoring on the target real-time data in the data acquisition unit (100) according to a preset public opinion monitoring mechanism, and transmit the monitoring results to the visualization display module (501) and the early warning notification module (503); The prediction control module (404) is used to establish a prediction control model based on a machine learning algorithm according to the historical data stored in the server unit (600), input the target real-time data into the prediction control model to predict the data transmission path of the target real-time data, and send the prediction result to the early warning notification module (503) to perform prediction control on public opinion; The topic tracking module (401), the hot spot monitoring module (402), the public opinion monitoring module (403) are respectively connected to the visualization display module (501) and the early warning notification module (503); The prediction control module (404) is in communication connection with the early warning notification module (503); The data application unit (400) is communicatively connected to the server unit (600).

6. The data asset management system according to claim 1, characterized in that: The data display unit (500) comprises a visual display module (501), a keyword search module (502) and an early warning notification module (503); The visualization display module (501) is used to display the topic tracking results, hot spot monitoring results, public opinion monitoring results and prediction control results transmitted by the data application unit (400) in real time, and to display the corresponding keyword tag system in response to the user's keyword search operation; The keyword search module (502) is used for users to search for target keywords, and the keyword search results are displayed through the visual display module (501); The early warning notification module (503) is used to issue different early warning notifications for different states of the topic tracking module (401), the hot spot monitoring module (402) and the public opinion monitoring module (403), and simultaneously display the early warning information through the visual display module (501); The visual display module (501) is in communication connection with the keyword search module (502); The visual display module (501) and the early warning notification module (503) are communicatively connected with the data application unit (400); The data display unit (500) is communicatively connected to the server unit (600).

7. The data asset management system according to claim 1, characterized in that: The server unit (600) comprises a data storage module (601) and a data calling module (602); The data storage module (601) is used to store the data acquired by the data acquisition unit (100) and the data generated by other modules; The data to be called module (602) is used to convert the data in the data storage module (601) into callable data.

8. A data asset management method, characterized in that: include: Acquire different types of target data and preprocess the target data, wherein the target data includes target historical data and target real-time data, and the different types include at least structured data, unstructured data, video data and specific data, and the specific data includes data acquired through different specific channels; Calculate the similarity of the target data after preprocessing, establish a data analysis system based on the calculation results, and judge the target real-time data in the target data after preprocessing in combination with the preset safety threshold logic; Based on the keywords input by the user, combined with the data analysis system and judgment results, corresponding keyword analysis and early warning are carried out.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

Citation Information

Patent Citations

  • Regional public opinion monitoring and decision-making auxiliary system and method based on big data

    CN104933093A

  • Short video data processing system and processing method based on image processing technology monitoring

    CN117235343A

  • Network public opinion monitoring method and system and storage medium

    CN117609481A

  • Project similarity judgment method

    CN117708279A