Data classification processing method and device, electronic device and computer readable medium

Abstract

The present application relates to the field of computer information processing, and more specifically, to a data hierarchical processing method and device, electronic device, and computer-readable medium. The method includes: obtaining data to be processed, grading the data to be processed to obtain graded data, hierarchical registration and storage based on the digital object system, and obtaining specific data of the digital object according to the search keyword, so as to achieve the technical effect that the document entity corresponding to the identifier can be parsed in any application system supporting DOI, and the present application can find the full text content through the content of any level of data by grading the data and storing the graded data in a hierarchical manner, thus realizing a flexible search method.

Description

Technical Field

[0001] The present application relates to the field of computer information processing, and in particular, to a data classification processing method and device, an electronic device, and a computer-readable medium. Background Art

[0002] The intersection and integration of the Internet and other networks (including telecommunications networks, mobile networks, and the Internet of Things) has further promoted the integration of human society, information space, and the physical world, forming a new computing environment that integrates man, machine, and object, and has given rise to man, machine, and object integrated applications such as smart homes, smart cities, and intelligent manufacturing. The integration of man, machine, and object indicates that we are moving from terminal interconnection, user interconnection, and application interconnection to the interconnection of all things. Information technology and its applications are becoming more ubiquitous, and "big data" has emerged. The third wave of informatization (informatization 3.0), that is, the intelligent stage characterized by deep mining and integrated application of data, is opening up. Data has become a basic strategic resource and a key production factor in the new world of "digitalization of all things."

[0003] A common requirement for the integration of man, machine and things is data interconnection and intelligent application. For example, smart cities need to gather government data, social data, and even personal data to understand and predict the operation status of the city, achieve intelligent decision-making and efficient governance; smart manufacturing needs to connect the data between upstream and downstream of the industrial chain, various departments of the enterprise, and various production equipment in the factory to achieve accurate monitoring and control of the entire production process.

[0004] The Internet solves the problem of data transmission between machines and provides a basic technical means for data interconnection and interoperability. However, the Internet cannot avoid and solve many problems brought about by the continuous development of information technology and its applications, such as information islands, data loss of control, and data rights confirmation. The fundamental reason is that the core of the Internet, the TCP / IP protocol, only focuses on the encoding and decoding of data at the "binary bit" level and transmission control, ensuring that data in "packets" are efficiently exchanged between different computing devices, and consistent binary packages are obtained on any device on the network, while the identification, encoding, and parsing of data all rely on upper-level application processing. Therefore, in order to achieve data interconnection and interoperability based on Internet data transmission, application systems need to coordinate themselves and reach consensus on data syntax, semantics, pragmatics, etc., and face challenges such as high coordination costs, difficulty in ensuring responsibility, power, and effectiveness, as well as inefficiency, error-proneness, and difficulty in review.

[0005] In order to achieve trusted, manageable and controllable data interconnection and intelligent applications on the untrusted and uncontrollable Internet, academia and industry have proposed a variety of solutions. For example, the data middle platform aggregates data and provides unified data access services for upper-level applications with standard APIs, reducing the complexity of data application development, thereby solving the problem of information islands in data interconnection; the federated learning platform trains models at the source of data generation, uses encryption mechanisms to exchange and aggregate intermediate calculation results, and can achieve joint training models without leaving the local area, thereby solving the problem of data loss of control in data interconnection.

[0006] Looking at the existing solutions, it is not difficult to find that they all achieve trustworthy, manageable and controllable data interconnection and intercommunication by building software platforms on the Internet. Although the data of all parties within the platform can be reliably circulated, the data flow between platforms is still difficult to understand, difficult to access, and difficult to control. This situation is very similar to the early development of the Internet. Various organizations use private protocols to build their internal networks. Computers within the network can achieve reliable connectivity, but it is difficult for networks to communicate with each other. Turing Award winners Vinton Cerf and Robert Kahn jointly developed the TCP / IP protocol, the core technology of the Internet, in 1973. The core idea is to solve the above-mentioned problem of heterogeneous network interconnection through open architecture and standardized protocols.

[0007] Therefore, in response to the problem of existing platform data interconnection and interoperability, a feasible idea is to draw on the design concept of the Internet, adopt a software-defined approach, and connect various heterogeneous data platforms and systems through a data-centric open software architecture and standardized interoperability protocols, thereby forming a "virtual / data" network on top of the "physical / machine" Internet, which can be called the "Internet of Data" (Internet of Data), or "Digital Network" for short, to achieve integrated data interconnection and intelligent applications across the entire network.

[0008] In fact, both at home and abroad, early explorations have been carried out on the concept of digital networking. Among them, the most representative is the Digital Object Architecture (DOA) proposed by Robert Kahn. After inventing the TCP / IP protocol for machine interconnection in the 1970s, Kahn formally proposed the Digital Object Interface Protocol (DOIP) and Digital Object Identifier Resolution Protocol (DO-IRP) for data interconnection in the early 21st century in order to establish a set of digital object architectures for co-construction, sharing and governance by countries around the world on the Internet, and established the Open Foundation for Technical Standards DONA in 2014 to promote DOA. At present, DOA has published two standard proposals in the International Telecommunication Union Telecommunication Standardization Bureau (ITU-T), established a global digital object identification resolution system and multiple application systems, making DOA the most influential technology and standard system in the emerging field of digital networking.

[0009] DOA has achieved global large-scale application in the field of digital libraries, namely the DOI system. By constructing digital resources such as books, papers, reports, videos, etc. as digital objects and assigning unique and persistent DOI identifiers, the document entity corresponding to the identifier can be resolved in any application system that supports DOI, avoiding the common problem of resource inaccessibility caused by changes in resource location in the Uniform Resource Locator (URL). As of May 2021, the DOI system has registered approximately 257 million digital objects worldwide, covering many domestic and foreign academic databases such as IEEE, ACM, Springer, Wanfang, and CNKI.

[0010] By searching in the DOI system, the technical effect of retrieving specific data based on metadata can be achieved. However, the current search method is relatively simple and cannot achieve flexible search. Summary of the invention

[0011] In view of this, the present application provides a data hierarchical processing method and device, an electronic device and a computer-readable medium, which avoid the common problem of resource inaccessibility caused by changes in resource location in a uniform resource locator (URL), and achieve the technical effect of being able to parse the document entity corresponding to the identifier in any application system that supports DOI. In addition, the present application, by grading the data and storing the graded data in a hierarchical manner, can search for the full text content through the content of any level of data, thereby realizing a flexible search method.

[0012] According to one aspect of the present application, a data hierarchical processing method is proposed, the method comprising: obtaining data to be processed; grading the data to be processed to obtain graded data; determining digital objects of each level according to the graded data, the digital objects including metadata and specific data; for digital objects of each level, performing the following steps: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in a registration system and allocating a data warehouse; storing the specific data of the digital object in the data warehouse, the specific data being information reflecting the content of the data to be processed itself, the metadata being used to describe the digital object, the metadata including the unique identifier of the digital object and a tag of the digital object, the tag of the digital object being used to match with a search keyword when a user searches for the digital object; determining a search keyword; matching the search keyword with the tag in the metadata, screening out the digital objects corresponding to the successfully matched tags and obtaining their unique identifiers; querying the data warehouse storing the specific data of the digital object from the registration system according to the unique identifier; obtaining the specific data of the digital object from the queried data warehouse.

[0013] Furthermore, the metadata also includes hierarchical data, and the hierarchical data includes unique identifiers of digital objects of the previous level and / or the next level, and the unique identifiers of data objects of different levels are distinguished using a preset method.

[0014] Furthermore, the method also includes: determining hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determining the digital object of the previous level step by step according to the hierarchical data until the digital object of the highest level is determined; determining all digital objects of the next level step by step according to the digital object of the highest level until all digital objects of the lowest level are determined; querying all data warehouses storing digital objects of the lowest level from the registration system; and querying the specific data of all digital objects of the lowest level from the data warehouse.

[0015] Furthermore, the data search step also includes: determining the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determining the highest-level digital object based on the hierarchical data; querying the data warehouse storing the highest-level digital object from the registration system; and querying the specific data of the highest-level digital object from the data warehouse.

[0016] Furthermore, the method also includes: determining the digital object of the previous level step by step according to the level data in the metadata until the digital object of the highest level is determined; and adding the newly added level data to the metadata of the digital object of the highest level.

[0017] According to another aspect of the present application, a data hierarchical processing device is proposed, the device comprising: a data acquisition unit, a data grading unit, a hierarchical registration and storage unit based on a digital object system, and a search unit, the data acquisition unit is used to: acquire data to be processed; the data grading unit is used to: grade the data to be processed to obtain graded data; the hierarchical registration and storage unit based on a digital object system is used to: determine digital objects of each level according to the graded data, the digital objects including metadata and specific data; for the digital objects of each level, the following steps are performed: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in the registration system and assigning a A data warehouse; storing specific data of a digital object in the data warehouse, wherein the specific data is information reflecting the content of the data to be processed itself; the metadata is used to describe the digital object, and the metadata includes a unique identifier of the digital object and a tag of the digital object; the tag of the digital object is used to match a search keyword when a user searches for the digital object; the search unit is used to: determine a search keyword; match the search keyword with the tag in the metadata, filter out the digital object corresponding to the successfully matched tag and obtain its unique identifier; query the data warehouse storing the specific data of the digital object from the registration system according to the unique identifier; and obtain the specific data of the digital object from the queried data warehouse.

[0018] Furthermore, the metadata also includes hierarchical data, and the hierarchical data includes unique identifiers of digital objects of the previous level and / or the next level, and the unique identifiers of data objects of different levels are distinguished using a preset method.

[0019] Furthermore, the device also includes an adding unit, which is used to: determine the digital object of the previous level step by step according to the hierarchical data in the metadata until the digital object of the highest level is determined; and add the newly added hierarchical data to the metadata of the digital object of the highest level.

[0020] According to another aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to another aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0022] The data hierarchical processing method provided by the present application includes: obtaining data to be processed; grading the data to be processed to obtain graded data; determining digital objects of each level according to the graded data, wherein the digital objects include metadata and specific data; for digital objects of each level, performing the following steps: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in a registration system and allocating a data warehouse; storing the specific data of the digital object in the data warehouse, wherein the specific data is information reflecting the content of the data to be processed itself, wherein the metadata is used to describe the digital object, wherein the metadata includes the unique identifier of the digital object and the label of the digital object, wherein the label of the digital object is used to match the search keyword when the user searches for the digital object; determining the search keyword; matching the search keyword with the label in the metadata, screening out the digital object corresponding to the label that successfully matches and obtaining its unique identifier; querying the data warehouse storing the specific data of the digital object from the registration system according to the unique identifier; obtaining the specific data of the digital object from the queried data warehouse. By adopting the DOA system / architecture, each digital object is assigned a globally unique identifier. As the core attribute of the digital object, the identifier does not change with the change of the owner, storage location, or access method of the digital object. The document entity corresponding to the identifier can be resolved in any application system that supports DOI, avoiding the problem of resource inaccessibility caused by changes in resource location that is common in uniform resource locators (URLs). In addition, this application can search for full-text content through the content of any level of data by classifying the data and storing the classified data in a hierarchical manner, thus achieving a flexible search method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0024] Figure 1 The figure is a flow chart of a data classification processing method according to an exemplary embodiment.

[0025] Figure 2 It is a schematic diagram showing a correlation analysis result according to an exemplary embodiment.

[0026] Figure 3 is a block diagram of a data hierarchical processing device according to another exemplary embodiment.

[0027] Figure 4 It is a block diagram of an electronic device according to an exemplary embodiment.

[0028] Figure 5 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0030] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0033] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0034] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0035] In this application, "metadata" and "meta information" have the same meaning; "data to be processed" and "information to be processed" have the same meaning; "hierarchical data" and "hierarchical information" have the same meaning. "Keyword" and "keyword" have the same meaning.

[0036] Figure 1 The following is a flow chart of a data classification processing method provided by the present application. The method is based on a digital object system.

[0037] The digital object architecture (DOA) includes: a registry, a handle system, and a repository.

[0038] The Registry is used to store the metadata of DO (Digital Object), the Handle System is used to register the DO and record in which Repository the DO is stored, and the Repository is used to store the DO.

[0039] like Figure 1 As shown, the method includes the following steps: a data acquisition step, a data classification step, a hierarchical registration and storage step based on a digital object system, and a search step.

[0040] Step S101: Obtain data to be processed.

[0041] The data to be processed can be data that has been collected and classified, or it can be collected. The collection methods include automatic collection and manual collection, such as distributed web crawlers or relevant experts who are very familiar with this field. When collecting, it is necessary to mark its affiliated institution or author, generation time and title, and if possible, the target location associated with the collected information.

[0042] As an optional implementation method, an automated crawler is used to first determine the information source of focus, such as the websites of major news media. After the information source is determined, the crawling scope needs to be determined, such as whether to track external links within the target website, how many layers to crawl, whether to crawl files or pictures, etc.

[0043] Step S102: classifying the data to be processed to obtain classified data.

[0044] The step of grading the data to be processed can also be called a structured parsing step. The data to be processed is graded according to its type, and after the grading is completed, the data to be processed is parsed into multiple minimum information units (also referred to as minimum units in this application).

[0045] Information (data to be processed) can be in the form of files. For example, a report contains multiple files. In this case, the report can be regarded as the parent information, and each file is the sub-information it contains. In addition to files, a web page may contain multiple links, so each link can also be regarded as a sub-information unit.

[0046] The text information of the website can be arranged in a tree structure from top to bottom (hierarchical). The top level is the website information, followed by the web page information, which contains different articles, and the sentences in the articles are the smallest units of digital objects.

[0047] As an optional implementation, structural analysis (grading) is performed by paragraph or sentence. Paragraph analysis is relatively simple, and only requires segmenting the article document according to the carriage return character. However, sentences cannot be segmented by periods like paragraphs, because some abbreviations such as USA also contain periods, so it is necessary to use an abbreviation corpus to assist in sentence segmentation.

[0048] The classification step includes not only segmenting the paragraphs and sentences of the article, but also extracting the attributes of the article, such as the author of the article, the organization of the article, etc. With the help of the knowledge base in the professional field, the entities in the sentence can also be annotated, such as the location and unit.

[0049] Step S103: hierarchical registration and storage based on the digital object system.

[0050] Step S103 specifically includes: determining digital objects at each level based on hierarchical data, where the digital objects include metadata and specific data, where the metadata includes hierarchical information of the digital objects, to facilitate future retrieval; for digital objects at each level, executing the following steps: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in a registration system and assigning a data warehouse; storing the specific data of the digital object in the data warehouse, where the specific data is information reflecting the content of the data to be processed itself, and the metadata is used to describe the digital object, where the metadata includes the unique identifier of the digital object and the tag of the digital object, and the tag of the digital object is used to match the search keyword when the user searches for the digital object.

[0051] Step S103 is a process of reorganizing and storing the hierarchical data obtained in step S102 according to the digital object system.

[0052] The specific data refers to the information reflecting the content of the data to be processed.

[0053] The hierarchical information of data is mainly reflected through the meta-information in digital objects.

[0054] Meta-information / metadata is mainly used to describe digital objects to facilitate user searches. The composition of metadata is relatively flexible, but generally speaking, it will include the following information: the ID number of the digital object (also referred to as DOID or the unique identifier of the digital object in this application), the type of the digital object, the length of the digital object, the format of the digital object, and several keywords (also referred to as tags in this application) of the digital object.

[0055] In a broad sense, any auxiliary information of the content of a digital object can be called metadata, such as the time the article was recorded, the person who recorded it, the recorder's confirmation of the credibility of the article, the validity period of the information, etc.

[0056] Step S104: Acquire specific data of the digital object according to the search keyword.

[0057] Step S104 specifically includes: determining the search keyword; matching the search keyword with the tag in the metadata, screening out the digital objects corresponding to the successfully matched tags and obtaining their unique identifiers; querying the data warehouse storing the specific data of the digital objects from the registration system according to the unique identifier; and obtaining the specific data of the digital objects from the queried data warehouse.

[0058] The data hierarchical processing method provided by the present application includes: obtaining data to be processed; grading the data to be processed to obtain graded data; determining digital objects at each level according to the graded data, wherein the digital objects include metadata and specific data; for digital objects at each level, performing the following steps: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in a registration system and allocating a data warehouse; storing the specific data of the digital object in a data warehouse, wherein the specific data is information reflecting the content of the data to be processed itself, and the metadata is used to describe the digital object, and the metadata includes the unique identifier of the digital object and the label of the digital object, and the label of the digital object is used to match the search keyword when the user searches for the digital object, and determine the search keyword; matching the search keyword with the label in the metadata, screening out the digital object corresponding to the successfully matched label and obtaining its unique identifier; querying the data warehouse storing the specific data of the digital object from the registration system according to the unique identifier; and obtaining the specific data of the digital object from the queried data warehouse. By adopting the DOA system / architecture, each digital object is assigned a globally unique identifier. As the core attribute of the digital object, the identifier does not change with the change of the owner, storage location, or access method of the digital object. The document entity corresponding to the identifier can be resolved in any application system that supports DOI, avoiding the problem of resource inaccessibility caused by changes in resource location that is common in uniform resource locators (URLs). In addition, this application can search for full-text content through the content of any level of data by classifying the data and storing the classified data in a hierarchical manner, thus achieving a flexible search method.

[0059] Next, the method of the present application will be described in detail.

[0060] For example, the body of an article includes 6 paragraphs in total, and the number of sentences included in these 6 paragraphs is 1, 2, 3, 4, 2, and 3. This article is the data to be processed (also called information to be processed). As an optional implementation, a sentence is used as the smallest information unit.

[0061] A sentence is an information unit (a sentence is the smallest information unit), a paragraph is also an information unit, and an article is also an information unit. There is a hierarchical relationship between these information units. Sentences are at the lowest level, paragraphs are at the middle level, and articles are at the highest level. Sentences are sub-information units of their paragraphs, and paragraphs are sub-information units of their articles.

[0062] This application implements nested storage of information by nesting IDs of information units at different levels.

[0063]

Paragraph 1

[0064]

Paragraph 2

[0065] The first sentence of

Paragraph 2

[0066] The second sentence of [Paragraph 2] is the third sentence of the entire text, and will be referred to as Sentence 3 below.

[0067]

Paragraph 3

[0068] The first sentence of

Paragraph 3

[0069] The second sentence of [Paragraph 3] is the fifth sentence of the entire text, and will be referred to as Sentence 5 below.

[0070] The third sentence of [Paragraph 3] is the sixth sentence of the entire text, and will be referred to as Sentence 6 below.

[0071]

Paragraph 4

[0072] The first sentence of

Paragraph 4

[0073] The second sentence of [Paragraph 4] is the 8th sentence of the entire text, and will be referred to as Sentence 8 below.

[0074] The third sentence of [Paragraph 4] is the 9th sentence of the entire text, and will be referred to as Sentence 9 below.

[0075] The fourth sentence of [Paragraph 4] is the 10th sentence of the entire text, and will be referred to as Sentence 10 below.

[0076]

Paragraph 5

[0077] The first sentence of

Paragraph 5

[0078] The second sentence of [Paragraph 5] is the 12th sentence of the entire text, and will be referred to as Sentence 12 below.

[0079]

Paragraph 6

[0080] The first sentence of

Paragraph 6

[0081] The second sentence of [Paragraph 6] is the 14th sentence of the entire text, and will be referred to as Sentence 14 below.

[0082] The third sentence of [Paragraph 6] is the 15th sentence of the entire text, and will be referred to as Sentence 15 below.

[0083] The metadata of data at different levels include different aspects. For example, at the article level, the organization that publishes the article, the time of article publication, the media that publishes the article, the author of the article, the keywords of the article, the ID of the article, etc. can be used as metadata; at the paragraph level, the keywords of the paragraph, the ID of the paragraph, etc. can be used as metadata; at the sentence level, the keywords of the sentence, the ID of the sentence, etc. can be used as metadata.

[0084] The inventor of the present application has found that if a sentence is stored in a data warehouse as the smallest information unit, the information that can be retrieved is at the sentence level. For example, the above article includes 15 sentences in total, and the metadata of these 15 sentences are stored in the registration table (as shown in Table 1), the mapping relationship between the ID in the metadata and the data warehouse is stored in the registration system (as shown in Table 2), and the specific data of these 15 sentences is stored in the data warehouse (as shown in Table 3). Then, when searching based on the search keywords, these sentences can be retrieved as long as the appropriate search keywords are used.

[0085] Table 1

[0086]

[0087] Table 2

[0088]

[0089]

[0090] Table 3

[0091]

[0092] The search keyword is "Keyword 2", and the keyword "Keyword 2" is matched with the tags in Table 1. The successfully matched IDs are 100001 and 100015.

[0093] In Table 2, it can be found that the data warehouse number corresponding to ID100001 is C0000051; the data warehouse number corresponding to ID100015 is C0000065.

[0094] In Table 3, by querying the specific data stored in the data warehouse numbered C0000051, the content of sentence 1 is obtained. In Table 3, by querying the specific data stored in the data warehouse numbered C0000065, the content of sentence 15 is obtained.

[0095] In this way, two sentences in the article whose tags contain "Keyword 2" are searched out.

[0096] However, the inventors found that if they want to perform data analysis based on search results, it is not enough to just know two separate sentences. They also need to know the content of the article where the two sentences are located, as well as information such as the author, organization, source, and publication time of the article.

[0097] The inventors have come up with a method to improve this, that is, to store the author, organization, source, publication time and other information of the article where the sentence is located as the metadata of the sentence in the registry. In this way, after the sentence is retrieved according to the search keyword, the author, organization, source, publication time and other information of the article where the sentence is located can be known according to the content of the metadata. However, there are still two problems: First, the specific content of the article where the sentence is located cannot be known. Second, the meta information of the sentence is too complex, and there is too much redundant information, which occupies a large amount of storage space. For example, the meta information of the above 15 sentences all contains the author, organization, source, publication time and other information of the article. And this information is exactly the same, which will cause a large amount of redundant information to occupy storage resources. This is only the case of shorter articles. If an article is relatively long, for example, there are 100 sentences, then the meta information of each sentence contains the author, organization, source, publication time and other information of the article, which will cause the author, organization, source, publication time and other information of the article to be stored 100 times in the registry, causing redundancy.

[0098] The inventor then further thought that if the paragraph where the sentence is located is also given an ID, the paragraph ID is stored as the meta information of the sentence, and then the article where the paragraph is located is also given an ID, and the article ID is stored as the meta information of the paragraph. In this way, when a sentence is searched out by searching keywords, the ID of the paragraph where the sentence is located can be obtained based on the meta information of the sentence, so that the paragraph can be located by the ID of the paragraph. Then, the ID of the article where the paragraph is located is obtained based on the meta information of the paragraph, so that the article where the sentence is located can be located by the ID of the article. The metadata of the article contains information such as the author, organization, source, and publication time of the article. In this way, only the information such as the author, organization, source, and publication time of the article needs to be used as the meta information of the article, and the information such as the author, organization, source, and publication time of the article does not need to be used as the meta information of each sentence, which greatly reduces the amount of stored data.

[0099] The inventor further thought that if the meta information of an article contains the ID of each paragraph contained in the article, and the meta information of a paragraph contains the ID of each sentence contained in the paragraph, then the ID of each paragraph can be known from the meta information of the article, thereby locating each paragraph, and then the ID of the sentence contained in each paragraph can be known from the meta information of each paragraph, thereby locating each sentence contained in each paragraph. In this way, by retrieving any sentence in a certain article, the entire content of the article can be obtained.

[0100] That is, articles, paragraphs, and sentences are three different levels, and the metadata of each level is selected differently. The metadata at the sentence level includes the sentence ID and the ID of the paragraph to which the sentence belongs. The metadata at the paragraph level includes the paragraph ID, the ID of the article to which the paragraph belongs, and the IDs of all sentences contained in the paragraph.

[0101] The IDs of data at different levels can be distinguished by prefixes, suffixes, etc. For example, in Table 4, the first digit 1 is the sentence ID, the first digit 2 is the paragraph ID, and the first digit 3 is the article ID. For example, the first digit of the 15 IDs "100001", "100002", ..., "100015" is 1, which is the sentence ID. For example, the first digit of the 6 IDs "200001", "200002", ..., "200006" is 2, which is the paragraph ID. For example, the first digit of "300001" is 3, which is the article ID.

[0102] Table 4

[0103]

[0104]

[0105] The search keyword is "Keyword 2", and the keyword "Keyword 2" is matched with the tags in Table 4. The successfully matched IDs are 100001, 100015, 200001, 200006, and 300001.

[0106] As mentioned above, in Table 4, the first digit 1 is the sentence ID, the first digit 2 is the paragraph ID, and the first digit 3 is the article ID. Therefore, we can know that ID100001 and ID100015 are sentence IDs, ID200001 and ID200006 are paragraph IDs, and ID300001 is the article ID.

[0107] The entire content of the article can be obtained based on any of these five IDs.

[0108] The method to query the entire content of the article according to ID100001 is as follows:

[0109] In Table 4, it can be found that the ID of the previous level data of ID100001 is 200001. In Table 4, the ID of the previous level data of ID200001 is queried, and the ID found is 300001. The metadata of ID300001 does not contain the ID of the previous level data. Therefore, ID300001 is the highest level data. In Table 4, the ID of the next level data of ID300001 is queried, and the IDs found are 200001, 200002, 200003, 200004, 200005, and 200006. In Table 4, the ID of the next level data of ID200001 is queried, and the ID found is 100001. In Table 4, the ID of the next level data of ID200002 is queried, and the IDs found are 100002 and 100003. In Table 4, the ID of the next level data of ID200003 is queried, and the IDs found are 100004, 100005, and 100006. In Table 4, the ID of the next level data of ID200004 is queried, and the IDs found are 100007, 100008, 100009, and 100010. In Table 4, the ID of the next level data of ID200005 is queried, and the IDs found are 100011 and 100012. In Table 4, the ID of the next level data of ID200006 is queried, and the IDs found are 100013, 100014, and 100015. Since the metadata of ID100001, ID100002, ID100003, ID100004, ID100005, ID100006, ID100007, ID100008, ID100009, ID100010, ID100011, ID100012, ID100013, ID100014, and ID100015 do not contain the ID of the next level of data, it is shown that these data are the lowest level data. In this way, the IDs of all sentences in the article (the IDs of all data at the lowest level) are known. According to the ID of each of the 15 sentences, the corresponding data warehouse number is queried in Table 2, and then the specific data stored in the data warehouse is queried in Table 3 according to the data warehouse number. In this way, the article in which a sentence is located can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0110] The method of querying the entire content of an article based on ID100015 is similar to the method of querying the entire content of an article based on ID100001, and will not be repeated here.

[0111] The method to query the entire content of the article according to ID200001 is as follows:

[0112] In Table 4, the ID of the data of the previous level of ID200001 is queried, and the ID found is 300001. The metadata of ID300001 does not contain the ID of the data of the previous level. Therefore, ID300001 is the data of the highest level. In Table 4, the ID of the data of the next level of ID300001 is queried, and the IDs found are 200001, 200002, 200003, 200004, 200005, and 200006. In Table 4, the ID of the data of the next level of ID200001 is queried, and the ID found is 100001. In Table 4, the ID of the data of the next level of ID200002 is queried, and the IDs found are 100002 and 100003. In Table 4, the ID of the data of the next level of ID200003 is queried, and the IDs found are 100004, 100005, and 100006. In Table 4, the ID of the next level data of ID200004 is queried, and the IDs found are 100007, 100008, 100009, and 100010. In Table 4, the ID of the next level data of ID200005 is queried, and the IDs found are 100011 and 100012. In Table 4, the ID of the next level data of ID200006 is queried, and the IDs found are 100013, 100014, and 100015. Since the metadata of ID100001, ID100002, ID100003, ID100004, ID100005, ID100006, ID100007, ID100008, ID100009, ID100010, ID100011, ID100012, ID100013, ID100014, and ID100015 do not contain the ID of the next level of data, it is shown that these data are the lowest level data. In this way, the IDs of all sentences in the article (the IDs of all data at the lowest level) are known. According to the ID of each of the 15 sentences, the corresponding data warehouse number is queried in Table 2, and then the specific data stored in the data warehouse is queried in Table 3 according to the data warehouse number. In this way, the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0113] The method of querying the entire content of an article based on ID200006 is similar to the method of querying the entire content of an article based on ID200001, and will not be repeated here.

[0114] The method to query the entire content of the article based on ID300001 is as follows:

[0115] In Table 4, the ID of the previous level data of ID300001 is queried, and it is found that the metadata of ID300001 does not contain the ID of the previous level data. Therefore, ID300001 is the highest level data. In Table 4, the ID of the next level data of ID300001 is queried, and the IDs found are 200001, 200002, 200003, 200004, 200005, and 200006. In Table 4, the ID of the next level data of ID200001 is queried, and the ID found is 100001. In Table 4, the ID of the next level data of ID200002 is queried, and the IDs found are 100002 and 100003. In Table 4, the ID of the next level data of ID200003 is queried, and the IDs found are 100004, 100005, and 100006. In Table 4, the ID of the next level data of ID200004 is queried, and the IDs found are 100007, 100008, 100009, and 100010. In Table 4, the ID of the next level data of ID200005 is queried, and the IDs found are 100011 and 100012. In Table 4, the ID of the next level data of ID200006 is queried, and the IDs found are 100013, 100014, and 100015. Since the metadata of ID100001, ID100002, ID100003, ID100004, ID100005, ID100006, ID100007, ID100008, ID100009, ID100010, ID100011, ID100012, ID100013, ID100014, and ID100015 do not contain the ID of the next level of data, it is shown that these data are the lowest level data. In this way, the IDs of all sentences in the article (the IDs of all data at the lowest level) are known. According to the ID of each of the 15 sentences, the corresponding data warehouse number is queried in Table 2, and then the specific data stored in the data warehouse is queried in Table 3 according to the data warehouse number. In this way, the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0116] If the metadata contains the ID of the previous level data but not the ID of the next level data, it means that the digital object is the lowest level data and corresponds to the smallest information unit. If the metadata contains the ID of the next level data but not the ID of the previous level data, it means that the digital object is the highest level data. If the metadata contains both the ID of the next level data and the ID of the previous level data, it means that the digital object is the middle level data.

[0117] For any ID retrieved according to the search keyword, the metadata of the digital object corresponding to the ID is searched in the registry. There are three possible situations.

[0118] The first case: If the metadata of the ID contains the ID of the previous level data but not the ID of the next level data, it means that this digital object is the lowest level data. At this time, search for the ID of the previous level data level by level until the ID of the highest level data is found. Then, determine the next level data based on the ID of the next level data contained in the metadata of the highest level data, and search for the ID of the next level data level by level until all the lowest level data is found. Query the data warehouse number in the registration system based on the ID of the lowest level data, and query the specific data in the data warehouse based on the data warehouse number to restore all the content.

[0119] The second case: If the metadata of the ID contains the ID of the previous level data and the ID of the next level data, it means that this digital object is the data of the middle level. At this time, search for the ID of the previous level data level by level until the ID of the highest level data is found. Then, determine the next level data based on the ID of the next level data contained in the metadata of the highest level data, and search for the ID of the next level data level by level until all the lowest level data is found. Query the data warehouse number in the registration system based on the ID of the lowest level data, and query the specific data in the data warehouse based on the data warehouse number to restore all the content.

[0120] The third case: If the metadata of the ID contains the ID of the next level of data, but does not contain the ID of the previous level of data, it means that this digital object is the highest level of data. At this time, the next level of data is determined based on the ID of the next level of data contained in the metadata, and the ID of the next level of data is searched level by level until all the lowest level of data is found. The data warehouse number is queried in the registration system based on the ID of the lowest level of data, and the specific data is queried in the data warehouse based on the data warehouse number, and all the content can be restored.

[0121] In this way, the technical effect of querying the full text of an article based on any ID at any level is achieved.

[0122] It can be easily found that this method has quite a few advantages. First, information such as the author of the article only needs to be stored once as the meta information of the article, and does not need to be stored as the meta information of the paragraph or sentence, which greatly reduces the amount of data stored in the registry. Second, as long as any sentence in the article is searched according to the search keyword, all the information of the article can be obtained, including the article body and the author of the article. Third, what is stored in the data warehouse is the smallest information unit (sentence), and the full text of the paragraph or article is not stored. The full text of the paragraph or article is obtained by combining the sentences. That is, the sentence where the search keyword is located can be located, and the paragraph and article where the search keyword is located can also be located. There is no need to store the sentences, paragraphs, and articles in the data warehouse in duplicate at the same time, thereby minimizing the amount of specific data that needs to be stored in the data warehouse.

[0123] The inventors found that, as an optional implementation, the metadata may also include the ID of the upper level data but not the ID of the lower level data, and the specific data of the highest level DO is stored in the data warehouse. For example, as shown in Table 5, the metadata of the sentence includes the ID of the paragraph. The metadata of the paragraph includes the ID of the article, but does not include the ID of the sentence. The metadata of the article does not include the ID of the paragraph. The specific data of the DO with ID 300001 is stored in the data warehouse.

[0124] Table 5

[0125]

[0126]

[0127] The search keyword is "keyword 2", and the keyword "keyword 2" is matched with the tags in Table 5. The successfully matched IDs are 100001, 100015, 200001, 200006, and 300001.

[0128] As mentioned above, in Table 5, the first digit 1 is the sentence ID, the first digit 2 is the paragraph ID, and the first digit 3 is the article ID. Therefore, we can know that ID100001 and ID100015 are sentence IDs, ID200001 and ID200006 are paragraph IDs, and ID300001 is the article ID.

[0129] The entire content of the article can be obtained based on any of these five IDs.

[0130] The method to query the entire content of the article according to ID100001 is as follows:

[0131] In Table 5, it can be found that the ID of the previous level data of ID100001 is 200001. In Table 5, the ID of the previous level data of ID200001 is 300001. The metadata of ID300001 does not contain the ID of the previous level data. Therefore, ID300001 is the highest level data. Query the specific data of ID300001 in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0132] The method of querying the entire content of an article based on ID100015 is similar to the method of querying the entire content of an article based on ID100001, and will not be repeated here.

[0133] The method to query the entire content of the article according to ID200001 is as follows:

[0134] In Table 5, the ID of the previous level data of ID200001 is queried, and the ID found is 300001. The metadata of ID300001 does not contain the ID of the previous level data. Therefore, ID300001 is the highest level data. Query the specific data of ID300001 in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0135] The method of querying the entire content of an article based on ID200006 is similar to the method of querying the entire content of an article based on ID200001, and will not be repeated here.

[0136] The method to query the entire content of the article based on ID300001 is as follows:

[0137] In Table 5, we query the ID of the previous level data of ID300001 and find that the metadata of ID300001 does not contain the ID of the previous level data. Therefore, ID300001 is the highest level data. We query the specific data of ID300001 in the data warehouse. In this way, we can restore the content of the article. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0138] The metadata contains the ID of the previous level data. In this way, after locating any DO by searching keywords, the ID of the previous level DO contained in the metadata of the DO is queried step by step until the ID of the highest level DO is queried. Then, the data warehouse corresponding to the ID of the highest level DO is queried in the Handle System, and then the specific data stored in the data warehouse is queried to obtain the specific data of the highest level DO.

[0139] The inventors found that metadata can also include the ID of target level data. For example, as shown in Table 6, the target level is the article level. The specific data of the DO of the target level is stored in the data warehouse (the data warehouse stores the specific data of the DO with ID 300001). In this way, the DO of the target level can be located by any ID, and then the specific data of the DO of the target level can be retrieved from the data warehouse.

[0140] Table 6

[0141]

[0142]

[0143] The search keyword is "keyword 2", and the keyword "keyword 2" is matched with the tags in Table 6. The successfully matched IDs are 100001, 100015, 200001, 200006, and 300001.

[0144] As mentioned above, in Table 6, the first digit 1 is the sentence ID, the first digit 2 is the paragraph ID, and the first digit 3 is the article ID. Therefore, we can know that ID100001 and ID100015 are sentence IDs, ID200001 and ID200006 are paragraph IDs, and ID300001 is the article ID.

[0145] The entire content of the article can be obtained based on any of these five IDs.

[0146] The method to query the entire content of the article according to ID100001 is as follows:

[0147] In Table 6, it can be found that the target level data contained in the metadata of ID100001 has an ID of 300001. The specific data of ID300001 is queried in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0148] The method to query the entire content of the article according to ID100015 is as follows:

[0149] In Table 6, it can be found that the target level data contained in the metadata of ID100015 has an ID of 300001. The specific data of ID300001 is queried in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0150] The method to query the entire content of the article according to ID200001 is as follows:

[0151] In Table 6, it can be found that the target level data contained in the metadata of ID200001 has an ID of 300001. The specific data of ID300001 is queried in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0152] The method to query the entire content of the article according to ID200006 is as follows:

[0153] In Table 6, it can be found that the target level data contained in the metadata of ID200006 has an ID of 300001. The specific data of ID300001 is queried in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0154] The method to query the entire content of the article based on ID300001 is as follows:

[0155] In Table 6, it can be found that the target level data contained in the metadata of ID300001 has an ID of 300001. The specific data of ID300001 is queried in the data warehouse. In this way, the content of the article can be restored. In addition, information such as the author of the article can also be obtained from the metadata of ID300001.

[0156] In this application, it is very easy to add hierarchical information. For example, as shown in Table 4, the highest level of data is the article level, and ID300001 corresponds to Article 1. There are other articles on the web page where the article is located, for example, there is also Article 2. Assume that Article 1 is crawled first, and it is graded and stored in a hierarchical manner according to the content of Article 1. Then Article 2 is crawled, and it is also graded and stored in a hierarchical manner according to the content of Article 2. Article 1 and Article 2 come from the same web page (for the convenience of narration, it is assumed to be Web Page 1). Now, to add the information of Web Page 1 to the information of Article 1, it is only necessary to add the ID of Web Page 1 to the ID column of the upper level data of ID300001, and then add the information of the web page. For example, the ID of Web Page 1 is 400001. Add "400001" to the blank column of the last row of Table 4, and then add a row of relevant information of ID400001 (assuming that Web Page 1 includes Article 1 and Article 2, and the ID of Article 2 is 300002). It can be seen that this application can flexibly deal with the level of adding information, the workload of modification is very small, and it is very convenient to modify. The fundamental reason is that the metadata of each level of data includes the data ID, the ID of the previous level of data, and / or the ID of the next level of data. Adding a level only affects the current highest level of data, but not the data of other levels. Therefore, the metadata of data of other levels does not need to be changed. The information of Article 1 in the registration system and data warehouse will not be affected by adding web page information.

[0157] In the traditional Internet, information is organized in the form of data and searched by addresses (such as URLs, etc.). Different from the traditional Internet, the information in the DO system is stored in the form of DO. Each DO contains metadata and specific data. The corresponding DO can be found by content rather than address. In addition, from the above content, it can be easily seen that this application can find the full text content through the content of any level of data by classifying the data and storing the classified data in a hierarchical manner, thus realizing a flexible search method.

[0158] In this application, the data is classified so that a more fine-grained search can be performed during the search.

[0159] For example, suppose an article has 5 paragraphs and 18 sentences in total, where the number of sentences in these 5 paragraphs is 1, 3, 4, 6, and 4. Each sentence has one keyword, each paragraph has one keyword, and the entire article has one keyword.

[0160] AAA

[0161] BBB HHH HHH

[0162] CCC DDD FFFF DDD

[0163] BBB DDD AAA EEE BBB BBB

[0164] AAA BBB GGG BBB

[0165] The keyword of the first sentence is AAA. The keyword of the second sentence is BBB. The keyword of the third sentence is HHH. The keyword of the fourth sentence is HHH. The keyword of the fifth sentence is CCC. The keyword of the sixth sentence is DDD. The keyword of the seventh sentence is FFF. The keyword of the eighth sentence is DDD. The keyword of the ninth sentence is BBB. The keyword of the tenth sentence is DDD. The keyword of the eleventh sentence is AAA. The keyword of the twelfth sentence is EEE. The keyword of the thirteenth sentence is BBB. The keyword of the fourteenth sentence is BBB. The keyword of the fifteenth sentence is AAA. The keyword of the sixteenth sentence is BBB. The keyword of the seventeenth sentence is GGG. The keyword of the eighteenth sentence is BBB.

[0166] The key words for the first paragraph are AAA. The key words for the second paragraph are HHH. The key words for the third paragraph are DDD. The key words for the fourth paragraph are BBB. The key words for the fifth paragraph are BBB.

[0167] The keyword of the full text is BBB.

[0168] If the article is not graded, the article only has one keyword: BBB. When the search keyword is "BBB", the article can be retrieved; when the search keyword is any other keyword, the article cannot be retrieved, for example, when the search keyword is "AAA", the article cannot be retrieved.

[0169] The article is graded according to paragraphs. The article has a full-text-level keyword: BBB, and also has paragraph-level keywords: AAA, HHH, DDD, BBB. Among them, "BBB" is both a full-text-level keyword and a keyword for the 4th and 5th paragraphs. When searching for the keyword "AAA", the 1st paragraph can be retrieved; when searching for the keyword "HHH", the 2nd paragraph can be retrieved; when searching for the keyword "DDD", the 3rd paragraph can be retrieved; when searching for the keyword "BBB", the 4th paragraph, the 5th paragraph and the article can be retrieved. According to the method above in this application, after retrieving a paragraph, the ID of the upper-level data in the metadata of the DO at the paragraph level can be used to locate the article where the paragraph is located, thereby retrieving the article.

[0170] It can be seen that grading articles according to paragraphs and using paragraph-level keyword search during retrieval can effectively increase the probability of articles being retrieved.

[0171] The article is graded into paragraphs, and the paragraphs are graded into sentences. The article has a full-text-level keyword: BBB, paragraph-level keywords: AAA, HHH, DDD, BBB, and sentence-level keywords: AAA, BBB, HHH, CCC, DDD, FFFF, EEE, GGG.

[0172] Among them, "BBB" is not only a keyword at the full-text level, but also a keyword in the 4th and 5th paragraphs, as well as a keyword in the 2nd, 9th, 13th, 14th, 16th and 18th sentences.

[0173] When searching for the keyword "AAA", the 1st, 11th, and 15th sentences can be retrieved. When searching for the keyword "HHH", the 3rd and 4th sentences can be retrieved. When searching for the keyword "CCC", the 5th sentence can be retrieved. When searching for the keyword "DDD", the 6th, 8th, and 10th sentences can be retrieved. When searching for the keyword "FFF", the 7th sentence can be retrieved. When searching for the keyword "GGG", the 17th sentence can be retrieved.

[0174] It can be seen that the benefit of extracting keywords from data at each level is that it can greatly increase the probability of articles being retrieved during search.

[0175] As an optional implementation, after the data search step, association analysis may be performed based on the search results, that is, the DOs may be associated according to the needs of the user.

[0176] Each user looks at information from a different perspective. In order to extract information with different focuses, users propose a topic for the issues they care about and list a set of customized keywords (i.e., search keywords) based on the topic. The system will search the digital object architecture based on the keywords, connect the digital objects related to the keywords and the keyword DO, and finally connect all the keyword DO to form an information network built for the topic.

[0177] In this application, search keywords can be customized according to the actual needs of the user.

[0178] You can use the institution name as a search keyword to search for articles from different institutions, and conduct comparative analysis of articles from different institutions to determine their subjective positions on certain topics.

[0179] You can use the author's name as a search keyword to search for articles published by the author and observe how their areas of focus change over time.

[0180] Finally, a knowledge graph can be established within the domain to mine the correlations between entities.

[0181] Example 1

[0182] With the exponential growth of Internet information, data is becoming more and more fragmented and decentralized. Valuable information is often submerged in a lot of messy information, and the value of information can only be reflected after effective association. Therefore, this application hopes to use a new Internet architecture to propose an effective information organization method to identify and connect potentially valuable information.

[0183] At present, the organization of information mainly relies on search engines or crawlers + manual analysis, but this method is based on the traditional Internet architecture rather than the information management method based on DOA. DOA was proposed by Turing Award winners Robert Khan and Vincent Cerf, who are also the inventors of TCP / IP technology. TCP / IP is an Internet architecture with end-to-end transmission as the core, while DOA is an Internet architecture with data as the core, which pays more attention to the organization and processing of data. It is more suitable for organizing, connecting and analyzing massive fragmented data.

[0184] This application organizes information in four steps.

[0185] The first step is to collect data on the Internet. It mainly targets multivariate, multimodal, and heterogeneous data, such as online articles, videos and their subtitles, pictures and charts, comments on social networks, files containing pictures or texts, and external links contained in the above information. When collecting, it is necessary to mark the organization or author, generation time and title, and if possible, the target location associated with the information needs to be collected. Collection methods include automated collection and manual collection, such as distributed web crawlers or relevant experts who are very familiar with this field.

[0186] The second step is to structure and analyze the collected data to extract fragmented information. Different types of information have different ways of arrangement. The arrangement can be done with the help of natural language processing technology (NLP), manual precise labeling, or a combination of manual and automated processing methods.

[0187] The simplest way is to organize information according to its granularity. For text information on a website, it can be arranged in a tree structure from top to bottom. The top level is website information, followed by web page information. The web page contains different articles, and the sentences in the articles are the smallest units of digital objects.

[0188] In addition, the news can also be arranged according to the argument. For a piece of news, first mark its main argument, and then discuss the sub-arguments. Each sub-argument is summarized as an information unit. Organizing according to the elements of the news is also a feasible way, including the time, place, people and agenda of the news. It is worth mentioning that if the news has a special emphasis on certain elements, it can also be divided into different information units. For example, when there are multiple important people, each one can be considered as an information unit.

[0189] In addition to news, information can also be in the form of files. For example, a report contains multiple files. In this case, the report can be regarded as the parent information, and each file is the sub-information it contains. In addition to files, a web page may contain multiple links, so each link can also be regarded as a sub-information unit.

[0190] The third step requires reorganizing the fragmented information using the digital object architecture.

[0191] First, organize the information structure arranged in the second step into the form of hierarchical DO. The smallest unit of DO is a sentence, a file or a link. The upper DO includes the paragraph, sub-argument or news element of this sentence. The upper DO containing the paragraph is an article or a report. Multiple articles are arranged in the same level as the lower DO of the web page. The top level is the entire website, which includes multiple web pages.

[0192] After that, the DO information is divided into three parts and put into the three components of DOA. First, the DO's meta information, that is, the description information (Meta Data) is put into the Registry. Secondly, the unique identifier of each DO is registered in the HandleSystem and a Repository is allocated for DO storage. Finally, the DO data is put into the Repository.

[0193] The fourth step is to associate DOs according to the needs of users. Each user has a different perspective on information. In order to extract information with different focuses, users propose a topic for the issues they care about and list a set of customized keywords based on the topic. The system will search the digital object architecture based on the keywords, connect the digital objects related to the keywords and the keyword DO, and finally connect all the keyword DOs to form an information network built for the topic.

[0194] Example 2

[0195] Information analysis using the DOA system mainly includes four steps: step one, information collection; step two, structured analysis; step three, reorganization based on DOA; step four, correlation analysis.

[0196] Step 1: Information Collection

[0197] This example mainly uses automated crawlers in information collection. First, the information sources of focus are determined, such as the websites of major news media.

[0198] After determining the information source, it is necessary to determine the scope of crawling, such as whether to track external links within the target website, how many layers to crawl, whether to crawl files or pictures, etc. Usually, as much relevant information as possible will be collected, because the searched data may serve as the information source for the digital objects DO generated in the next few steps.

[0199] Step 2: Structured analysis

[0200] Structural analysis is a very important step in information organization, which helps to parse and organize data in a fine-grained manner. Related technologies of natural language processing are needed here, such as lexical analysis, syntactic analysis, dependency analysis, etc.

[0201] In this example, the simplest way is to perform structural analysis by paragraph and sentence. Paragraph analysis is relatively simple, and only needs to segment the article document according to the carriage return. However, sentences cannot be segmented by periods like paragraphs, because some abbreviations such as USA also contain periods, so it is necessary to use the abbreviation corpus to assist in sentence segmentation.

[0202] In addition to segmenting the paragraphs and sentences of the article, we also hope to extract the attributes of the article, such as the author of the article, the organization of the article, etc. If there is a knowledge base in a professional field, we can also annotate the entities in the sentence, such as the location, the unit, etc. This will help with the subsequent information collation and organization.

[0203] Step 3: Reorganize based on DOA

[0204] In this step, Cordra is used to convert structured information into DO, and the article-based information source is used as an example. First, each article is taken as a DO, and the author and organization of the article are taken as the attributes of this DO. Then the paragraphs in the article are taken as sub-DOs, and the DOID of the corresponding article of the paragraph is annotated as an attribute in the paragraph DO. Finally, the sentences in the article are generated as the next level DO, and the DOID of the corresponding paragraph of the sentence is annotated as an attribute in the sentence DO.

[0205] After the DO is generated, the metadata is placed in the Registry according to the DOA process, the DOID is registered in the HandleSystem, and finally the data is stored in the Repository.

[0206] Step 4: Correlation Analysis

[0207] Finally, the generated DO is used for association analysis. This step integrates, analyzes and mines the information collected from different sources. For example, articles from different institutions can be compared and analyzed to determine their subjective positions on certain topics. Different authors can also be tracked to observe changes in their areas of focus. Finally, a knowledge graph can be established in the field to mine the relevance of entities and conduct association analysis. Figure 2 It is a schematic diagram showing a correlation analysis result according to an exemplary embodiment.

[0208] The embodiment of the present application also provides a data classification processing device, which can execute the above-mentioned data classification processing method.

[0209] The device is based on a digital object system, which includes a registration table, a registration system, and a data warehouse.

[0210] like Figure 3 As shown, the device includes: a data acquisition unit 10, a data classification unit 20, a classification registration and storage unit 30 based on a digital object system, and a data search unit 40.

[0211] The data acquisition unit 10 is used to acquire data to be processed.

[0212] The data classification unit 20 is used to classify the data to be processed to obtain classified data.

[0213] The hierarchical registration and storage unit 30 based on the digital object system is used to: determine the digital objects of each level according to the hierarchical data, the digital objects include metadata and specific data, the metadata includes the hierarchical information of the digital objects, which is convenient for future retrieval; for the digital objects of each level, perform the following steps: store the metadata of the digital object in the registration table; register the unique identifier of the digital object in the registration system and allocate a data warehouse; store the specific data of the digital object in the data warehouse, the specific data is information reflecting the content of the data to be processed itself, the metadata is used to describe the digital object, the metadata includes the unique identifier of the digital object and the label of the digital object, the label of the digital object is used to match the search keyword when the user searches for the digital object.

[0214] The data search unit 40 is used to: determine the search keyword; match the search keyword with the tag in the metadata, screen out the digital objects corresponding to the successfully matched tags and obtain their unique identifiers; query the data warehouse storing the specific data of the digital objects from the registration system according to the unique identifier; and obtain the specific data of the digital objects from the queried data warehouse.

[0215] Optionally, the metadata further includes hierarchical data, and the hierarchical data includes unique identifiers of digital objects at the previous level and / or the next level.

[0216] Optionally, the device further includes a data search unit, which is used to: determine the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determine the digital object of the previous level step by step according to the hierarchical data until the digital object of the highest level is determined; determine all digital objects of the next level step by step according to the digital object of the highest level until all digital objects of the lowest level are determined; query the data warehouse storing all digital objects of the lowest level from the registration system; query the specific data of all digital objects of the lowest level from the data warehouse. Optionally, the data search unit is used to: determine the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determine the digital object of the highest level according to the hierarchical data; query the data warehouse storing the digital object of the highest level from the registration system; query the specific data of the digital object of the highest level from the data warehouse.

[0217] Optionally, the unique identifiers of data objects at different levels are distinguished using a preset method.

[0218] Optionally, the device further comprises: a query unit, configured to restore the content of the data to be processed according to the specific data of the digital objects at multiple levels and the hierarchical data in the metadata that are queried.

[0219] Optionally, the device further includes an adding unit, which is used to: determine the digital object of the previous level step by step according to the hierarchical data in the metadata until the digital object of the highest level is determined; and add the newly added hierarchical data to the metadata of the digital object of the highest level.

[0220] Figure 4 It is a block diagram of an electronic device according to an exemplary embodiment.

[0221] Refer to the following Figure 4 The electronic device 400 according to this embodiment of the present application is described. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0222] like Figure 4 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), a display unit 440, etc.

[0223] The storage unit stores a program code, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 410 can perform the following steps: Figure 1 Follow the steps shown in .

[0224] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 4201 and / or a cache memory unit 4202 , and may further include a read-only memory unit (ROM) 4203 .

[0225] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0226] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0227] The electronic device 400 may also communicate with one or more external devices 400' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) to enable a user to communicate with the electronic device 400, and / or any device (e.g., routers, modems, etc.) that the electronic device 400 can communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface 450. In addition, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 460. The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0228] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 5 As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0229] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0230] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0231] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0232] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium realizes the following functions: obtaining data to be processed; grading the data to be processed to obtain graded data; determining digital objects of each level according to the graded data, and the digital objects include metadata and specific data; for digital objects of each level, performing the following steps: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in the registration system and assigning a data warehouse; storing the specific data of the digital object in the data warehouse, the specific data is information reflecting the content of the data to be processed itself, and the metadata is used to describe the digital object, and the metadata includes the unique identifier of the digital object and the tag of the digital object, and the tag of the digital object is used to match the search keyword when the user searches for the digital object.

[0233] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0234] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0235] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A data classification processing method, characterized in that: The method comprises: Get the data to be processed; Classifying the data to be processed to obtain classified data; Determine digital objects at each level according to the hierarchical data, wherein the digital objects include metadata and specific data; For each level of digital objects, the following steps are performed: storing the metadata of the digital object in a registration table; registering the unique identifier of the digital object in a registration system and assigning a data warehouse; storing the specific data of the digital object in the data warehouse, wherein the specific data is information reflecting the content of the data to be processed, and the metadata is used to describe the digital object, and the metadata includes the unique identifier of the digital object and the label of the digital object, and the label of the digital object is used to match the search keyword when the user searches for the digital object; Determine the search keywords; Matching the search keyword with the tags in the metadata, screening out the digital objects corresponding to the successfully matched tags and obtaining their unique identifiers; querying a data warehouse storing specific data of the digital object from the registration system according to the unique identifier; Get the specific data of the digital object from the queried data warehouse, The metadata also includes hierarchical data, which includes unique identifiers of digital objects at the previous level and / or the next level. The unique identifiers of data objects at different levels are distinguished using a preset method. The method further comprises: Determine the hierarchical data in the metadata of the digital object corresponding to the tag that matches successfully; determine the digital object of the previous level step by step according to the hierarchical data until the digital object of the highest level is determined; determine all the digital objects of the next level step by step according to the digital object of the highest level until all the digital objects of the lowest level are determined; query all the data warehouses storing the digital objects of the lowest level from the registration system; query the specific data of all the digital objects of the lowest level from the data warehouse, or, Determine the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determine the highest-level digital object according to the hierarchical data; query the data warehouse storing the highest-level digital object from the registration system; query the specific data of the highest-level digital object from the data warehouse.

2. The data classification processing method according to claim 1, characterized in that: The method further comprises: Determine the digital object of the previous level step by step according to the hierarchical data in the metadata until the digital object of the highest level is determined; The newly added level of data is added to the metadata of the top-level digital object.

3. A data classification processing device, characterized in that: The device comprises: a data acquisition unit, a data classification unit, a classification registration and storage unit based on a digital object system, and a search unit. The data acquisition unit is used to: acquire data to be processed; The data classification unit is used to: classify the data to be processed to obtain classified data; The hierarchical registration and storage unit based on the digital object system is used to: determine the digital objects of each level according to the hierarchical data, and the digital objects include metadata and specific data; for the digital objects of each level, perform the following steps: store the metadata of the digital object in the registration table; register the unique identifier of the digital object in the registration system and allocate a data warehouse; store the specific data of the digital object in the data warehouse, and the specific data is information reflecting the content of the data to be processed itself, and the metadata is used to describe the digital object, and the metadata includes the unique identifier of the digital object and the label of the digital object, and the label of the digital object is used to match the search keyword when the user searches for the digital object. The search unit is used to: determine a search keyword; match the search keyword with the tag in the metadata, screen out the digital object corresponding to the successfully matched tag and obtain its unique identifier; query the data warehouse storing the specific data of the digital object from the registration system according to the unique identifier; obtain the specific data of the digital object from the queried data warehouse, The metadata also includes hierarchical data, which includes unique identifiers of digital objects at the previous level and / or the next level. The unique identifiers of data objects at different levels are distinguished using a preset method. The device also includes a data search unit, which is used to: determine the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determine the digital object of the previous level step by step according to the hierarchical data until the digital object of the highest level is determined; determine all digital objects of the next level step by step according to the digital object of the highest level until all digital objects of the lowest level are determined; query all data warehouses storing digital objects of the lowest level from the registration system; query specific data of all digital objects of the lowest level from the data warehouse, or the data search unit is used to: determine the hierarchical data in the metadata of the digital object corresponding to the successfully matched tag; determine the digital object of the highest level according to the hierarchical data; query the data warehouse storing digital objects of the highest level from the registration system; query specific data of digital objects of the highest level from the data warehouse.

4. The data hierarchical processing device according to claim 3, characterized in that: The device further comprises an adding unit, the adding unit being used to: determine the digital object of the previous level step by step according to the level data in the metadata until the digital object of the highest level is determined; The newly added level of data is added to the metadata of the top-level digital object.

5. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data classification processing method as described in any one of claims 1-2.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data classification processing method as described in any one of claims 1-2 is implemented.

Images