An artificial intelligence-based distributed data management system and method

Abstract

The application discloses a kind of distributed data management system and method based on artificial intelligence, it is related to data processing field, including: upload module, for uploading the data content needing to be executed storage and management, the uploaded data content is placed in temporary storage in storage medium;Feature recognition module is used to traverse the data content temporarily stored in storage medium, and the feature of data content is identified;Analysis module is used to obtain the feature recognition result of data content in feature recognition module, and the feature correlation of data content is analyzed based on feature recognition result, the uploaded data of the application can be automatically cleaned, remove duplicate, improve storage utilization rate.Can accurately identify the feature of different format data, mine the association between data, provide valuable information for users.In storage management, according to the fine distinction of storage according to data characteristics, improve storage management accuracy.Users can also customize key preset value, meet diversified needs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an artificial intelligence-based distributed data management system and method. Background Art

[0002] Data storage management is the process of effectively storing, organizing, and maintaining data. It involves selecting appropriate storage media and technologies, ensuring data security, integrity, and efficient access, as well as performing data backup and recovery, and properly allocating and managing storage space.

[0003] The invention patent application with application number 202411118702.X discloses a distributed data matching method based on artificial intelligence, which includes: constructing a data set, collecting historical matching data in the data set, and dividing the historical matching data; obtaining feature data in the historical matching data, and analyzing the association rules of each feature data in the historical matching data, and generating a data model at the same time: tracing data in the historical matching data, collecting procurement or manufacturing processes corresponding to the matching data, uploading the data traceability information to the data set, and updating the data set; obtaining data with the most historical matches in the data set, collecting association features between the data and the procurement or manufacturing process, and data labeling the association features. This application aims to solve the problem that "existing distributed data matching technology is difficult to connect the upstream, midstream and downstream data information of intelligent manufacturing when matching and searching for data in the production or procurement process, sometimes resulting in incomplete matching data information, thereby affecting the accuracy of data matching, reducing data processing and matching efficiency, and causing delays in procurement or production of intelligent manufacturing."

[0004] However, for massive amounts of network data content, existing technologies, when performing storage management, mostly perform differentiated management based on simple conditions such as data update time and data format. The accuracy of this differentiated management is limited, so when configuring a search engine to retrieve the target data content, the engine load is large and the retrieval takes a long time.

[0005] Therefore, an artificial intelligence-based distributed data management system and method are proposed. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a distributed data management system and method based on artificial intelligence, which can effectively solve the problems of the prior art.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] The present invention discloses a distributed data management system based on artificial intelligence, comprising:

[0009] The upload module is used to upload data content that needs to be stored and managed, and temporarily store the uploaded data content in the storage medium; the feature recognition module is used to traverse the data content temporarily stored in the storage medium and perform feature recognition on the data content; the analysis module is used to obtain the feature recognition results of the data content in the feature recognition module, analyze the feature correlation of the data content based on the feature recognition results, and output the related data content based on the data content feature correlation analysis results; the management module is used to receive the feature recognition results of each data content in the feature recognition module, distinguish the data content based on the feature recognition results, and forward the data content to the cloud database based on the distinction results; the cloud database is used to obtain the data content forwarded from the management module, and perform differentiated storage operations on the data content based on its differentiation results in the management module; the control module is used to monitor whether there is new data content uploaded in the upload module, and when the monitoring result is yes, control the system to refresh and run

[0010] Furthermore, the data content uploaded in the upload module originates from any authorized system end user, the storage space size of the storage medium is manually configured by any system end user, and when the data content is temporarily stored in the storage medium, it is sent to the storage medium via medium transmission. Each time new temporarily stored data content is added to the storage medium, the storage medium performs a data cleaning operation through a preset built-in program;

[0011] Among them, when the data content performs data cleaning operations based on a preset built-in program in the storage medium, it obeys: the data content is distinguished based on the data format, and a data cleaning operation is performed in each partition interval. During the data cleaning operation stage, the similarity of each data content in the same partition interval is compared, and any data in a group of data with a similarity of 100% is used as a deletion target, and a deletion operation is performed.

[0012] Furthermore, the feature recognition module is provided with a picking unit and an extraction unit at the lower level. The picking unit is used for data content containing text information and picking up characteristic text information in the data content. The extraction module is used for data content in image format and extracting characteristic information in the data content.

[0013] Among them, the feature recognition module runs continuously. Each time the feature recognition module runs, it takes the data content stored in a partitioned interval in the storage medium as the processing target, identifies the processing target format, and when the processing target format is data content containing text information, sends each data in the processing target to the picking unit in sequence. When the processing target format is an image format, sends each data in the processing target to the extraction unit in sequence.

[0014] Furthermore, during the operation phase of the picking unit, after receiving the data content, the picking logic of the characteristic text information in the data content is expressed as follows:

[0015] The data content is segmented to obtain a word set, which is recorded as W = {w1, w2, ..., w n}, calculate the frequency score, part of speech score, and position score of each word respectively;

[0016]

[0017] Total word score: S(ω i )=S f (ω i )×α+pos(w i )×β+S l ×γ;

[0018] Where: S(ω i ) is the total score of the word; S f (ω i ),pos(w i ), S l is the word frequency score, part of speech score, and position score; α, β, and γ are weights; n is the total number of words in the text; f(ω i ) is a word (w i ) appears in the text; f(ω j ) is the sum of the number of occurrences of all words in the text;

[0019] in, Indicates that when the word is a noun, verb, adjective, or other part of speech, its corresponding values ​​are 0.4, 0.3, 0.2, and 0.1. It means that when the word appears in the first 10% or last 10% of the text or is the beginning word of a paragraph, the value is 0.3, and the value is 0.1 in other positions. After the total score of each word is calculated, the words are sorted in descending order based on the score, and the top 5% of the words in the descending order are selected as feature text information.

[0020] Furthermore, during the operation phase of the picking unit, after receiving the data content, the logic for extracting the feature information in the data content is expressed as follows:

[0021] Logic 1: Obtain data content and identify objects and object names in the data content based on the target recognition algorithm;

[0022] Logic 2: Traverse the data content, identify the color value of each pixel in the data content, and based on the color value identification results, count the number of pixels corresponding to each color value;

[0023] The Logic 2, when executed, takes the pixels in the region image other than the region image representing the object as the identification and statistical target after the object and the object name are identified by the Logic 1, and records the statistical result of the object name and the number of pixels corresponding to each color value as the feature information.

[0024] Further, the analysis module records the identification result of the feature text information and the feature information as data set 1 and data set 2 when analyzing the correlation of the data content features, takes each group of feature information in the data set 2 as the analysis target, and performs feature correlation analysis on each feature text information in the data set 1.

[0025] The feature correlation analysis stage takes the object name in the feature information as the comparison content, compares with the feature text information, and records the data content from which each of the feature information and the feature text information is derived as the associated data content with the highest matching degree.

[0026] Further, the matching degree comparison logic of the comparison content and the feature text information is represented as:

[0027]

[0028] In the formula, K is the matching degree of the comparison content and the feature text information; m(a x ∩b x ) is the number of intersections of the object name in the comparison content a x and the words in the feature text information b x in the data set 1; is the total number of object names in the comparison content a x and the total number of words in the feature text information b x ; d(r, t) is the semantic similarity of the rth object name and the tth word; is the average symbol;

[0029] In the formula, the larger K represents the higher matching degree of the comparison content and the feature text information. The similarity of each comparison content and each feature text information is obtained by the above formula, and the comparison content with the highest matching degree of each feature text information is obtained. The data content from which the comparison content is derived and the data content from which the feature text information is derived are recorded as the associated data content.

[0030] Furthermore, when distinguishing data content, the management module distinguishes data content containing text information based on the semantic similarity of each of its characteristic text information, so that the semantic similarity between data content containing text information stored in each distinction interval is not less than a preset value of one; and distinguishes data content in image format based on the repetition rate of object names in each of its characteristic information, so that the repetition rate of object names in the characteristic information of data content in image format stored in each distinction interval is not less than a preset value of two;

[0031] After the data content is differentiated based on the management module, the differentiated storage intervals storing the data content containing text information are used as processing targets, the associated data content of each data content in the processing target is obtained, the associated data content is retrieved from the differentiated storage interval where the associated data content is located, and transmitted to the processing target;

[0032] Among them, the preset value 1 and the preset value 2 are both customized by the system end user. The preset value 1 and the preset value 2 are initially set to 80% and 50%. The data content stored in the cloud database is deleted synchronously in the upload module.

[0033] Furthermore, the upload module is interactively connected to the feature recognition module via a wireless network, the feature module is interactively connected to the picking unit and the extraction unit via a wireless network, the feature recognition module is interactively connected to the analysis module and the management module via a wireless network, the management module is interactively connected to the cloud database via a wireless network, and the cloud database is interactively connected to the control module via a wireless network.

[0034] In another aspect, a distributed data management method based on artificial intelligence includes:

[0035] Step 1: Upload data content, identify the basic format of the data content, differentiate the data content based on the basic format of the data content, and temporarily store it using storage media;

[0036] Step 2: Clean the data stored in the storage medium, perform feature recognition on the cleaned data, and analyze the relevance of each data content based on the feature recognition results to obtain related data content;

[0037] Step 3: Obtain the feature recognition results of each data content, analyze the matching degree of each data content, i.e., semantic similarity, based on the comparison of the feature recognition results of each data content, and distinguish the data content by combining the semantic similarity and data content matching degree;

[0038] Step 4: Create a cloud database and use the cloud database to store the data content. During the data storage phase, the cloud database stores the data content based on the differentiation results of the data content in the previous step.

[0039] Step 5: When new data content is uploaded, the refresh step is executed.

[0040] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0041] The present invention can efficiently clean the uploaded data, automatically identify and delete duplicate data, reduce redundant storage, and improve the utilization rate of storage space. For data in different formats, whether it is data containing text information or data in image format, it can accurately identify its characteristics, find out the related data content by analyzing the feature correlation of the data content, and provide users with more valuable data correlation information. In terms of storage management, the system can make fine distinctions and store the data according to the characteristics of the data, such as the semantic similarity of text information and the repetition rate of object names in image information, so that the storage management accuracy is greatly improved. When uploading to the cloud for storage, it can be customized through preset values ​​to meet the diverse needs of different users. After cloud storage, the original uploaded data will be deleted synchronously, further ensuring the security and efficiency of data management. Based on this, the efficiency and accuracy of later data in searching are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0043] Figure 1 This is a structural diagram of a distributed data management system based on artificial intelligence;

[0044] Figure 2 The figure is a flowchart of a distributed data management method based on artificial intelligence. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] The present invention will be further described below with reference to the embodiments.

[0047] Example 1:

[0048] This embodiment is a distributed data management system based on artificial intelligence, such as Figure 1 Shown, including:

[0049] The upload module is used to upload data content that needs to be stored and managed, and temporarily store the uploaded data content in the storage medium;

[0050] The data content uploaded in the upload module comes from any authorized system end user. The storage space size of the storage medium is manually configured by any system end user. When the data content is temporarily stored in the storage medium, it is sent to the storage medium via media transmission. Every time the temporarily stored data content is added to the storage medium, the storage medium performs data cleaning operations through a preset built-in program;

[0051] Among them, when the data content is stored in the storage medium and the data cleaning operation is performed based on the preset built-in program, the data content is divided based on the data format, and a data cleaning operation is performed in each divided interval. During the data cleaning operation phase, the similarity of each data content in the same divided interval is compared, and any data in a group of data with a similarity of 100% is selected as a deletion target and a deletion operation is performed;

[0052] A feature recognition module is used to traverse the data content temporarily stored in the storage medium and perform feature recognition on the data content;

[0053] The data content uploaded in the upload module comes from any authorized system end user. The storage space size of the storage medium is manually configured by any system end user. When the data content is temporarily stored in the storage medium, it is sent to the storage medium via media transmission. Every time the temporarily stored data content is added to the storage medium, the storage medium performs data cleaning operations through a preset built-in program;

[0054] Among them, when the data content is stored in the storage medium and the data cleaning operation is performed based on the preset built-in program, the data content is divided based on the data format, and a data cleaning operation is performed in each divided interval. During the data cleaning operation phase, the similarity of each data content in the same divided interval is compared, and any data in a group of data with a similarity of 100% is selected as a deletion target and a deletion operation is performed;

[0055] During the operation phase of the picking unit, after receiving the data content, the picking logic of the characteristic text information in the data content is expressed as follows:

[0056] The data content is segmented to obtain a word set, which is recorded as W = {w1, w2, ..., w n}, calculate the frequency score, part of speech score, and position score of each word respectively;

[0057]

[0058] Total word score: S(ω i )=S f (ω i )×α+pos(w i )×β+S l ×γ;

[0059] Where: S(ω i ) is the total score of the word; S f (ω i ),pos(w i ), S l is the word frequency score, part of speech score, and position score; α, β, and γ are weights; n is the total number of words in the text; f(ω i ) is a word (w i ) appears in the text; f(ω j ) is the sum of the number of occurrences of all words in the text;

[0060] in, Indicates that when the word is a noun, verb, adjective, or other part of speech, its corresponding values ​​are 0.4, 0.3, 0.2, and 0.1. Indicates that when a word appears in the first 10% or last 10% of the text or at the beginning of a paragraph, it takes a value of 0.3, and in other positions it takes a value of 0.1. After calculating the total score of each word, each word is sorted in descending order based on the score, and the top 5% of the words in the descending order are selected as the feature text information;

[0061] By calculating the above logic formula, the words from the data content are scored, and this is used as the picking logic of the feature text information.

[0062] During the operation phase of the picking unit, after receiving the data content, the logic for extracting feature information from the data content is expressed as follows:

[0063] Logic 1: Obtain data content and identify objects and object names in the data content based on the target recognition algorithm;

[0064] Logic 2: Traverse the data content, identify the color value of each pixel in the data content, and based on the color value identification results, count the number of pixels corresponding to each color value;

[0065] When Logic 2 is executed, after identifying the object and its name using Logic 1, the pixels in the area image outside the area image representing the object are used as the identification and statistical targets. The statistical results of the object name and the number of pixels corresponding to each color value are recorded as feature information.

[0066] An analysis module is used to obtain the feature recognition results of the data content in the feature recognition module, analyze the feature relevance of the data content based on the feature recognition results, and output the related data content using the data content feature relevance analysis results;

[0067] When analyzing the relevance of data content features, the analysis module records the feature text information and the recognition results of the feature information as data set 1 and data set 2. Each set of feature information in data set 2 is used as the analysis target, and feature relevance analysis is performed with each feature text information in data set 1.

[0068] In the feature correlation analysis phase, the object name in the feature information is used as the comparison content and compared with the feature text information. The feature information and feature text information with the highest matching degree are recorded as the source data content of each feature information and the feature text information as the associated data content.

[0069] The logic for comparing the matching degree between the content and the feature text information is expressed as follows:

[0070]

[0071] Where: K is the matching degree between the comparison content and the feature text information; m(a x ∩b x ) is the comparison content a in the feature information x Contains the object name and the feature text information b in the dataset x The number of intersections containing words; To compare content a x The total amount of object names and characteristic text information b x The total number of words in the sentence; d(r,t) is the semantic similarity between the r-th object name and the t-th word; To find the mean symbol;

[0072] The larger K is, the higher the matching degree between the comparison content and the characteristic text information is. Based on the above formula, the similarity between each comparison content and each characteristic text information is calculated and compared to obtain the comparison content with the highest matching degree for each characteristic text information. The data content to which the characteristic information of the comparison content belongs and the data content of the characteristic text information source are recorded as the associated data content.

[0073] By calculating the above logic formula, the matching degree between the content and the characteristic text information is compared and analyzed to provide support for further operation of the management module of the system in this embodiment.

[0074] The management module is used to receive the feature recognition results of each data content in the feature recognition module, distinguish the data content based on the feature recognition results, and forward the data content to the cloud database based on the distinction results;

[0075] When distinguishing data content, the management module distinguishes data content containing text information based on the semantic similarity of each of its characteristic text information, so that the semantic similarity between data content containing text information stored in each distinction interval is not less than a preset value of one. It also distinguishes data content in image format based on the repetition rate of object names in each of its characteristic information, so that the repetition rate of object names in the characteristic information of data content in image format stored in each distinction interval is not less than a preset value of two.

[0076] After the data content is divided based on the management module, the divided storage intervals of each data content containing text information are used as processing targets, the associated data content of each data content in the processing target is obtained, the associated data content is retrieved from the divided storage interval where the associated data content is located, and transmitted to the processing target;

[0077] Among them, the preset value 1 and the preset value 2 are both customized by the system end user. The initial default settings of the preset value 1 and the preset value 2 are 80% and 50%. The data content stored in the cloud database is deleted synchronously in the upload module;

[0078] The cloud database is used to obtain the data content forwarded by the management module and perform differentiated storage operations on the data content based on the differentiation results in the management module;

[0079] The control module is used to monitor whether there is new data content uploaded to the upload module. When the monitoring result is yes, the system refreshes and runs;

[0080] The upload module is interactively connected to the feature recognition module through a wireless network, the feature module is interactively connected to the picking unit and the extraction unit through a wireless network, the feature recognition module is interactively connected to the analysis module and the management module through a wireless network, the management module is interactively connected to the cloud database through a wireless network, and the cloud database is interactively connected to the control module through a wireless network.

[0081] In this embodiment, the upload module uploads data content that needs to be stored and managed, and temporarily stores the uploaded data content in a storage medium. The feature recognition module further traverses the data content temporarily stored in the storage medium and performs feature recognition on the data content. The picking unit synchronizes the data content containing text information and picks up feature text information in the data content. The extraction module extracts feature information from the data content in real-time image format. The analysis module is post-operated to obtain feature recognition results of the data content in the feature recognition module, analyze the feature correlation of the data content based on the feature recognition results, and output related data content based on the data content feature correlation analysis results. The management module then receives the feature recognition results of each data content in the feature recognition module, distinguishes the data content based on the feature recognition results, and forwards the data content to the cloud database based on the distinction results. The cloud database obtains the data content forwarded from the management module and performs a distinguishing storage operation on the data content based on the distinction results in the management module. Finally, the control module monitors whether new data content is uploaded to the upload module. When the monitoring result is yes, the control system refreshes and runs.

[0082] Through the operation of the system in the above embodiment, a new, more refined technology is effectively provided for data storage management, which is conducive to later data retrieval and query, and ensures the stability of massive data in storage scenarios.

[0083] Example 2:

[0084] In terms of specific implementation, based on Example 1, this example refers to Figure 2 The distributed data management system based on artificial intelligence in Example 1 is further described in detail:

[0085] A distributed data management method based on artificial intelligence, comprising:

[0086] Step 1: Upload data content, identify the basic format of the data content, differentiate the data content based on the basic format of the data content, and temporarily store it using storage media;

[0087] Step 2: Clean the data stored in the storage medium, perform feature recognition on the cleaned data, and analyze the relevance of each data content based on the feature recognition results to obtain related data content;

[0088] Step 3: Obtain the feature recognition results of each data content, analyze the matching degree of each data content, i.e., semantic similarity, based on the comparison of the feature recognition results of each data content, and distinguish the data content by combining the semantic similarity and data content matching degree;

[0089] Step 4: Create a cloud database, apply the cloud database to store data content, and store the data content in the cloud database. Based on the results of the data content differentiation in the previous step, the data content is stored differently.

[0090] Step 5: When new data content is uploaded, refresh the step execution.

[0091] In summary, the system in the above embodiments can automatically clean up uploaded data and remove duplicates to improve storage utilization. It can accurately identify the characteristics of different formats of data, mine the associations between data, and provide valuable information to users. In terms of storage management, it can store data according to its characteristics to improve the accuracy of storage management. Users can also customize key preset values to meet diverse needs. After storing data in the cloud, the data on the upload end is automatically deleted to ensure the security of data management, and the overall data management is efficient, accurate, and secure.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some technical features; and these modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed data management system based on artificial intelligence, characterized in that: include: The upload module is used to upload data content that needs to be stored and managed, and temporarily store the uploaded data content in the storage medium; A feature recognition module is used to traverse the data content temporarily stored in the storage medium and perform feature recognition on the data content; The feature recognition module is provided with a picking unit and an extraction unit at the lower level. The picking unit is used for data content containing text information and picking up feature text information in the data content. The extraction module is used for data content in image format and extracting feature information in the data content. The feature recognition module operates continuously, and each time the feature recognition module operates, the feature recognition module uses the data content stored in a partitioned interval in the storage medium as a processing target, identifies the format of the processing target, and when the processing target format is data content containing text information, sequentially sends each data in the processing target to the pickup unit; when the processing target format is an image format, sequentially sends each data in the processing target to the extraction unit; During the operation phase of the pickup unit, after receiving the data content, the logic for extracting the feature information in the data content is expressed as follows: Logic 1: Obtain data content and identify objects and object names in the data content based on the target recognition algorithm; Logic 2: Traverse the data content, identify the color value of each pixel in the data content, and based on the color value identification results, count the number of pixels corresponding to each color value; When Logic 2 is executed, after identifying the object and its name using Logic 1, the pixels in the area image outside the area image representing the object are used as the identification and statistical targets. The statistical results of the object name and the number of pixels corresponding to each color value are recorded as feature information. An analysis module is used to obtain the feature recognition results of the data content in the feature recognition module, analyze the feature relevance of the data content based on the feature recognition results, and output the related data content using the data content feature relevance analysis results; When analyzing the relevance of data content features, the analysis module records the feature text information and the recognition results of the feature information as data set 1 and data set 2, takes each set of feature information in data set 2 as the analysis target, and performs feature relevance analysis with each feature text information in data set 1; In the feature correlation analysis stage, the object name in the feature information is used as the comparison content and compared with the feature text information, and the feature information and feature text information with the highest matching degree are recorded as the source data content of each feature information and feature text information as the associated data content; A management module is used to receive the feature recognition results of each data content in the feature recognition module, distinguish the data content based on the feature recognition results, and forward the data content to the cloud database based on the distinction results; The cloud database is used to obtain the data content forwarded by the management module and perform differentiated storage operations on the data content based on the differentiation results in the management module; The control module is used to monitor whether there is new data content uploaded to the upload module. When the monitoring result is yes, the system refreshes and runs.

2. The distributed data management system based on artificial intelligence according to claim 1, characterized in that: The data content uploaded in the upload module comes from any authorized system end user. The storage space size of the storage medium is manually configured by any system end user. When the data content is temporarily stored in the storage medium, it is sent to the storage medium via medium transmission. Each time the temporarily stored data content is added to the storage medium, the storage medium performs a data cleaning operation through a preset built-in program. Among them, when the data content performs data cleaning operations based on a preset built-in program in the storage medium, it obeys: the data content is distinguished based on the data format, and a data cleaning operation is performed in each partition interval. During the data cleaning operation stage, the similarity of each data content in the same partition interval is compared, and any data in a group of data with a similarity of 100% is used as a deletion target, and a deletion operation is performed.

3. The distributed data management system based on artificial intelligence according to claim 1, characterized in that: During the operation phase of the picking unit, after receiving the data content, the picking logic of the characteristic text information in the data content is expressed as follows: The data content is segmented to obtain a word set, which is recorded as W = {w1, w2, ..., w n }, calculate the frequency score, part of speech score, and position score of each word respectively; Total word score: S(ω i )=S f (ω i )×α+pos(w i )×β+S l ×γ; Where: S(ω i ) is the total score of the word; S f (ω i ),pos(w i ), S l is the word frequency score, part of speech score, and position score; α, β, and γ are weights; n is the total number of words in the text; f(ω i ) is a word (w i ) appears in the text; f(ω j ) is the sum of the number of occurrences of all words in the text; in, Indicates that when the word is a noun, verb, adjective, or other part of speech, its corresponding values ​​are 0.4, 0.3, 0.2, and 0.

1. It means that when the word appears in the first 10% or last 10% of the text or is the beginning word of a paragraph, the value is 0.3, and the value is 0.1 in other positions. After the total score of each word is calculated, the words are sorted in descending order based on the score, and the top 5% of the words in the descending order are selected as feature text information.

4. The distributed data management system based on artificial intelligence according to claim 1, characterized in that: The logic for comparing the matching degree between the comparison content and the characteristic text information is expressed as follows: Where: K is the matching degree between the comparison content and the feature text information; m(a x ∩b x ) is the comparison content a in the feature information x Contains the object name and the feature text information b in the dataset x The number of intersections containing words; To compare content a x The total amount of object names and characteristic text information b x The total number of words in the sentence; d(r,t) is the semantic similarity between the r-th object name and the t-th word; To find the mean symbol; Among them, the larger K is, the higher the matching degree between the comparison content and the feature text information is. Based on the above formula, the similarity between each comparison content and each feature text information is calculated for comparison to obtain the comparison content with the highest matching degree for each feature text information. The data content belonging to the feature information of the comparison content source and the data content of the feature text information source are recorded as associated data content.

5. The distributed data management system based on artificial intelligence according to claim 1, characterized in that: When distinguishing data content, the management module distinguishes data content containing text information based on the semantic similarity of each of its characteristic text information, so that the semantic similarity between data content containing text information stored in each distinction interval is not less than a preset value of one; and distinguishes data content in image format based on the repetition rate of object names in each of its characteristic information, so that the repetition rate of object names in the characteristic information of data content in image format stored in each distinction interval is not less than a preset value of two; After the data content is differentiated based on the management module, the differentiated storage intervals storing the data content containing text information are used as processing targets, the associated data content of each data content in the processing target is obtained, the associated data content is retrieved from the differentiated storage interval where the associated data content is located, and transmitted to the processing target; Among them, preset value 1 and preset value 2 are customized by the system user. The initial default settings of preset value 1 and preset value 2 are 80% and 50%, respectively. The data content is stored in the cloud database and is deleted synchronously in the upload module.

6. The distributed data management system based on artificial intelligence according to claim 1, characterized in that: The upload module is interactively connected to the feature recognition module via a wireless network, the feature recognition module is interactively connected to the picking unit and the extraction unit via a wireless network, the feature recognition module is interactively connected to the analysis module and the management module via a wireless network, the management module is interactively connected to the cloud database via a wireless network, and the cloud database is interactively connected to the control module via a wireless network.

7. A distributed data management method based on artificial intelligence, wherein the method is an implementation method of the distributed data management system based on artificial intelligence according to any one of claims 1 to 6, characterized in that: include: Step 1: Upload data content, identify the basic format of the data content, differentiate the data content based on the basic format of the data content, and temporarily store it using storage media; Step 2: Clean the data stored in the storage medium, perform feature recognition on the cleaned data, and analyze the relevance of each data content based on the feature recognition results to obtain related data content; Step 3: Obtain the feature recognition results of each data content, analyze the matching degree and semantic similarity of each data content based on the comparison of the feature recognition results of each data content, and distinguish the data content by combining the semantic similarity and data content matching degree; Step 4: Create a cloud database and use the cloud database to store the data content. During the data storage phase, the cloud database stores the data content based on the differentiation results of the data content in the previous step. Step 5: When new data content is uploaded, the refresh step is executed.

Images