A method and system for integrating public data resources based on artificial intelligence

Abstract

The present invention discloses a method and system for integrating public data resources based on artificial intelligence, and relates to the technical field of data resource integration. The method for integrating public data resources based on artificial intelligence obtains public resource data, and the public resource data includes text data and picture data; screens the public resource data to obtain text data to be classified and picture data to be classified; integrates and classifies the public resource data based on an integrated classification model stored in a database and then stores the integrated classification model, and the integrated classification model includes a text data integrated classification model and a picture data integrated classification model, which solves the problem that the amount of public data is huge, the data is scattered and difficult to manage.

Description

Technical Field

[0001] The present invention relates to the technical field of data resource integration, and in particular to a method and system for integrating public data resources based on artificial intelligence. Background Art

[0002] Public data resource integration refers to the process of collecting, screening, classifying and storing scattered data from different channels and sources in order to unify management and efficiently use these data. These data usually come from governments, public institutions, social media, open databases, etc., and may contain text, pictures, audio, video and other formats.

[0003] However, the amount of public data is huge and the data is scattered. Existing integration technology cannot effectively manage such huge and scattered data resources. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method and system for integrating public data resources based on artificial intelligence, which solves the problems of huge amount of public data, scattered data and difficult management.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a public data resource integration method based on artificial intelligence, comprising the following steps: obtaining public resource data, wherein the public resource data includes text data and image data; screening the public resource data to obtain text data to be classified and image data to be classified; integrating and classifying the public resource data based on an integrated classification model stored in a database and then storing the integrated classification model, wherein the integrated classification model includes a text data integrated classification model and an image data integrated classification model.

[0006] Furthermore, obtaining the text data to be classified and the image data to be classified includes the following steps: determining whether the obtained public resource data is to be retained, and deleting it if not; screening the retained public resource data to obtain the text data to be classified and the image data to be classified.

[0007] Furthermore, determining whether text data in public resource data is to be retained comprises the following steps: obtaining the character length of the text data; determining whether the character length of the text data is greater than a character length threshold stored in a database: if the character length of the text data is not greater than the character length threshold stored in the database, the text data is not retained; if the character length of the text data is greater than the character length threshold stored in the database, obtaining feature data of the text data, the feature data comprising a text error rate and a text sensitive word index; and determining whether the text data is to be retained based on the feature data.

[0008] Furthermore, the method of judging whether to retain text data based on feature data includes the following steps: obtaining parameter feature data stored in a database, the parameter feature data including a parameter text error rate and a parameter text sensitive word index; comprehensively processing the feature data and the parameter feature data to obtain a text conformity judgment coefficient; judging whether the text conformity judgment coefficient is greater than a text conformity judgment threshold stored in the database: if the text conformity judgment coefficient is greater than the text conformity judgment threshold stored in the database, the text data is not retained; if the text conformity judgment coefficient is not greater than the text conformity judgment threshold stored in the database, the text data is retained.

[0009] Furthermore, obtaining the text data to be classified includes the following steps: performing similarity screening on the retained text data, dividing the retained text data into similar text data sets and dissimilar text data sets; performing secondary screening on the similar text data sets, retaining the best text data in each similar text data set; and combining each best text data set and dissimilar text data set to form the text data to be classified.

[0010] Furthermore, retaining the optimal text data in each similar text data set includes the following steps: obtaining text evaluation parameters, the text evaluation parameters including text completeness, text content repetition rate and text information entropy; obtaining text parameter evaluation parameters stored in a database, the text parameter evaluation parameters including text parameter completeness, text parameter content repetition rate and text parameter information entropy; obtaining the optimal evaluation index of each text data in the similar text data set based on the text evaluation parameters and the text parameter evaluation parameters; outputting the text data corresponding to the smallest optimal evaluation index as the optimal text data of the similar text data set.

[0011] Furthermore, obtaining the image data to be classified includes the following steps: judging whether the image quality is qualified, and deleting it if it is unqualified; screening the image data with qualified image quality to obtain the image data to be classified.

[0012] Furthermore, the determination of whether the image quality is qualified includes the following steps: obtaining image quality data, the image quality data including clarity, saturation and distortion rate; obtaining image quality parameter data stored in a database, the image quality parameter data including parameter clarity, parameter saturation and parameter distortion rate; fusing the image quality data with the image quality parameter data to obtain an image quality assessment index; if the image quality assessment index is greater than the image quality assessment threshold stored in the database, the image quality is unqualified; if the image quality assessment index is not greater than the image quality assessment threshold stored in the database, the image quality is qualified.

[0013] Furthermore, the screening of the image data with qualified image quality to obtain the image data to be classified includes the following steps: performing similarity detection on the image data with qualified image quality to determine whether there are identical images; if there are no identical images, outputting all the image data with qualified image quality as the image data to be classified; if there are identical images, retaining the image with the smallest image quality assessment threshold among the identical images, deleting the remaining identical images, and outputting the retained images as the image data to be classified.

[0014] A public data resource integration system based on artificial intelligence includes a resource acquisition module, a screening module and an integration classification module, wherein: the resource acquisition module is used to acquire public resource data, and the public resource data includes text data and image data; the screening module is used to screen the public resource data to acquire text data to be classified and image data to be classified; the integration classification module is used to integrate and classify the public resource data based on an integration classification model stored in a database and then store it, and the integration classification model includes a text data integration classification model and an image data integration classification model.

[0015] The present invention has the following beneficial effects:

[0016] (1) This public data resource integration method based on artificial intelligence is not only time-consuming and laborious to manually screen and classify massive amounts of text and image data, but is also prone to human errors. However, through AI automated processing, data processing efficiency is greatly improved, ensuring that high-quality data is screened, classified and stored, which can effectively solve the problem of huge amounts of public data, scattered data and difficulty in management.

[0017] (2) This AI-based public data resource integration method greatly reduces manual intervention, saves time and costs, and significantly improves the accuracy of data classification and reduces the error rate through intelligent classification models. It can ensure data compliance, automatically filter sensitive information, and avoid legal risks. Ultimately, users can quickly and accurately access the information they need, improving the efficiency of decision-making and analysis.

[0018] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the public data resource integration method based on artificial intelligence of the present invention.

[0020] Figure 2 This is a flowchart of the public data resource integration system based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0021] The embodiment of the present application uses a method and system for integrating public data resources based on artificial intelligence. In the face of massive amounts of text and image data, manual screening and classification is not only time-consuming and laborious, but also prone to human errors. Through AI automated processing, data processing efficiency is greatly improved, ensuring that high-quality data is screened, classified and stored, which can effectively solve the problem of huge amounts of public data, scattered data and difficult management.

[0022] See also Figure 1 , an embodiment of the present invention provides a technical solution: a method for integrating public data resources based on artificial intelligence, comprising the following steps: obtaining public resource data, the public resource data including text data and image data; screening the public resource data to obtain text data to be classified and image data to be classified; integrating and classifying the public resource data based on the integrated classification model stored in the database and then storing it, the integrated classification model including a text data integrated classification model and an image data integrated classification model. The text data integrated classification model can be BERT (Bidirectional Encoder Representations from Transformers), which is a pre-trained language model based on the Transformer architecture and is widely used in tasks such as text classification, named entity recognition, and topic modeling. The text data integrated classification model can also be FastText, a text classification and word embedding model developed by Facebook AI Research, which can quickly classify large-scale text data, is lightweight and fast, and is suitable for large-scale text classification tasks. The image data integrated classification model can be ResNet (Residual Networks), which is a deep convolutional neural network (CNN) model suitable for image classification and object recognition tasks. By introducing the "residual" structure, the gradient vanishing problem in the deep network is solved, and it can handle very deep networks. The image data integration classification model can also be VGG (Visual Geometry Group Network). VGG is a classic convolutional neural network that is widely used in image classification tasks. It has a simple structure and stable effects, and is especially suitable for small and medium-sized image classification tasks.

[0023] First, collect public resource data from various channels, including text and pictures. This data can come from government websites, public databases, social media, etc. After collection, the system screens the data and identifies the text data and picture data to be classified. This step is to ensure the accuracy and effectiveness of subsequent classification. For the text data to be classified, use the pre-trained text data integration classification model to classify it through natural language processing technology, such as topic identification and sentiment analysis. Similarly, the picture data to be classified will also be processed through the picture data integration classification model, using computer vision technology to identify image content, such as object detection, image segmentation, etc. After classification processing, the results of the integrated classification will be stored to facilitate subsequent data retrieval and analysis.

[0024] It can automatically process a large amount of public resource data, saving labor costs and time. By integrating data through classification models, public resources can be better organized and managed, making it easier for users to quickly obtain the information they need. Classification based on advanced machine learning and deep learning models can improve classification accuracy and reduce human errors. This integration method can be continuously optimized and expanded as the amount of data increases and technology develops to meet the needs of different fields.

[0025] Specifically, obtaining the text data to be classified and the image data to be classified includes the following steps: determining whether the obtained public resource data is to be retained, and deleting it if not; screening the retained public resource data to obtain the text data to be classified and the image data to be classified.

[0026] Determining whether text data in public resource data is to be retained includes the following steps: obtaining the character length of the text data; determining whether the character length of the text data is greater than a character length threshold stored in a database: if the character length of the text data is not greater than the character length threshold stored in the database, the text data is not retained; if the character length of the text data is greater than the character length threshold stored in the database, obtaining feature data of the text data, the feature data including a text error rate and a text sensitive word index; and determining whether the text data is to be retained based on the feature data.

[0027] Determine whether the acquired public resource data should be retained. For data that does not meet the retention conditions, the system will actively delete it to ensure the quality of subsequent data processing. For the retained data, further screening will be carried out to obtain text data to be classified and image data to be classified. This process ensures the standardization and organization of the data.

[0028] Compare the character length of the text data with the length threshold set in the database: If the character length is less than or equal to the threshold, the text data is not retained and the system deletes it. If the length is greater than the threshold, the system further processes the text. Extract feature data from the retained text data, including: Text error rate: refers to the proportion of spelling errors or grammatical errors in the text. Text sensitive word index: measures the degree to which the text contains sensitive words.

[0029] After feature data analysis, it is further determined whether the text data meets the retention criteria. If the feature data indicators meet the requirements, the text data is considered to be retained; otherwise, it will be deleted.

[0030] Through effective screening and judgment mechanisms, we can ensure that only high-quality and high-value data is retained, reduce noise, and improve the effectiveness of subsequent processing. Delete texts with too short character length or a large number of errors and sensitive content to reduce the waste of system resources. By detecting sensitive words in the text, we can ensure that the retained data complies with relevant laws and regulations and reduce potential legal risks. The screened data to be classified is easier to carry out subsequent integration and classification, thereby improving the training and application effects of the classification model.

[0031] The method of judging whether to retain text data based on feature data includes the following steps: obtaining parameter feature data stored in a database, wherein the parameter feature data includes a parameter text error rate and a parameter text sensitive word index; comprehensively processing the feature data and the parameter feature data to obtain a text conformity judgment coefficient; judging whether the text conformity judgment coefficient is greater than a text conformity judgment threshold stored in the database: if the text conformity judgment coefficient is greater than the text conformity judgment threshold stored in the database, the text data is not retained; if the text conformity judgment coefficient is not greater than the text conformity judgment threshold stored in the database, the text data is retained.

[0032] Extract the reference feature data from the database, mainly including: Reference text error rate: identifies the acceptable text error rate threshold. Reference text sensitive word index: identifies the acceptable occurrence of sensitive words. The feature data of the text to be judged (text error rate and sensitive word index) and the reference feature data are processed comprehensively to calculate the text conformity judgment coefficient. This coefficient reflects the degree of conformity of the text to be judged with the standard.

[0033] The calculated text conformity judgment coefficient is compared with the text conformity judgment threshold stored in the database: if the text conformity judgment coefficient is greater than the threshold, the text data is not retained. If the text conformity judgment coefficient is not greater than the threshold, the text data is retained.

[0034] By setting clear judgment criteria (thresholds), the screening process of text data becomes more scientific and quantitative, avoiding the uncertainty caused by subjective judgment. The parameterized feature data and thresholds can be adjusted according to different needs to adapt to different scenarios and business requirements, providing greater flexibility. By comprehensively analyzing the error rate and sensitive words of the text, the quality of the retained text content can be effectively ensured, risks can be reduced, and usability can be improved. Using automated conformance judgment, large amounts of text data can be processed quickly, significantly improving the efficiency of data screening and retention.

[0035] The method for obtaining the text conformity judgment coefficient is as follows:

[0036] ;

[0037] In the formula, is the text conformity judgment coefficient, is the text error rate, is the text sensitive word index, Parameter text error rate, To determine the sensitive word index of the text, is a natural constant.

[0038] In this implementation scheme, the text error rate is usually calculated by the following steps: cleaning the original text, including removing extra spaces, normalizing punctuation, and converting to a unified character encoding (such as UTF-8). Using a spelling check algorithm (such as Levenshtein distance, N-gram model, etc.), compare each word in the text with the vocabulary in the standard dictionary to identify spelling errors. Use natural language processing (NLP) tools (such as dependency parsing, syntax trees, etc.) to identify grammatical errors in the text. Divide the number of errors detected by the total number of words to calculate the text error rate.

[0039] The calculation of the sensitive word index of a text usually includes the following steps: Create a word library containing sensitive words, which is usually defined by laws, regulations or industry specifications and may be updated regularly. Preprocess the text to be detected, such as removing punctuation marks, standardizing text formats, etc., to ensure more accurate sensitive word matching. Use string matching algorithms (such as regular expressions, Trie trees, KMP algorithms, etc.) to detect the frequency of occurrence of sensitive words in the text. The sensitive word index is usually obtained by calculating the ratio between the number of occurrences of sensitive words in the text and the total number of words in the text.

[0040] The obtained text error rate and sensitive word index will each be used as part of the feature data, and then used to comprehensively judge the retention of the text content.

[0041] Specifically, obtaining text data to be classified includes the following steps: performing similarity screening on the retained text data, dividing the retained text data into similar text data sets and dissimilar text data sets; performing secondary screening on the similar text data sets, retaining the best text data in each similar text data set; and combining each best text data set and dissimilar text data set to form the text data to be classified.

[0042] Specifically, retaining the optimal text data in each similar text data set includes the following steps: obtaining text evaluation parameters, the text evaluation parameters including text completeness, text content repetition rate and text information entropy; obtaining text parameter evaluation parameters stored in a database, the text parameter evaluation parameters including text parameter completeness, text parameter content repetition rate and text parameter information entropy; obtaining the optimal evaluation index of each text data in the similar text data set based on the text evaluation parameters and the text parameter evaluation parameters; outputting the text data corresponding to the smallest optimal evaluation index as the optimal text data of the similar text data set.

[0043] First, a dataset is obtained from the retained text data. These texts are considered to meet the retention criteria in the initial screening. The retained texts are analyzed for similarity, usually using techniques for calculating similarity (such as cosine similarity, Jaccard similarity, Manhattan distance, etc.). Based on the similarity results, the retained text data is divided into two subsets: Similar text dataset: a group of texts that are similar in content or topic. Dissimilar text dataset: a group of texts with large differences in content or topic. For each similar text dataset, further screening is performed to retain the "best text data" in each dataset. This step usually relies on some kind of evaluation criteria (such as text quality, information completeness, etc.) to select texts with representative or best information. The best text data in all similar text datasets are merged with the dissimilar text dataset to form the final text dataset to be classified. This step ensures the diversity of the dataset while retaining high-quality representative text.

[0044] After similarity screening and secondary screening, the text dataset to be classified contains high-quality and representative texts, improving the overall data quality. By merging similar texts and removing duplicates from similar data, the redundancy of the dataset is reduced and processing efficiency is improved. By focusing on the optimal text and avoiding duplication and redundancy of information, it helps to extract more valuable content during the classification process.

[0045] The optimal evaluation index is obtained as follows:

[0046] ;

[0047] In the formula, is the optimal evaluation index, For text completeness, is the text content repetition rate, is the text information entropy, Determine the completeness of the text. Set the content repetition rate for the text parameter. Determine the information entropy for the text.

[0048] In this embodiment, the text completeness usually evaluates the completeness and coherence of the information in the text, and the acquisition method includes: determining the standard of text completeness, such as whether the content contains all necessary information, whether it is logically coherent, etc. Clean the text, remove unnecessary characters, standardize the format, and prepare for analysis. Use NLP tools and algorithms to detect the structure and logical coherence of the text. For example, the completeness of the information in the text can be evaluated using techniques such as text summarization and key sentence extraction. Compare the missing information in the text with the content of the ideal text to calculate the completeness score.

[0049] The repetition rate of text content refers to the proportion of identical or similar content in a text. It can be obtained as follows: Segment and standardize the text for repeatability analysis. Use similarity calculation algorithms (such as cosine similarity, Jaccard index, etc.) to evaluate the similarity of the text. You can divide the text into sentences or paragraphs for comparison and calculate the frequency of repeated content. Count the number of repeated paragraphs, sentences, or phrases in the text. Compare the number of repeated content with the total content to get the repetition rate.

[0050] Text information entropy is used to measure the complexity and diversity of information in a text. The method of obtaining it is as follows: segment the text, count the frequency of each word, calculate the probability of each word, and use the Shannon entropy formula to calculate the information entropy.

[0051] These three parameters—text completeness, text content repetition rate, and text information entropy—are obtained by cleaning, preprocessing, and analyzing text content. They each reflect different aspects of the text and provide a comprehensive assessment of text quality and information richness. High-quality text should have high completeness, low repetition rate, and moderate information entropy, which is of great significance for text analysis, machine learning, and natural language processing tasks.

[0052] Specifically, obtaining the image data to be classified includes the following steps: determining whether the image quality is qualified, and deleting if it is unqualified; and screening the image data with qualified image quality to obtain the image data to be classified.

[0053] The method of judging whether the image quality is qualified includes the following steps: obtaining image quality data, wherein the image quality data includes clarity, saturation and distortion rate; obtaining image quality parameter data stored in a database, wherein the image quality parameter data includes parameterized clarity, parameterized saturation and parameterized distortion rate; fusing the image quality data with the image quality parameter data to obtain an image quality assessment index; if the image quality assessment index is greater than an image quality assessment threshold stored in the database, the image quality is unqualified; if the image quality assessment index is not greater than the image quality assessment threshold stored in the database, the image quality is qualified.

[0054] The image quality features, including clarity, saturation and distortion rate, are extracted through image processing technology. The quality standards or thresholds stored in the database are obtained as references, including the parameterized clarity, parameterized saturation and parameterized distortion rate. The quality data obtained from the image is integrated with the reference parameter data to form a comprehensive image quality evaluation index. This index reflects the performance of the image in multiple dimensions such as clarity, saturation and distortion rate, providing a basis for subsequent judgment.

[0055] The calculated image quality assessment index is compared with the quality assessment threshold stored in the database: if the image quality assessment index is greater than the assessment threshold, the image quality is considered unqualified and needs to be deleted. If the image quality assessment index is not greater than the assessment threshold, the image quality is considered qualified and is retained for classification. After the above steps, only the image data with qualified quality is retained as the object of subsequent classification processing.

[0056] By eliminating images with substandard quality, noise and errors can be reduced, thereby improving the effectiveness and accuracy of subsequent classification algorithms. Low-quality images can be screened out during the data preprocessing stage to reduce the amount of data that needs to be processed in subsequent processing, thereby reducing computing costs and time. Defining clear quality standards and evaluation indicators will help form a standardized image quality evaluation process and ensure the consistency of results. Using images of qualified quality for classification can provide better visual effects and user experience, and increase user satisfaction.

[0057] The image quality assessment index is obtained as follows:

[0058] ;

[0059] In the formula, is the image quality assessment index, For clarity, is saturation, is the distortion rate, To determine the clarity, To determine the saturation, To determine the distortion rate.

[0060] Clarity usually refers to the clarity of details in an image. Methods for obtaining clarity include: using the Laplace operator to detect edges on an image. The more obvious the edges of an image are, the higher the clarity. The response function of the Laplace equation can be used to calculate the clarity index of an image. Saturation refers to the intensity of the color in an image. Common calculation methods include: converting an image from an RGB color space to an HSV (hue, saturation, value) color space. The value of the saturation channel directly reflects the richness of the color. Calculate the average of the saturation values ​​of all pixels in the HSV space to obtain the saturation of the entire image. Distortion rate reflects the degree of degradation of image quality. Common calculation methods include: calculating the mean square error between the original image and the distorted image. The smaller the value, the smaller the distortion.

[0061] Through image processing and analysis technology, the clarity, saturation and distortion rate can be obtained more accurately. Quantitative evaluation of these indicators helps to judge the quality of the picture and further realize the screening and classification of pictures.

[0062] The method of screening the image data with qualified image quality to obtain the image data to be classified includes the following steps: performing similarity detection on the image data with qualified image quality to determine whether there are identical images; if there are no identical images, outputting all the image data with qualified image quality as the image data to be classified; if there are identical images, retaining the image with the smallest image quality assessment threshold among the identical images, deleting the remaining identical images, and outputting the retained images as the image data to be classified.

[0063] In this implementation scheme, similarity detection is performed on all qualified pictures, and feature extraction technology (such as SIFT, SURF, ORB, etc.) or hash algorithm (such as perceptual hashing) can be used to determine whether the pictures are the same or similar. According to the results of the similarity detection, it is determined whether there are identical or similar pictures among all qualified pictures. If no duplicate pictures are found after detection, all qualified pictures will be directly output as picture data to be classified. If duplicate pictures are detected, the same pictures will be found and the quality of these pictures will be evaluated. The picture with the smallest evaluation quality threshold is selected to be retained, and other identical pictures will be deleted. Finally, the retained pictures are output as picture data to be classified to ensure the uniqueness and quality of the data set.

[0064] A public data resource integration system based on artificial intelligence, such as Figure 2As shown, it includes a resource acquisition module, a screening module and an integrated classification module, wherein: the resource acquisition module is used to acquire public resource data, and the public resource data includes text data and image data; the screening module is used to screen the public resource data, and acquire the text data to be classified and the image data to be classified; the integrated classification module is used to integrate and classify the public resource data based on the integrated classification model stored in the database and then store it, and the integrated classification model includes a text data integrated classification model and a picture data integrated classification model.

[0065] An electronic device comprises: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the public data resource integration method based on artificial intelligence as described above.

[0066] A computer-readable storage medium is used to store a program, which, when executed by a processor, implements the public data resource integration method based on artificial intelligence as described above.

[0067] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0072] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for integrating public data resources based on artificial intelligence, characterized in that: The following steps are involved: Acquiring public resource data, wherein the public resource data includes text data and image data; Screening public resource data to obtain text data and image data to be classified; The public resource data is integrated and classified based on the integrated classification model stored in the database and then stored, wherein the integrated classification model includes a text data integrated classification model and a picture data integrated classification model; The process of obtaining the text data to be classified and the image data to be classified includes the following steps: Determine whether the acquired public resource data should be retained, and delete it if not; Screening the retained public resource data to obtain text data to be classified and image data to be classified; Wherein, judging whether the text data in the public resource data is retained includes the following steps: Get the character length of text data; Determine whether the character length of the text data is greater than the character length threshold stored in the database: If the character length of the text data is not greater than the character length threshold stored in the database, the text data is not retained; If the character length of the text data is greater than the character length threshold stored in the database, then acquiring feature data of the text data, the feature data including a text error rate and a text sensitive word index; Determine whether to retain text data based on feature data; Wherein, judging whether the text data is to be retained based on the feature data comprises the following steps: Acquire the reference feature data stored in the database, wherein the reference feature data includes a reference text error rate and a reference text sensitive word index; Comprehensively process the feature data and the parameter feature data to obtain the text conformity judgment coefficient; Determine whether the text conformity judgment coefficient is greater than the text conformity judgment threshold stored in the database: If the text conformity judgment coefficient is greater than the text conformity judgment threshold stored in the database, the text data will not be retained; If the text conformity judgment coefficient is not greater than the text conformity judgment threshold stored in the database, the text data is retained; The method for obtaining the text conformity judgment coefficient is as follows: ； In the formula, is the text conformity judgment coefficient, is the text error rate, is the text sensitive word index, Parameter text error rate, To determine the sensitive word index of the text, is a natural constant; Wherein, obtaining the text data to be classified includes the following steps: Perform similarity screening on the retained text data, and divide the retained text data into similar text data sets and dissimilar text data sets; Perform secondary screening on similar text data sets and retain the best text data in each similar text data set; Combining each optimal text data set with a dissimilar text data set to form text data to be classified; The optimal text data in each similar text data set is retained, including the following steps: Acquiring text evaluation parameters, wherein the text evaluation parameters include text completeness, text content repetition rate, and text information entropy; Obtaining text parameter evaluation parameters stored in a database, wherein the text parameter evaluation parameters include text parameter completeness, text parameter content repetition rate, and text parameter information entropy; Obtaining an optimal evaluation index for each text data in a similar text data set based on a text evaluation parameter and a text parameterized evaluation parameter; Output the text data corresponding to the smallest optimal evaluation index as the optimal text data of the similar text data set; Among them, the optimal evaluation index is obtained as follows: ； In the formula, is the optimal evaluation index, For text completeness, is the text content repetition rate, is the text information entropy, Determine the completeness of the text. Set the content repetition rate for the text parameter. Determine the information entropy for the text.

2. The method for integrating public data resources based on artificial intelligence according to claim 1, characterized in that: Obtaining the image data to be classified includes the following steps: Determine whether the image quality is qualified and delete it if it is not qualified; Screen the image data with qualified image quality to obtain the image data to be classified.

3. The method for integrating public data resources based on artificial intelligence according to claim 2, characterized in that: The step of judging whether the image quality is qualified comprises the following steps: Acquiring picture quality data, wherein the picture quality data includes clarity, saturation and distortion rate; Acquire the picture quality parameter data stored in the database, wherein the picture quality parameter data includes parameter definition, parameter saturation and parameter distortion rate; The image quality data is integrated with the image quality parameter data to obtain an image quality assessment index; If the image quality assessment index is greater than the image quality assessment threshold stored in the database, the image quality is unqualified; If the image quality assessment index is not greater than the image quality assessment threshold stored in the database, the image quality is qualified.

4. The method for integrating public data resources based on artificial intelligence according to claim 3, characterized in that: The step of screening the image data with qualified image quality to obtain the image data to be classified includes the following steps: Perform similarity detection on image data with qualified image quality to determine whether there are identical images; If there are no identical pictures, all the picture data with qualified picture quality are output as the picture data to be classified; If there are identical pictures, the picture with the smallest picture quality assessment threshold is retained, and after deleting the remaining identical pictures, the retained picture is output as the picture data to be classified.

5. A public data resource integration system based on artificial intelligence, applying the public data resource integration method based on artificial intelligence according to any one of claims 1 to 4, characterized in that: It includes resource acquisition module, screening module and integrated classification module, among which: The resource acquisition module is used to acquire public resource data, and the public resource data includes text data and picture data; The screening module is used to screen the public resource data to obtain the text data to be classified and the image data to be classified; The integration classification module is used to integrate and classify the public resource data based on the integration classification model stored in the database and then store it. The integration classification model includes a text data integration classification model and a picture data integration classification model.

Images