Data acquisition method and system based on neural network, computing equipment and storage medium

Through the data acquisition method based on neural network model, the semantic correlation index is calculated and the model is trained, which solves the problem of insufficient accuracy in semantic complex texts, and realizes efficient and accurate data acquisition and automated processing.

CN118940764AInactive Publication Date: 2024-11-12BEIJING ALL VIEW CLOUD DATA TECH CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202410936249.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When traditional data collection methods face text content with complex semantics and high ambiguousness, their accuracy and correlation are insufficient, resulting in a decrease in the purity and value of the data.

Method used

Using a data acquisition method based on neural network model, we determine the target topic, calculate the semantic correlation index of text fragments, filter out text fragments that meet preset requirements, build a data set and train a neural network model to achieve accurate collection of target data.

Benefits of technology

It significantly improves the accuracy and purity of data acquisition, reduces the cost and time of subsequent data processing and analysis, realizes automated and intelligent data acquisition, and builds high-quality data sets to improve the performance of machine learning models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940764A_ABST
    Figure CN118940764A_ABST
Patent Text Reader

Abstract

The invention provides a data acquisition method and system based on a neural network, computing equipment and a storage medium. The method comprises the steps of determining a target theme, and collecting original data based on the target theme; the original data is converted into text data, the text data comprises a plurality of text segments, the semantic correlation index of each text segment is calculated, and the semantic correlation index is used for indicating the correlation degree of key information in the text segments and the target theme; screening out text fragments meeting a preset semantic correlation index requirement to construct a data set, and training a neural network model through the data set; and collecting target data based on the target theme through the trained neural network model. According to the technical scheme provided by the invention, the accuracy of data acquisition can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data acquisition technology, and in particular to a data acquisition method, system, computing device and storage medium based on a neural network. Background Art

[0002] With the rapid development of the Internet and the advent of the big data era, data has become a key factor in promoting scientific and technological progress and social development. In this context, data collection technology is particularly important. It is not only the basis for big data analysis and artificial intelligence applications, but also the key to achieving intelligent decision-making, personalized services and market insights. However, facing the vast ocean of network information, how to efficiently and accurately filter out information that is highly relevant to specific topics or needs from massive data has become a major challenge in the field of data science.

[0003] Traditional data collection methods often rely on keyword matching or simple rule engines. Although they can quickly capture large amounts of data, their accuracy and relevance are often greatly reduced when faced with text content with complex semantics and high ambiguity. For example, data captured based on the keyword "apple" alone may include descriptions of fruits, news about the technology company Apple, or even information related to gardening or works of art, which undoubtedly greatly reduces the purity and value of the data. Summary of the invention

[0004] The embodiments of the present application provide a data collection method, system, computing device and storage medium based on a neural network, so as to solve the problem of poor accuracy of data collection in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a data collection method based on a neural network model, comprising:

[0006] Determine a target topic and collect original data based on the target topic;

[0007] Converting the original data into text data, the text data including a plurality of text segments, and calculating a semantic relevance index of each text segment, the semantic relevance index being used to indicate the relevance of key information in the text segment to the target topic;

[0008] Screening out text segments that meet the requirements of a preset semantic relevance index to construct a data set, and training a neural network model through the data set;

[0009] Target data is collected based on the target topic through the trained neural network model.

[0010] Optionally, determining a target topic and collecting raw data based on the target topic includes:

[0011] Extract relevant key information from the target topic, wherein the key information at least includes keywords or phrases;

[0012] Identify multiple data sources related to the target topic, and select an appropriate data collection tool according to the type of each data source;

[0013] The data collection tool collects raw data from the multiple data sources according to the key information.

[0014] Optionally, the converting the original data into text data, the text data comprising a plurality of text segments, including:

[0015] Removing non-text elements from the original data to obtain adjusted original data; the non-text elements at least include: HTML tags and JavaScript codes;

[0016] Using regular expressions, text data is extracted from the adjusted original data, and the text data is segmented into a plurality of text segments at a sentence level; or, the text data is segmented into a plurality of text segments at a paragraph level according to line breaks or paragraph marks.

[0017] Optionally, calculating the semantic relevance index of each text segment includes:

[0018] By formula: Calculate the semantic relevance index of each text segment;

[0019] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, v j It represents all the words in the text segment T, n represents the number of key information, and m represents the number of all words.

[0020] Optionally, screening out text segments that meet preset semantic relevance index requirements to construct a data set includes:

[0021] Setting a threshold value of a preset semantic relevance index, and taking a text segment whose semantic relevance index is higher than the threshold value of the preset semantic relevance index as a text segment that meets the requirement of the preset semantic relevance index;

[0022] The text segments that meet the preset semantic relevance index requirements are divided into a training set and a test set, and a neural network model is trained based on the training set, and the hyperparameters of the neural network model are adjusted based on the test set until the neural network model completes the training.

[0023] Optionally, it also includes:

[0024] Converting the original data into object data, wherein the object data includes at least one of text data, audio data or video data or a combination of any several of the above;

[0025] Using multimodal fusion technology, different types of object data are integrated into a data set, and the neural network model is trained with the data set;

[0026] Target data is collected based on the target topic through the trained neural network model.

[0027] Optionally, calculating the semantic relevance index of each text segment includes:

[0028] By formula:

[0029]

[0030] Calculate the semantic relevance index of each text segment;

[0031] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, CtxSim(w i , T) is represented by the key information w i The semantic similarity with the text segment T, wherein the semantic similarity refers to the key information w i The average cosine similarity between the context embedding vector of the text segment T and the context embedding vectors of other words in the text segment T, v j represents all the words in the text segment T, n represents the number of key information, m represents the number of all words, S represents the target topic, and SemSim(T, S) represents the cosine similarity between the encoding vector of the text segment T and the encoding vector of the target topic S.

[0032] In a second aspect, an embodiment of the present application provides a data acquisition system based on a neural network model, comprising:

[0033] A collection module, used to determine a target topic and collect raw data based on the target topic;

[0034] A conversion module, used for converting the original data into text data, wherein the text data includes a plurality of text segments;

[0035] A calculation module, used to calculate a semantic relevance index of each text segment, wherein the semantic relevance index is used to indicate the relevance degree between the key information in the text segment and the target topic;

[0036] A training module, used to screen out text segments that meet the requirements of a preset semantic relevance index to construct a data set, and train a neural network model using the data set;

[0037] The acquisition module is also used to acquire target data based on the target subject through a trained neural network model.

[0038] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the data acquisition method based on the neural network model as described in the first aspect above.

[0039] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, the data acquisition method based on the neural network model as described in the first aspect above is implemented.

[0040] In an embodiment of the present application, a target topic is determined, and raw data is collected based on the target topic; the raw data is converted into text data, the text data includes multiple text fragments, and a semantic relevance index of each text fragment is calculated, the semantic relevance index is used to indicate the degree of relevance between key information in the text fragment and the target topic; text fragments that meet the preset semantic relevance index requirements are screened out to construct a data set, and a neural network model is trained through the data set; and target data is collected based on the target topic through the trained neural network model.

[0041] The data collection method based on the neural network model proposed in this application shows significant superiority and innovation compared with traditional data collection technology, which is specifically reflected in the following aspects:

[0042] Accurate semantic relevance assessment: By calculating the semantic relevance index of each text segment, this solution can effectively distinguish high-quality data that is highly relevant to the target topic from irrelevant noise data, significantly improving the accuracy and purity of data collection and reducing the cost and time of subsequent data processing and analysis.

[0043] Deep semantic understanding capability: Using pre-trained neural network models such as BERT, RoBERTa, or XLNet, this solution can deeply understand the deep semantic structure of the text, capture the complex relationship between keywords and text context, and maintain high recognition accuracy even in the face of ambiguous or implicit text content.

[0044] Automated and intelligent data collection: Based on the trained neural network model, this solution realizes the automation and intelligence of data collection. It can automatically filter and capture relevant data according to the target topic, greatly improving the efficiency of data collection while reducing errors and deviations caused by human participation.

[0045] Construction of high-quality datasets: By screening out text segments that meet the preset semantic relevance index requirements, this solution can construct a high-quality dataset, providing a solid foundation for subsequent model training and data analysis, and helping to improve the performance and generalization ability of machine learning models.

[0046] In summary, the present application scheme can improve the accuracy of data collection by deeply integrating the neural network model with the data collection process.

[0047] These and other aspects of the present application will become more apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flowchart of a data acquisition method based on a neural network model provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a data acquisition device based on a neural network model provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0054] The inventors have found that traditional data collection methods often rely on keyword matching or simple rule engines. Although they can quickly capture large amounts of data, their accuracy and relevance are often greatly reduced when faced with semantically complex and highly ambiguous text content.

[0055] In view of this, the present application provides a data collection method based on a neural network model, the method comprising: determining a target topic and collecting original data based on the target topic; converting the original data into text data, the text data comprising multiple text segments, and calculating a semantic relevance index for each text segment, the semantic relevance index being used to indicate the degree of relevance between key information in the text segment and the target topic; screening out text segments that meet the requirements of a preset semantic relevance index to construct a data set, and training a neural network model through the data set; and collecting target data based on the target topic through the trained neural network model.

[0056] This method can effectively improve the accuracy of data collection by deeply integrating the neural network model with the data collection process.

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] Figure 1 A flowchart of a data acquisition method based on a neural network model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0059] 101. Determine a target topic, and collect original data based on the target topic;

[0060] In this step, the target theme indicates the direction and focus of data collection, which can be a specific event, field, product or problem, such as "New Energy Vehicle Market Trends", "Global Climate Change Research", etc. Raw data collection is to obtain unprocessed raw information from sources such as the Internet, databases, files, social media, etc. according to the target theme, including text, images, audio and other formats.

[0061] In this example, we assume that our target topic is "global climate change". We first define a series of keywords, such as "global warming", "carbon emissions", "greenhouse effect", etc., as well as the geographical area and time range of interest. Next, we use web crawler tools to crawl articles, papers and reports related to these keywords from news websites, academic journals, government reports and other channels. In addition, we also collect users' discussions and opinions on climate change through social media APIs.

[0062] Optionally, in the embodiment of the present application, step 101 may specifically include the following steps:

[0063] 1011. Extract relevant key information from the target topic, where the key information at least includes keywords or phrases;

[0064] In this step, key information refers to keywords or phrases closely related to the target topic, which are used to guide the direction of data collection. These keywords or phrases can capture the core content and key features of the topic and help locate valuable information in massive data.

[0065] For example, suppose our target topic is "Application of artificial intelligence in the medical industry". First, we extract a series of keywords and phrases from this target topic, such as "artificial intelligence", "medical diagnosis", "machine learning", "case analysis", "telemedicine", etc. These keywords and phrases will serve as guiding clues for subsequent data collection.

[0066] 1012. Identify multiple data sources related to the target topic, and select an appropriate data collection tool according to the type of each data source;

[0067] In this step, data sources refer to sources from which information related to the target topic can be obtained, including but not limited to academic journals, news websites, social media, professional forums, government reports, corporate announcements, etc.

[0068] Data collection tools: According to different types of data sources, choose appropriate tools and technologies for data capture, such as web crawlers, API interfaces, RSS subscriptions, database queries, etc.

[0069] For example, in the example of the theme "Application of Artificial Intelligence in the Medical Industry", we identified multiple relevant data sources, including the PubMed medical database, the ArXiv preprint server, the TechCrunch technology news website, and the Twitter social media platform. For academic resources such as PubMed and ArXiv, we use specialized academic database APIs to query data; for news websites such as TechCrunch, we wrote web crawler scripts to automatically crawl relevant articles based on keywords; on Twitter, we monitor tweet streams containing keywords through the official API. In this way, we can obtain comprehensive information about the application of artificial intelligence in the medical industry from different angles and sources.

[0070] 1013. Collect original data from the multiple data sources according to the key information using the data collection tool.

[0071] In this step, data collection refers to the use of appropriate tools and techniques to capture raw data related to the target topic based on the identified key information and data sources, laying the foundation for subsequent data analysis and processing.

[0072] For example, continuing with the example of "the application of artificial intelligence in the medical industry", we used the above selected data collection tools to capture a large number of articles, papers, news reports and social media discussions from data sources such as PubMed, ArXiv, TechCrunch and Twitter based on keywords such as "artificial intelligence" and "medical diagnosis". For example, we captured the abstracts of research papers published in the past year on "the application of machine learning in early diagnosis of cancer" from the PubMed database; we captured news reports about a startup using AI technology to improve the management of diabetic patients from the TechCrunch website; and we collected discussions and opinions on the prospects of AI in the medical field from doctors, patients and technology enthusiasts on Twitter. Through the collection of these raw data, we have obtained a diverse perspective and in-depth insights on the theme of "the application of artificial intelligence in the medical industry", providing rich materials for subsequent data analysis and topic exploration.

[0073] 102. Convert the original data into text data, wherein the text data includes a plurality of text segments, and calculate a semantic relevance index of each text segment, wherein the semantic relevance index is used to indicate the relevance degree between key information in the text segment and the target topic;

[0074] In this step, text data conversion refers to filtering out non-text content (such as HTML tags, images, audio, etc.) in the original data and retaining only text information to facilitate subsequent text analysis.

[0075] Text fragmentation refers to dividing a long text into shorter paragraphs or sentences so that their semantic relevance can be analyzed independently.

[0076] The semantic relevance index (SRI) refers to an indicator that quantifies the semantic relevance between a text segment and a target topic, and is usually calculated by combining the TF-IDF value of the key information and the contextual semantic information.

[0077] Optionally, in the embodiment of the present application, the step of “converting the original data into text data, wherein the text data includes a plurality of text segments” in step 102 may specifically include the following steps:

[0078] 1021. Clear non-text elements in the original data to obtain adjusted original data; the non-text elements at least include: HTML tags and JavaScript codes;

[0079] In this step, non-text elements refer to the raw data captured from sources such as web pages, PDF documents, and emails. In addition to text content, they may also contain non-text elements such as HTML tags, JavaScript codes, CSS styles, images, and tables. These elements are not directly helpful for text analysis, but will interfere with subsequent processing.

[0080] In this embodiment of the application, suppose we crawled an article about AI technology from the TechCrunch website. The original data contains non-text elements such as HTML tags, image links, and ad codes. First, we need to use Python's BeautifulSoup library to parse the HTML document and remove non-text elements using the following code:

[0081]

[0082] In this way, we get the adjusted raw data, leaving only the plain text content of the article: "As..."

[0083] 1022. Use regular expressions to extract text data from the adjusted original data, and segment the text data into multiple text segments at the sentence level; or segment the text data into multiple text segments at the paragraph level according to line breaks or paragraph marks.

[0084] In this step, text data segmentation: In order to facilitate subsequent semantic analysis and model training, long texts need to be segmented into smaller text segments, such as sentences or paragraphs. This helps the model learn local semantic structures and improve the accuracy of analysis.

[0085] In the present embodiment, based on the previous embodiment, we continue to process the obtained plain text data. First, use regular expressions to segment the text into sentence levels:

[0086] import re

[0087] sentences=re.split(r'(?<=[.!?])\s+',text)

[0088] Assume that the content of text is: "With the development of AI technology, the medical industry is undergoing a transformation. (First sentence) Many hospitals have begun to adopt AI-assisted diagnosis systems to improve the accuracy of disease detection. (Second sentence)", after using the above code, the sentences list will contain two string elements: "With the development of AI technology, the medical industry is undergoing a transformation." and "Many hospitals have begun to adopt AI-assisted diagnosis systems to improve the accuracy of disease detection."

[0089] If you need to split into paragraphs, you can simply split based on line breaks:

[0090] paragraphs = text.split('\n')

[0091] Assuming that text contains line breaks, the paragraphs list will contain each line as an element, forming a collection of paragraph-level text fragments.

[0092] Through the above steps, we successfully converted the raw data into structured text data, preparing the necessary materials for subsequent semantic relevance analysis and model training.

[0093] Furthermore, as a possible implementation scheme, the process of calculating the semantic relevance index of each text segment includes:

[0094] By formula: Calculate the semantic relevance index of each text segment;

[0095] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, v j It represents all the words in the text segment T, n represents the number of key information, and m represents the number of all words.

[0096] In this solution, assuming that our target topic is "new energy vehicle technology", we extracted two key information from the text segment: w1 = "battery technology", w2 = "electric motor". We calculated the TF-IDF value of these two key information in the text segment T:

[0097] TF-IDF("Battery Technology")=0.6;

[0098] TF-IDF("motor")=0.4;

[0099] At the same time, we counted the total number of words in the text segment T, m = 100. Next, we apply Formula 1 to calculate the semantic relevance index SRI(T) of the text segment T:

[0100]

[0101] Assuming that the average TF-IDF value of all words in the text segment T is 0.3, substituting it into the above formula yields:

[0102] SRI(T)=((0.6+0.4)) / (100×0.3)=1 / 30=0.0333.

[0103] To further improve the accuracy of the semantic relevance index of the text segment T, as another possible implementation scheme, the semantic relevance index of each text segment may also include the following scheme:

[0104] By formula:

[0105]

[0106] Calculate the semantic relevance index of each text segment;

[0107] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, CtxSim(w i , T) is represented by the key information w i The semantic similarity with the text segment T, wherein the semantic similarity refers to the key information w i The average cosine similarity between the context embedding vector of the text segment T and the context embedding vectors of other words in the text segment T, v j represents all the words in the text segment T, n represents the number of key information, m represents the number of all words, S represents the target topic, and SemSim(T, S) represents the cosine similarity between the encoding vector of the text segment T and the encoding vector of the target topic S.

[0108] In this solution, continuing with the above example, we introduce two additional variables: CtxSim(w i, T) represents the key information w i The semantic similarity with the text segment T, SemSim(T,S) represents the semantic similarity between the text segment T and the target topic S.

[0109] Suppose we use the pre-trained BERT model to calculate these two similarities:

[0110] CtxSim("battery technology", T) = 0.85 (indicating that "battery technology" is highly relevant to the context of text segment T); CtxSim("electric motor", T) = 0.75;

[0111] SemSim(T,S) = 0.9 (indicating that text segment T is highly relevant to the target topic “new energy vehicle technology”);

[0112] We set the balance factor α = 0.8 and substitute it into the above formula:

[0113]

[0114] Substituting the TF-IDF values ​​and similarities calculated previously, as well as m=100 and the average TF-IDF value of all words being 0.3:

[0115] SRI(T)=0.8·((0.6·0.85+0.4·0.75) / (100·0.3))+(1-0.8)·0.9=0.21;

[0116] Through these two embodiments, we can see that the formula of this scheme can more comprehensively evaluate the relevance of text fragments to the target topic by combining the contextual semantics of keywords and the overall semantic similarity of text fragments, thereby providing a more accurate semantic relevance index.

[0117] 103. Screening out text segments that meet the preset semantic relevance index requirements to construct a data set, and training a neural network model using the data set;

[0118] In this step, the screening condition refers to selecting text segments that are highly relevant to the target topic according to the SRI threshold (the SRI threshold can be determined according to step 102) to construct a training data set.

[0119] Neural network model training refers to using a constructed data set to train a neural network model so that it can automatically identify information related to the target topic.

[0120] In the example of this application, continuing to take "global climate change" as an example, we set the SRI threshold to 0.6, which means that only text fragments with an SRI greater than or equal to 0.6 will be selected into the training data set. After screening, we obtained about 10,000 high-quality text fragments. Next, we use these text fragments to train a neural network model based on the Transformer architecture. Through multiple rounds of iterations, we optimize the model parameters so that it can accurately determine whether new text data is related to the topic of "global climate change".

[0121] Optionally, in the embodiment of the present application, step 103 may specifically include the following steps:

[0122] 1031. Setting a threshold of a preset semantic relevance index, and taking a text segment whose semantic relevance index is higher than the threshold of the preset semantic relevance index as a text segment that meets the requirement of the preset semantic relevance index;

[0123] In this step, a semantic relevance index threshold is preset: this is a predetermined numerical standard used to screen text fragments. Only when the semantic relevance index of a text fragment is higher than this threshold, it is considered to be highly relevant to the target topic and is thus selected for constructing a training dataset.

[0124] In the embodiment of the present application, it is assumed that we have calculated the semantic relevance index of all text fragments in the previous stage. Now, we need to set a threshold, such as 0.7, as a screening criterion. Any text fragment with a semantic relevance index greater than or equal to 0.7 will be considered highly relevant to the target topic. Suppose we have the following five text fragments and their corresponding semantic relevance indexes as shown in Table 1 below:

[0125] Table 1

[0126] Text snippets Semantic Relevance Index Clip 1: Application of AI technology in medical diagnosis 0.8 Clip 2: Summary of last year’s financial report 0.2 Clip 3: How AI will change future education 0.75 Clip 4: Analysis of global climate change trends 0.6 Clip 5: The potential of AI in manufacturing 0.85

[0127] According to the threshold of 0.7, we only select fragment 1, fragment 3 and fragment 5 as text fragments that meet the preset semantic relevance index requirements.

[0128] 1032. Divide the text segments that meet the preset semantic relevance index requirements into a training set and a test set, train a neural network model based on the training set, and adjust the hyperparameters of the neural network model based on the test set until the neural network model completes training.

[0129] In this step, the training set and the test set are divided into two parts: one part is used to train the neural network model so that the model can learn how to identify text features related to the target topic; the other part is used to test the performance of the model and evaluate the performance of the model on unseen data to prevent overfitting.

[0130] Hyperparameter tuning: The training process of a neural network model involves the selection of multiple hyperparameters, such as learning rate, batch size, number of layers, number of nodes, etc. By evaluating the model performance on the test set, these hyperparameters can be adjusted to find the optimal combination to improve the generalization ability and prediction accuracy of the model.

[0131] In the previous example, we have selected three highly relevant text segments: segment 1, segment 3, and segment 5. Now, we decide to divide these three segments into training and test sets in a ratio of 7:3. Assume that segments 1 and 3 are assigned to the training set, and segment 5 is assigned to the test set.

[0132] Next, we use the training set (fragments 1 and 3) to train a neural network model based on the Transformer architecture. The initial learning rate is 0.001 and the batch size is 32. After several epochs of training, we use the test set (fragment 5) to evaluate the performance of the model. Assuming that the accuracy of the model on the test set is 85%, this may mean that there is still room for improvement in the model.

[0133] So, we try to adjust the hyperparameters, such as reducing the learning rate to 0.0005, and train the model again until the performance of the model on the test set reaches a satisfactory level, such as an accuracy rate of more than 90%. At this time, we can say that the neural network model has completed training and can be used for subsequent target data collection tasks.

[0134] 104. Collect target data based on the target subject through the trained neural network model.

[0135] In this step, the trained neural network model is deployed to the actual data collection process to automatically filter and capture data related to the target topic.

[0136] In the example of this application, in the scenario of the theme of "global climate change", we use the trained neural network model to analyze the real-time incoming news, social media updates and research reports. The model can quickly determine whether each piece of information is related to "global climate change" and only save the highly relevant information, which greatly improves the efficiency and accuracy of data collection. For example, the model successfully identified a research report on the reduction of Arctic sea ice area, while ignoring sports news that is not related to the topic.

[0137] Compared with the traditional scheme of collecting data through keywords, the technical solution provided by the embodiment of the present application adopts deep learning technology, especially pre-trained neural network models (such as BERT, RoBERTa or XLNet), which can deeply understand the deep semantic structure of the text, capture the subtle connection between keywords and text context, and significantly improve the accuracy and relevance of data collection. By building and training the neural network model, the automation and intelligence of data collection are realized, and relevant data can be automatically screened and captured according to the target topic, which greatly improves the efficiency of data collection and reduces the errors and deviations of manual participation.

[0138] Optionally, in an embodiment of the present application, as a possible implementation scheme, the method also includes: converting the original data into object data, wherein the object data includes at least one of text data, audio data or video data, or a combination of any several of them; utilizing multimodal fusion technology to integrate different types of object data into a data set, and using the data set to train a neural network model; and collecting target data based on the target subject through the trained neural network model.

[0139] In this step, object data refers to the original information obtained through data collection, which can be in various forms, including but not limited to text data, audio data, and video data. Converting raw data into object data is an important step in data preprocessing, preparing for subsequent data analysis and model training.

[0140] Multimodal fusion refers to the integration of different types of object data (such as text, audio, and video) into a unified data set for neural network model learning and training. Multimodal fusion technology can make full use of the advantages of various data types and improve the generalization ability and prediction accuracy of the model.

[0141] Target data collection: Once the neural network model is trained, we can use it to automatically filter and capture data that is highly relevant to the target topic. This step is the closed loop of the entire data collection process. Through the predictive ability of the model, the efficiency and accuracy of data collection can be greatly improved.

[0142] In this example, assume that our target topic is "Public perception of global climate change". We collected three types of data from different data sources:

[0143] Text data: A large number of reports, comments, and research reports on climate change were crawled from news websites, blogs, and academic papers.

[0144] Audio data: Audio files of discussions and speeches on climate change were downloaded from podcasts, radio shows, and lectures.

[0145] Video data: Documentaries, popular science videos, and interview videos on climate change were collected from YouTube and other video sharing platforms.

[0146] First, we need to convert these raw data into object data. For text data, we use natural language processing technology for preprocessing, including word segmentation, stop word removal, stem extraction, etc., to finally obtain structured text data. For audio data, we use speech recognition technology to transcribe it into text and perform text preprocessing in the same way. For video data, we first use video-to-text technology to extract subtitles or narrations from the video, and then perform the same text preprocessing.

[0147] Furthermore, in order to integrate these different types of text data, we used a multimodal fusion technology based on deep learning. First, we used the pre-trained BERT model to encode each type of text data separately to obtain their respective semantic vector representations. Then, we designed a multimodal fusion network that contains a shared feature extraction layer and a fusion layer. The shared feature extraction layer is used to extract the common features of each data type, while the fusion layer fuses these features together to form a comprehensive representation vector.

[0148] In the training phase, we use the fused vectors as input to train a multi-task neural network model that can simultaneously learn patterns in text, audio, and video data related to the topic of "Public Perception of Global Climate Change". We use the cross-entropy loss function to supervise the learning process of the model and adjust the model parameters by gradient descent to minimize the loss function.

[0149] After the model is trained, we deploy it to real-time data streams to monitor and capture the latest information related to the topic of "Public Perception of Global Climate Change". Whether it is related content posted on news websites, social media, podcasts, or video platforms, as long as it is related to the topic, the model can quickly identify and automatically collect it. For example, when a new climate change documentary is released on YouTube, or a well-known podcast releases a program about the impact of climate change, our model can detect these contents in time and incorporate them into the dataset, providing rich multimodal data support for subsequent analysis and research.

[0150] Compared with the traditional solution of collecting data through keywords, the technical solution provided in the embodiment of the present application can integrate different types of object data (text, audio, video) into a unified data set, fully utilize the advantages of various data types, improve the generalization ability and prediction accuracy of the model, and provide a more comprehensive perspective for data collection.

[0151] Figure 2A schematic diagram of a data acquisition system based on a neural network model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the system includes:

[0152] The acquisition module 21 is used to determine the target subject and acquire raw data based on the target subject;

[0153] A conversion module 22, configured to convert the original data into text data, wherein the text data includes a plurality of text segments;

[0154] A calculation module 23, used to calculate a semantic relevance index of each text segment, wherein the semantic relevance index is used to indicate the relevance between the key information in the text segment and the target topic;

[0155] A training module 24 is used to select text segments that meet the requirements of a preset semantic relevance index to construct a data set, and train a neural network model using the data set;

[0156] The acquisition module 21 is also used to acquire target data based on the target subject through a trained neural network model.

[0157] Optionally, in an embodiment of the present application, the acquisition module 21 is specifically used to extract relevant key information from the target subject, wherein the key information includes at least keywords or phrases; identify multiple data sources related to the target subject, and select an adaptive data acquisition tool according to the type of each data source; and collect original data from the multiple data sources according to the key information through the data acquisition tool.

[0158] Optionally, in an embodiment of the present application, the conversion module 22 is specifically used to remove non-text elements in the original data to obtain adjusted original data; the non-text elements include at least: HTML tags, JavaScript codes; using regular expressions, extracting text data from the adjusted original data, and dividing the text data into multiple text fragments at the sentence level; or, dividing the text data into multiple text fragments at the paragraph level according to line breaks or paragraph marks.

[0159] Optionally, in the embodiment of the present application, the calculation module 23 is specifically used to calculate the following formula:

[0160] Calculate the semantic relevance index of each text segment;

[0161] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i) is represented by the TF-IDF value of the key information, v j It represents all the words in the text segment T, n represents the number of key information, and m represents the number of all words.

[0162] Optionally, in an embodiment of the present application, the training module 24 is specifically used to set a threshold value of a preset semantic relevance index, and to take a text segment whose semantic relevance index is higher than the threshold value of the preset semantic relevance index as a text segment that meets the requirements of the preset semantic relevance index; the text segments that meet the requirements of the preset semantic relevance index are divided into a training set and a test set, and a neural network model is trained based on the training set, and the hyperparameters of the neural network model are adjusted based on the test set until the neural network model completes training.

[0163] Optionally, in the embodiment of the present application, the conversion module 22 is further used to convert the original data into object data, where the object data includes at least one of text data, audio data or video data, or a combination of any several of them;

[0164] The training module 24 is also used to integrate different types of object data into a data set using multimodal fusion technology, and train the neural network model with the data set;

[0165] The acquisition module 21 is also used to acquire target data based on the target subject through a trained neural network model.

[0166] Optionally, in the embodiment of the present application, the calculation module 23 is specifically used to calculate the following formula:

[0167]

[0168] Calculate the semantic relevance index of each text segment;

[0169] Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, CtxSim(w i , T) is represented by the key information w i The semantic similarity with the text segment T, wherein the semantic similarity refers to the key information w i The average cosine similarity between the context embedding vector of the text segment T and the context embedding vectors of other words in the text segment T, v jrepresents all the words in the text segment T, n represents the number of key information, m represents the number of all words, S represents the target topic, and SemSim(T, S) represents the cosine similarity between the encoding vector of the text segment T and the encoding vector of the target topic S.

[0170] Figure 2 The data acquisition system based on the neural network model can perform Figure 1 The implementation principle and technical effect of the data acquisition method based on the neural network model described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the data acquisition system based on the neural network model in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0171] In one possible design, Figure 2 The data acquisition system based on the neural network model of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0172] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0173] The processing component 32 is used to: determine a target topic and collect raw data based on the target topic; convert the raw data into text data, the text data includes multiple text segments, and calculate the semantic relevance index of each text segment, the semantic relevance index is used to indicate the degree of relevance between the key information in the text segment and the target topic; screen out text segments that meet the preset semantic relevance index requirements to construct a data set, and train a neural network model through the data set; collect target data based on the target topic through the trained neural network model.

[0174] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0175] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0176] Of course, a computing device may also include other components, such as input / output interfaces, communication components, etc.

[0177] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0178] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0179] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0180] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The data collection method based on the neural network model of the illustrated embodiment.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data collection method based on a neural network model, characterized in that: include: Determine a target topic and collect original data based on the target topic; Converting the original data into text data, the text data including a plurality of text segments, and calculating a semantic relevance index of each text segment, the semantic relevance index being used to indicate the relevance of key information in the text segment to the target topic; Screening out text segments that meet the requirements of a preset semantic relevance index to construct a data set, and training a neural network model through the data set; Target data is collected based on the target topic through the trained neural network model.

2. The method according to claim 1, characterized in that: The step of determining a target topic and collecting raw data based on the target topic includes: Extract relevant key information from the target topic, wherein the key information at least includes keywords or phrases; Identify multiple data sources related to the target topic, and select an appropriate data collection tool according to the type of each data source; The data collection tool collects raw data from the multiple data sources according to the key information.

3. The method according to claim 1, characterized in that The converting the original data into text data, wherein the text data includes a plurality of text segments, including: Removing non-text elements from the original data to obtain adjusted original data; the non-text elements at least include: HTML tags and JavaScript codes; Using regular expressions, text data is extracted from the adjusted original data, and the text data is segmented into a plurality of text segments at a sentence level; or, the text data is segmented into a plurality of text segments at a paragraph level according to line breaks or paragraph marks.

4. The method according to claim 3, characterized in that The step of calculating the semantic relevance index of each text segment includes: By formula: Calculate the semantic relevance index of each text segment; Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) is represented by the TF-IDF value of the key information, v j It represents all the words in the text segment T, n represents the number of key information, and m represents the number of all words.

5. The method according to claim 4, characterized in that The step of screening out text segments that meet the preset semantic relevance index requirements to construct a data set includes: Setting a threshold value of a preset semantic relevance index, and taking a text segment whose semantic relevance index is higher than the threshold value of the preset semantic relevance index as a text segment that meets the requirement of the preset semantic relevance index; The text segments that meet the preset semantic relevance index requirements are divided into a training set and a test set, and a neural network model is trained based on the training set, and the hyperparameters of the neural network model are adjusted based on the test set until the neural network model completes the training.

6. The method according to claim 1, characterized in that Also includes: Converting the original data into object data, wherein the object data includes at least one of text data, audio data or video data or a combination of any several of the above; Using multimodal fusion technology, different types of object data are integrated into a data set, and the neural network model is trained with the data set; Target data is collected based on the target topic through the trained neural network model.

7. The method according to claim 4, characterized in that The step of calculating the semantic relevance index of each text segment includes: By formula: Calculate the semantic relevance index of each text segment; Wherein, T represents the text segment, SRI(T) represents the semantic relevance index of the text segment T, and w i It is expressed as the key information related to the target topic, TF-IDF(w i ) represents the TF-IDF value of the key information, CtxSim(w i , T) is represented by the key information w i The semantic similarity with the text segment T, wherein the semantic similarity refers to the key information W i The average cosine similarity between the context embedding vector of the text segment T and the context embedding vectors of other words in the text segment T, v j represents all the words in the text segment T, n represents the number of key information, m represents the number of all words, S represents the target topic, and SemSim(T, S) represents the cosine similarity between the encoding vector of the text segment T and the encoding vector of the target topic s.

8. A data acquisition system based on a neural network model, characterized in that: include: A collection module, used to determine a target topic and collect raw data based on the target topic; A conversion module, used for converting the original data into text data, wherein the text data includes a plurality of text segments; A calculation module, used to calculate a semantic relevance index of each text segment, wherein the semantic relevance index is used to indicate the relevance degree between the key information in the text segment and the target topic; A training module, used to screen out text segments that meet the requirements of a preset semantic relevance index to construct a data set, and train a neural network model using the data set; The acquisition module is also used to acquire target data based on the target subject through a trained neural network model.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the data acquisition method based on the neural network model as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer executes the computer program, the data acquisition method based on the neural network model as claimed in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Spark-based multi-feature combined efficient Chinese text clustering method

    CN107590218A

  • Label automatic generating method and system, computer readable storage medium and equipment

    CN108959431A

  • Information extraction method and device, electronic equipment and computer readable storage medium

    CN116151238A

  • Data acquisition method and device, chip and terminal

    CN116975246A

  • Data retrieval method and device based on word vector matching and medium

    CN117786049A